Kernels were the topic. Now the containers those kernels operate on: model formats. The journey from training to production is a journey through formats, and each one makes a different trade.
safetensors: the safe default
safetensors is the modern format for raw weights. It's a simple, safe, zero-copy serialization of tensors. It's what Hugging Face uses, and it's what you get when you download a model from the Hub.
Why it matters for inference: it's the input to everything else. You load safetensors, then convert to whatever your engine needs. It's the lingua franca of weights.
ONNX: the portable intermediate
ONNX is a graph format, not just weights. It describes the operations, not just the numbers. Its superpower is portability: export once, run anywhere, from PyTorch to ONNX Runtime to TensorRT.
The tradeoff: a graph is more than weights, so it's bigger and slower to load. And the portability comes at a cost, the graph has to be re-optimized for each backend.
TensorRT: the compiled engine
TensorRT is the endgame of formats. It's not a format so much as a compiled engine: the whole model, fused into optimized kernels, ready to run. Maximum performance, minimum flexibility.
This is what TensorRT-LLM produces. The tradeoff is build time (compilation can take minutes to hours) and hardware specificity (an engine built for one GPU doesn't run on another).
safetensors is the ingredients, ONNX is the recipe, TensorRT is the fully cooked meal. You shop with one, cook with the recipe, and serve the meal.
Which one when
- safetensors when you need portability and zero-copy loading.
- ONNX when you need to move between frameworks.
- TensorRT when you need every last drop of performance and control the hardware.
- GGUF (bonus) when you're on CPU or edge, the llama.cpp format.
The takeaway
Formats are bets on portability vs performance. Choose the one that matches your deployment, because the format is the first optimization decision you make.
Next: vLLM, where it all comes together.