Quantization is the easiest 2-4x you'll ever get. It doesn't require new hardware, new kernels, or new architecture. It just requires the right tooling to do it without wrecking quality. NVIDIA's Model Optimizer is that tooling.
Quantisation is the cheapest large win I know of, and also the one I am most careful about. A 2x that costs you quality you did not measure is not a 2x, it is a decision you made without noticing.
What Model Optimizer is
Model Optimizer is a unified library of state-of-the-art model optimization techniques: quantization, distillation, pruning, neural architecture search, and speculative decoding. It compresses models for downstream deployment frameworks like TensorRT-LLM, TensorRT, and vLLM, to optimize inference speed.
The key word is unified: instead of stitching together separate tools for each technique, you get one library that does them all, with output that drops straight into your serving runtime.
The quantization formats
- NVFP4: the 4-bit float format for Blackwell, the one that makes B200's 9000 TFLOPS real.
- FP8: the 8-bit float format for Hopper and Blackwell.
- INT8: the classic integer quantization, still the workhorse for many models.
- INT4: the aggressive option, with 4-bit integers.
Model Optimizer supports all of them, plus advanced algorithms like SmoothQuant that shift the quantization burden from activations to weights, where it's easier to absorb.
The quality-preservation algorithms
Raw quantization loses quality. The algorithms in Model Optimizer are designed to minimize that loss:
- SmoothQuant: migrate activation outliers into weights, so the quantization ranges are tighter.
- AWQ: activation-aware weight quantization, protecting the salient weights.
- GPTQ: one-shot weight quantization via approximate second-order information.
- Quantization-aware training: train with quantization in the loop, so the model learns to be robust to it.
Quantization is like compressing a photo to JPEG. Done naively, you get artifacts. Done with the right algorithm, nobody can tell the difference and the file is a quarter the size.
Memory, bandwidth, and what they buy
- Memory. A 2-4x smaller model means more models per GPU, or longer contexts, or both.
- Bandwidth. Smaller weights mean less HBM traffic, which is the decode bottleneck.
- Tensor core utilization. FP4/FP8 map to the tensor core paths that make Blackwell fast.
Quantization is the easiest 2-4x you'll ever get, and with the right tooling it doesn't have to cost quality.
Sources
- Model-Optimizer repo: the unified optimization library.
- Model Optimizer docs: the formats, algorithms, and examples.
- TensorRT-LLM repo: the serving runtime these optimizations target.
- SmoothQuant: the algorithm that shifts outliers from activations to weights.
- AWQ: activation-aware weight quantization.