Quantization is the art of making a model smaller by using fewer bits per weight. A 70B model in FP16 is 140GB, too big for one GPU. In INT4 it's 35GB, which fits on an A100. The question is how to lose those bits without losing the model.
Three algorithms dominate, and each is a different answer to "what should we protect?"
GPTQ: round-to-nearest, but smart
GPTQ (arxiv.org/abs/2210.17323, Frantar et al. 2022) starts from the obvious: round each weight to the nearest 4-bit value. Then it fixes the damage. When you round one weight, you can adjust the remaining unquantized weights to compensate, using the Hessian to weight the correction. It's error compensation, column by column.
The insight: quantization error isn't uniform. Some weights matter more than others, and the Hessian tells you which. GPTQ spends its error budget on the weights that matter least, and corrects the ones that matter most.
AWQ: activation-aware scaling
AWQ (arxiv.org/abs/2306.00978, Lin et al. 2023, MIT) makes a different bet: the weights that matter are the ones multiplied by large activations. A quantization error on a weight is amplified by the activation it multiplies, so a weight that gets big activations needs more precision.
The elegant part: instead of keeping those weights in FP16 (mixed precision, which is hardware-unfriendly), AWQ scales them up before quantizing. Scaling a weight channel up before rounding and scaling it back after is mathematically equivalent to giving it more precision, but keeps everything in uniform INT4.
No backprop, no reconstruction, no calibration overfitting. Just a per-channel scaling factor derived from activation statistics. And it works: TinyChat gets 3x speedup over HF FP16, and 70B Llama-2 runs on a phone.
SmoothQuant: shift the outliers
SmoothQuant (github.com/mit-han-lab/smoothquant) targets the problem AWQ avoids: activation outliers. LLM activations have channels with values 100x larger than the median, which breaks INT8 quantization of activations. But weights are smooth, activations are spiky.
So SmoothQuant migrates the difficulty: it mathematically shifts the outliers from activations into the weights via a per-channel scaling factor, making activations smooth enough for INT8 while keeping weights in INT8 too. The result is W8A8 quantization, which is what makes FP8-era hardware (like the B200) usable.
GPTQ corrects errors after the fact. AWQ protects the important weights before rounding. SmoothQuant moves the problem from activations to weights. Three different bets on where the damage is.
Which one when?
- GPTQ when you want the best quality per bit on weight-only quantization, and can afford calibration time.
- AWQ when you want fast, calibration-light, hardware-friendly INT4 with no backprop. Great for deployment at scale.
- SmoothQuant when you need W8A8 for FP8 hardware, or when activations are your bottleneck.
All three are in production today. GPTQ powers many 4-bit deployments, AWQ is the default in vLLM and TensorRT-LLM, and SmoothQuant's W8A8 is the path to FP8.
The takeaway
Quantization is a story about which weights matter. GPTQ says the Hessian knows. AWQ says the activations know. SmoothQuant says the outliers know. They're all right, and that's why they all work.
Next: speculative decoding, where we spend idle compute to go faster.