INT8 quantization pipeline

I've referenced quantization in several posts now: it halves the memory traffic for memory-bound decode, it doubles the KV cache capacity, and serving engines like vLLM support it out of the box. But I've been treating quantization as a black box. Today I'm opening it up and implementing INT8 weight quantization from scratch to understand exactly what happens to the numbers.

What quantization does

A model trained in FP16 stores each weight as a 16-bit floating-point number. Quantization maps those weights to a lower-precision representation (8-bit integers in our case) plus a small amount of metadata (scale factors) that allows approximate reconstruction. The key insight: for inference, we don't need the exact weight values. We need values close enough that the model's output quality doesn't degrade noticeably.

INT8 quantization maps FP16 values to the range [-128, 127]. The mapping is linear: divide the range of weights by 255 to get a scale factor, then round each weight to the nearest integer.

Symmetric per-tensor quantization

The simplest scheme: find the maximum absolute value in the tensor, compute a single scale factor, and round everything.

import torch

def quantize_symmetric(weight_fp16):
    """
    Symmetric per-tensor quantization to INT8.
    weight_fp16: (out_features, in_features) in float16
    Returns: (quantized_int8, scale_float32)
    """
    # Find the range
    abs_max = weight_fp16.abs().max()

    # Scale factor: maps [-abs_max, abs_max] to [-127, 127]
    scale = abs_max / 127.0

    # Quantize: divide by scale, round to nearest integer, clamp
    quantized = torch.clamp(torch.round(weight_fp16 / scale), -128, 127).to(torch.int8)

    return quantized, scale.float()

def dequantize(quantized_int8, scale):
    """Reconstruct approximate FP16 values from INT8."""
    return quantized_int8.float() * scale

The reconstruction is lossy. The error for each weight is at most scale / 2, which is the rounding error. For a weight tensor with abs_max of 1.0, the scale is 1/127 ≈ 0.0079, so the maximum per-element error is about 0.004. For most model weights, this is small enough to be harmless.

Per-channel quantization: much better

Per-tensor quantization uses one scale factor for the entire weight matrix. If some output channels have much larger weights than others, the small channels lose precision. Per-channel (also called per-row) quantization uses a separate scale for each output channel:

def quantize_per_channel(weight_fp16):
    """
    Per-channel symmetric quantization to INT8.
    Each output channel (row) gets its own scale factor.
    """
    # abs_max per row: (out_features,)
    abs_max = weight_fp16.abs().amax(dim=1)
    abs_max = torch.clamp(abs_max, min=1e-8)  # avoid division by zero

    # Scale per channel: (out_features,)
    scales = abs_max / 127.0

    # Quantize each row with its own scale
    quantized = torch.clamp(
        torch.round(weight_fp16 / scales.unsqueeze(1)),
        -128, 127
    ).to(torch.int8)

    return quantized, scales.float()

def dequantize_per_channel(quantized_int8, scales):
    return quantized_int8.float() * scales.unsqueeze(1)

Per-channel quantization is strictly better than per-tensor for weight matrices because each channel is scaled independently. The cost is storing one float32 scale per output channel instead of one per tensor, but that's negligible (4 bytes per row vs. millions of bytes for the weights).

The full quantization pipeline

Quantizing a model for inference involves more than just converting weights. Here's the complete pipeline:

• Step 1: Identify which layers to quantize. Linear layers (the bulk of parameters) are the primary targets. Embedding layers, layer norms, and the final LM head are typically left in FP16 because they're small and sensitive to precision loss.
• Step 2: Choose the quantization granularity. Per-channel for weights, per-tensor or per-token for activations (if doing dynamic quantization).
• Step 3: Calibrate (for static quantization). Run a small calibration dataset through the model to determine the range of activations, which sets the activation scale factors. Weight-only quantization skips this step.
• Step 4: Quantize and save. Convert weights to INT8, store scale factors alongside them, and save in a format the serving engine understands.

def quantize_model_weights(model):
    """Quantize all linear layers in a model to INT8."""
    quantized_state = {}

    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            q_weight, scales = quantize_per_channel(module.weight.data)
            quantized_state[f"{name}.weight_int8"] = q_weight
            quantized_state[f"{name}.weight_scale"] = scales

            # Keep bias in FP16 (it's tiny)
            if module.bias is not None:
                quantized_state[f"{name}.bias"] = module.bias.data.half()
        else:
            # Keep non-linear layers in original precision
            for pname, param in module.named_parameters(recurse=False):
                quantized_state[f"{name}.{pname}"] = param.data

    return quantized_state

Weight-only vs. weight-activation quantization

There are two flavors of INT8 quantization for inference:

• Weight-only (W8A16). Weights are stored as INT8, but dequantized to FP16 before the matmul. The matmul itself runs in FP16. This halves memory for weight storage and loading, which directly helps the memory-bound decode phase. It's simple and almost lossless.
• Weight-activation (W8A8). Both weights and activations are INT8, and the matmul runs on INT8 Tensor Cores. This gives both memory and compute savings: INT8 Tensor Cores on H100 deliver roughly double the FLOPS of FP16 Tensor Cores. But quantizing activations requires calibration data to determine activation ranges, and outlier activations can cause accuracy problems.

Measuring quality loss

The real question with quantization is always: does the model still produce good outputs? The standard way to measure this is perplexity on a held-out dataset:

def measure_perplexity(model, tokenizer, texts, max_length=2048):
    """Compute perplexity on a list of texts."""
    total_loss = 0.0
    total_tokens = 0

    for text in texts:
        inputs = tokenizer(text, return_tensors="pt",
                          truncation=True, max_length=max_length).to(model.device)
        with torch.no_grad():
            outputs = model(**inputs, labels=inputs["input_ids"])
        total_loss += outputs.loss.item() * inputs["input_ids"].size(1)
        total_tokens += inputs["input_ids"].size(1)

    avg_loss = total_loss / total_tokens
    return torch.exp(torch.tensor(avg_loss)).item()

For well-implemented INT8 weight-only quantization on a 70B model, the perplexity increase over FP16 is typically less than 0.1 points on WikiText-2. That's within noise for most applications. W8A8 can show slightly larger degradation (0.1 to 0.3 points) depending on the model and calibration data.

What I take away

INT8 quantization is the lowest-risk, highest-reward optimization in LLM inference. It halves memory, doubles decode throughput on memory-bound workloads, and causes negligible quality loss for most models. Per-channel quantization is worth the trivial extra complexity over per-tensor. And weight-only quantization is the safe default: add W8A8 only when you need the compute savings and can afford the calibration effort.

Next: GPTQ-style quantization with Hessian weighting, where the quantization gets smarter by considering which weights matter most.