GPTQ-style round-to-nearest with Hessian weighting

In yesterday's INT8 post I implemented naive round-to-nearest quantization. It works well for INT8 because 256 levels give you enough precision that most rounding errors are harmless. But push to INT4 (16 levels) or INT3, and naive rounding causes real quality loss. GPTQ solves this by being smarter about how it rounds: it considers which weights matter most and compensates for each rounding error by adjusting the weights it hasn't quantized yet.

The key insight: not all weights are equal

When you quantize a weight matrix W in a linear layer y = Wx, the error in the output depends on two things: the rounding error in each weight, and how much that weight contributes to the output. A weight that multiplies a large, frequently activated input feature matters more than one that multiplies a near-zero input. The Hessian matrix captures this: it tells you how sensitive the layer's output is to changes in each weight.

For a linear layer with input X (the calibration data), the Hessian of the squared error with respect to the weights is:

H = 2 * X^T X

The diagonal of H tells you how sensitive the output is to each weight column. Large diagonal entries mean that column of weights is important: rounding errors there will cause large output errors. GPTQ uses this information to quantize weights in an order that minimizes cumulative error.

The OBQ/GPTQ algorithm

GPTQ is built on Optimal Brain Quantization (OBQ), which itself extends Optimal Brain Surgeon. The core algorithm processes one column of the weight matrix at a time:

• Step 1: Pick the next column to quantize (GPTQ processes them in order, left to right, for efficiency).
• Step 2: Round that column's weights to the nearest quantization level.
• Step 3: Compute the rounding error for each weight in that column.
• Step 4: Distribute the error to the remaining (not yet quantized) columns, weighted by the Hessian. This is the key step: it adjusts future weights to compensate for the rounding error you just introduced.

import torch

def gptq_quantize_layer(weight, hessian, n_bits=4, group_size=128):
    """
    GPTQ quantization for one linear layer.

    weight: (out_features, in_features) float16
    hessian: (in_features, in_features) float32, H = X^T @ X / num_samples
    n_bits: quantization bit width
    group_size: number of columns sharing a scale factor
    """
    W = weight.clone().float()
    rows, cols = W.shape
    H = hessian.clone()

    # Add small diagonal dampening for numerical stability
    damp = 0.01 * torch.mean(torch.diag(H))
    H += damp * torch.eye(cols, device=H.device)

    # Cholesky decomposition of the Hessian
    # H_inv is used to compute the error compensation
    H_inv = torch.linalg.cholesky(H)
    H_inv = torch.cholesky_inverse(H_inv)
    H_inv_chol = torch.linalg.cholesky(H_inv, upper=True)

    quantized = torch.zeros_like(W, dtype=torch.int8)
    scales = torch.zeros(rows, cols // group_size, device=W.device)
    zeros = torch.zeros(rows, cols // group_size, device=W.device)

    # Process columns left to right
    for col_start in range(0, cols, group_size):
        col_end = min(col_start + group_size, cols)

        # Compute scale for this group
        w_group = W[:, col_start:col_end]
        w_min = w_group.min(dim=1).values
        w_max = w_group.max(dim=1).values
        qmin, qmax = 0, 2**n_bits - 1
        scale = (w_max - w_min) / (qmax - qmin)
        scale = torch.clamp(scale, min=1e-8)
        zero_point = torch.round(-w_min / scale).clamp(qmin, qmax)

        group_idx = col_start // group_size
        scales[:, group_idx] = scale
        zeros[:, group_idx] = zero_point

        for j in range(col_start, col_end):
            # Quantize column j
            w_col = W[:, j]
            q_col = torch.clamp(
                torch.round(w_col / scale + zero_point),
                qmin, qmax
            )

            # Dequantize to get the quantized value
            w_hat = (q_col - zero_point) * scale

            # Quantization error
            error = (w_col - w_hat) / H_inv_chol[j, j]

            # Store quantized values
            quantized[:, j] = q_col.to(torch.int8)

            # Compensate remaining columns
            if j < cols - 1:
                W[:, j+1:] -= error.unsqueeze(1) * H_inv_chol[j, j+1:].unsqueeze(0)

    return quantized, scales, zeros

The magic is in the error compensation step. When you round column j and introduce an error, you adjust columns j+1 through n to partially cancel out that error's effect on the output. The Hessian inverse tells you exactly how to distribute the compensation.

Computing the Hessian

The Hessian H = X^T X is computed from calibration data: you run a small set of representative inputs through the model and collect the activations at each layer's input.

def collect_hessian(model, layer_name, calibration_loader, num_samples=128):
    """Collect the Hessian (X^T @ X) for a specific linear layer."""
    H = None
    n_samples = 0

    def hook_fn(module, input, output):
        nonlocal H, n_samples
        x = input[0].detach().float()  # (batch, seq_len, in_features)
        x = x.reshape(-1, x.shape[-1])  # (batch*seq_len, in_features)

        if H is None:
            H = torch.zeros(x.shape[1], x.shape[1], device=x.device)
        H += x.T @ x
        n_samples += x.shape[0]

    # Register forward hook on the target layer
    target = dict(model.named_modules())[layer_name]
    handle = target.register_forward_hook(hook_fn)

    for batch in calibration_loader:
        if n_samples >= num_samples * 2048:  # enough tokens
            break
        with torch.no_grad():
            model(batch["input_ids"].to(model.device))

    handle.remove()
    return H / n_samples

The calibration dataset should be representative of real inference inputs. Most GPTQ implementations use 128 samples from C4, WikiText-2, or a similar corpus. The choice matters less than having enough samples: 128 sequences of 2048 tokens gives a stable Hessian estimate.

Why GPTQ works so well at INT4

At INT8 (256 levels), naive round-to-nearest is already very good because the quantization grid is fine enough that errors are small. The advantage of GPTQ over naive rounding is minimal at INT8.

At INT4 (16 levels), the story changes. With only 16 possible values per weight, rounding errors are large, and they compound across the thousands of weights in a column. GPTQ's error compensation prevents this compounding: each rounding error is actively counteracted by adjusting future weights. The result is that GPTQ at INT4 often achieves perplexity within 0.5 points of FP16, while naive INT4 quantization can degrade by 2 or more points.

The practical pipeline

In practice, you don't implement GPTQ from scratch for production use. The AutoGPTQ library and Hugging Face's integration make it a few lines:

# Using AutoGPTQ (the practical way)
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_id = "meta-llama/Llama-3.1-70B-Instruct"
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=True,    # Use activation-order (slower but more accurate)
    damp_percent=0.01,
)

model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Calibration
calibration_data = [tokenizer(text, return_tensors="pt") for text in calibration_texts]
model.quantize(calibration_data)

# Save quantized model
model.save_quantized("llama-3.1-70b-gptq-int4")

Quantization of a 70B model with 128 calibration samples takes roughly 3 to 4 hours on a single A100. The result is a model that uses roughly 35 GB (down from 140 GB in FP16) and runs significantly faster on memory-bound decode workloads.

GPTQ vs. AWQ vs. naive: when to use what

• INT8, any method: Naive round-to-nearest is fine. The precision loss is negligible. Use it for the simplest possible pipeline.
• INT4 with GPTQ: The gold standard for 4-bit quantization. Requires calibration data and takes hours to quantize, but produces the best quality. Use for production deployments where you quantize once and serve many times.
• INT4 with AWQ: Activation-aware weight quantization. Similar quality to GPTQ but faster to quantize because it scales weights by activation magnitude rather than using full Hessian-based compensation. Often preferred when quantization speed matters.
• INT4 with naive rounding: Don't. The quality loss is too large for most models.

Understanding GPTQ's internals matters even if you use a library, because it tells you why calibration data matters, why group size affects quality, and why some models quantize better than others. A model with smooth, well-distributed weights quantizes easily. A model with outlier weights (large values concentrated in a few channels) fights the quantizer at every step.

Next: a quantization sweep measuring perplexity vs. compression across bit widths and methods, so we can see the tradeoffs in hard numbers.