Skand Tandon
Deep Implementation

Quantization sweep: perplexity vs compression

Yesterday I built the GPTQ pipeline. Today I actually run the sweep: FP16 baseline, INT8, INT4 symmetric, INT4 GPTQ, and NF4 (the QLoRA format). I measure perplexity on WikiText-2 at each level and plot the tradeoff curve that every quantization decision comes down to.

2025-11-07 deep implementation 8 min read

After spending day 35 on INT8 quantization and day 36 on GPTQ, I realized I had all the pieces but no head-to-head comparison. I kept seeing claims like "INT4 loses less than 1% accuracy" floating around Twitter without any reproducible backing. So today I decided to run the sweep myself and find out where different quantization schemes actually land on the perplexity-compression curve.

The goal is simple: take one model, quantize it five different ways, measure perplexity on the same evaluation set, record the model size, and plot both numbers against each other. No cherry-picking, no "it works fine in practice." Just numbers.

The setup

I used a 1.3B parameter GPT-2 style model for this sweep. Why not a 7B? Because I wanted to run every configuration on a single GPU without worrying about memory limits muddying the comparison. The quantization math is identical regardless of model size; what changes is how much slack the model has to absorb rounding errors, and smaller models have less slack, which makes quality differences more visible.

Evaluation dataset: WikiText-2 test split, 245K tokens. I compute perplexity using a sliding window of 2048 tokens with a stride of 512, following the protocol from the Hugging Face evaluate library. This avoids edge effects from truncation.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

def compute_perplexity(model, tokenizer, dataset, stride=512, max_length=2048):
    encodings = tokenizer("\n\n".join(dataset["text"]), return_tensors="pt")
    input_ids = encodings.input_ids.to(model.device)
    seq_len = input_ids.size(1)
    nlls = []
    prev_end = 0

    for begin in range(0, seq_len, stride):
        end = min(begin + max_length, seq_len)
        target_len = end - prev_end
        input_slice = input_ids[:, begin:end]

        with torch.no_grad():
            outputs = model(input_slice, labels=input_slice)
            # Only count loss on the new tokens (not the overlap)
            neg_log_likelihood = outputs.loss * target_len

        nlls.append(neg_log_likelihood)
        prev_end = end
        if end == seq_len:
            break

    ppl = torch.exp(torch.stack(nlls).sum() / prev_end)
    return ppl.item()

The five configurations

Here is what I tested, in order of decreasing model size:

How I ran each one

For INT8, I used PyTorch's built-in dynamic quantization on linear layers:

model_int8 = torch.quantization.quantize_dynamic(
    model_fp16, {torch.nn.Linear}, dtype=torch.qint8
)

For INT4 RTN and GPTQ, I used the auto-gptq library with group size 128. The difference between RTN and GPTQ is just whether you pass calibration data:

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# RTN: no calibration data, just round
config_rtn = BaseQuantizeConfig(bits=4, group_size=128, damp_percent=0.0)

# GPTQ: with calibration data and Hessian damping
config_gptq = BaseQuantizeConfig(bits=4, group_size=128, damp_percent=0.01)
model_gptq.quantize(calibration_dataset)  # 128 samples from C4

For NF4, I used bitsandbytes with the nf4 quantization type:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

The results

Here is what I measured. Model sizes are the actual file sizes on disk, not theoretical calculations:

Key observation

INT8 is nearly free. The perplexity hit is within noise. INT4 RTN is where things get interesting: naive rounding at 4 bits costs almost 2 perplexity points, but GPTQ and NF4 recover most of that through smarter quantization. GPTQ's Hessian weighting buys you about 1.4 points over naive RTN at the same compression ratio.

What the curve looks like

If you plot compression ratio on the x-axis and perplexity on the y-axis, you see a characteristic shape: flat from FP16 through INT8, then a knee at 4 bits. The knee is where the method matters. RTN sits noticeably above the other two 4-bit methods. GPTQ and NF4 cluster together, with GPTQ slightly ahead.

This is why people say "quantization is mostly free" when they mean INT8, and why the research community spent so much effort on GPTQ, AWQ, and friends: the hard problem is not going from 16 bits to 8 bits. It is going from 8 bits to 4 bits without losing the model's tail knowledge.

Where the quality loss hides

Perplexity is an average over the entire test set. It can hide localized damage. I noticed that the biggest per-token loss increases in INT4 RTN cluster around rare tokens and long-range dependencies. The model's most confident predictions (common tokens, short contexts) barely change. It is the uncertain, low-probability predictions that get hammered by naive rounding, because those predictions depend on precise weight interactions that rounding destroys.

GPTQ mitigates this by using the Hessian to identify which weights are "load-bearing" for the loss function. It rounds those weights more carefully and lets unimportant weights absorb more error. NF4 takes a different approach: by matching the quantization levels to the Gaussian distribution of weights, it minimizes the expected quantization error across the whole tensor without needing calibration data.

The right quantization scheme depends on your constraints. Need zero calibration data? Use NF4. Have calibration data and care about every fraction of a perplexity point? Use GPTQ. Just need something that works and INT8 is enough compression? Use per-channel symmetric and move on.

Practical takeaways

After running this sweep, three things crystallized for me:

Tomorrow I am moving away from quantization and into a completely different latency optimization: speculative decoding, where you use a small draft model to propose tokens and a large target model to verify them. The idea is beautiful, and the acceptance sampling math is surprisingly elegant.