Skand Tandon
Production Systems

FP16 vs INT8 vs INT4 throughput

Quantization promises smaller models and faster inference. But how much faster, and at what cost to quality? This post walks through the mechanics of why lower precision helps, what the real throughput gains look like, and where quality starts to degrade.

2025-12-12 production systems 9 min read

After profiling GPU memory, the next natural question is: what happens if I shrink the model? Quantization reduces the number of bytes per weight, which directly affects both memory footprint and throughput. But the relationship between precision and speed is not as simple as "half the bytes, double the speed." Understanding why requires going back to the roofline model.

Why quantization helps decode

Recall that autoregressive decode is memory-bandwidth-bound. Each decode step loads the entire model weights from HBM to compute a single token. The speed of decode is determined by how fast you can stream those weights through the memory bus, not by how fast the Tensor Cores can multiply.

This is why quantization has such a dramatic effect on decode throughput. If you cut each weight from 2 bytes (FP16) to 1 byte (INT8), you halve the memory traffic per decode step. Since the memory bus was the bottleneck, you nearly double the tokens per second. Cut to 0.5 bytes (INT4) and you theoretically quadruple decode speed relative to FP16.

Prefill is different. Prefill is compute-bound because it processes the entire prompt in a large matrix multiplication. Reducing weight precision helps prefill only if the hardware has faster compute units for that precision. The H100 has INT8 Tensor Cores that run at 2x the FP16 rate, so INT8 does help prefill too, but through a different mechanism.

The three precision levels

Let me be precise about what each format means and how the math works:

Memory footprint comparison

For Llama-3.1-8B (8.03 billion parameters):

# Weight memory by precision
params = 8.03e9

fp16_gb  = params * 2 / 1e9     # 16.06 GB
int8_gb  = params * 1 / 1e9     #  8.03 GB
int4_gb  = params * 0.5 / 1e9   #  4.02 GB

# KV cache is always in FP16 (or FP8 with recent vLLM)
# So KV cache memory does not change with weight quantization

The memory savings are straightforward. What matters for serving is not just fitting the model, but how much HBM is left for KV cache. Using the budget from yesterday on an 80 GB H100:

Going from FP16 to INT4 frees 12 GB for KV cache, which means roughly 22% more concurrent sequences. That is a real capacity gain.

Throughput: the theory

The theoretical decode throughput on a bandwidth-limited GPU is:

tokens_per_second = memory_bandwidth / bytes_per_token

# H100: 3.35 TB/s memory bandwidth
# bytes_per_token = model_size_bytes (full weight load per token)

fp16_tps = 3.35e12 / (16.06e9)   # ~209 tokens/s
int8_tps = 3.35e12 / (8.03e9)    # ~417 tokens/s
int4_tps = 3.35e12 / (4.02e9)    # ~833 tokens/s

These are theoretical single-sequence decode speeds assuming perfect bandwidth utilization and zero overhead. Real numbers are lower due to attention computation, KV cache reads, kernel launch overhead, and the dequantization work itself. But the ratios hold approximately: INT8 is roughly 1.5-2x faster than FP16 for decode, and INT4 is roughly 2.5-3.5x faster.

Where quality degrades

There is no free lunch. Quantization introduces error because you are rounding continuous-valued weights to a discrete grid. The question is whether that error matters for your use case.

Here is what I have observed across models and tasks:

Rule of thumb

INT8 is almost always safe. INT4 is safe for 7B+ models on most tasks. Below 3B parameters, test carefully before deploying INT4. Always validate on your actual evaluation suite, not just perplexity.

Running the benchmark

To measure throughput properly, you need to control for batch size, sequence length, and whether you are measuring prefill or decode. Here is how I set up a fair comparison using vLLM:

# Serve each quantization level
# FP16
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --dtype float16 \
    --port 8000

# INT8 (using SmoothQuant or a pre-quantized checkpoint)
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B \
    --quantization squeezellm \
    --dtype float16 \
    --port 8001

# INT4 (AWQ)
python -m vllm.entrypoints.openai.api_server \
    --model casperhansen/llama-3.1-8b-awq \
    --quantization awq \
    --dtype float16 \
    --port 8002

Then use the async batch client to send the same set of prompts to each server and measure output tokens per second:

# Benchmark configuration
# - Fixed prompt: 512 tokens
# - Fixed max_tokens: 256
# - Sweep batch sizes: 1, 4, 16, 64, 128
# - Measure: output tokens/s, TTFT, total time

What the numbers actually show

The general patterns that emerge from these benchmarks, consistently across different models and hardware:

The Marlin kernel advantage

Not all INT4 implementations are equal. Naive INT4 dequantization can actually be slower than FP16 because the dequantization overhead eats into the bandwidth savings. The Marlin kernel (developed by IST Austria, integrated into vLLM) solves this with a carefully optimized CUDA kernel that dequantizes INT4 weights in registers during the matrix multiply, achieving near-ideal throughput.

When using vLLM with AWQ or GPTQ models, Marlin is used automatically when the hardware supports it (Ampere and later). If you see INT4 performing worse than expected, check whether Marlin is actually being used by looking at the vLLM startup logs for "Using Marlin kernel."

Quantization is the single highest-ROI optimization for LLM serving. INT8 gives you a near-free 1.5-2x throughput boost with negligible quality loss. INT4 doubles that again if your model and task can tolerate it. Measure quality on your actual workload, not on leaderboard benchmarks.

Tomorrow I will look at a very different optimization strategy: speculative decoding, where you use a small draft model to generate candidate tokens and verify them in parallel with the large model.