Number formats: FP8, INT8, INT4, NVFP4

Quantization is the single most impactful optimization in LLM inference. Not because it is clever (it is, but that is not the point), but because it directly attacks the bottleneck. As we covered in the ops:byte ratio post, decode is memory-bandwidth-bound. Every weight has to travel from HBM to the compute units for every token generated. Cut the weight from 16 bits to 8 bits, and you move half the bytes. Cut it to 4 bits, and you move a quarter. The speedup is nearly linear with the compression ratio, because you are relieving the actual bottleneck.

But not all number formats are equal. Each one makes a different trade-off between range, precision, and hardware support. Let me walk through them.

FP16 and BF16: the baselines

Most models are trained in BF16 (brain floating point 16) or FP16 (IEEE half precision). Both use 16 bits per value but allocate them differently:

• FP16: 1 sign bit, 5 exponent bits, 10 mantissa bits. Good precision, limited range (max ~65504).
• BF16: 1 sign bit, 8 exponent bits, 7 mantissa bits. Less precision, but the same range as FP32 (max ~3.4e38). This is why BF16 became the default for training: you rarely overflow.

At 2 bytes per parameter, a 70B model takes 140 GB. That is nearly two H100 80GB GPUs just for the weights, before you even think about KV cache or activations.

FP8: the new standard

FP8 came to prominence with NVIDIA's Hopper architecture (H100), which added native FP8 Tensor Core support. There are two variants defined in the OFP8 standard:

• E4M3: 1 sign, 4 exponent, 3 mantissa bits. Range up to 448, precision to about 0.125. Used for weights and activations in forward passes.
• E5M2: 1 sign, 5 exponent, 2 mantissa bits. Wider range (up to 57344), less precision. Used for gradients in training.

# FP8 E4M3 bit layout
# [S][EEEE][MMM]
# S = sign, E = exponent, M = mantissa
#
# Example: 1.5 in E4M3
# Sign: 0, Exponent: 0111 (bias=7, so exp=0), Mantissa: 100
# Value: (-1)^0 * 2^0 * 1.100 = 1.5

The key insight with FP8 is that it is a floating-point format. It has an exponent, which means it can represent both very small and moderately large values. This makes it more forgiving than INT8 for weights that span a wide dynamic range. On the H100, FP8 Tensor Cores deliver roughly 2x the FLOPS of FP16 (1,979 TFLOPS vs 989 TFLOPS for dense operations). That is a free doubling of compute for prefill-bound workloads.

The catch: 3 mantissa bits means only 8 distinct values per exponent range. You need per-tensor or per-channel scaling factors to map the weight distribution into this narrow precision window. The quality of your scaling determines the quality of your quantized model.

INT8: the workhorse

INT8 quantization predates FP8 and is supported on every NVIDIA GPU from Turing onward. It maps floating-point weights to 256 integer values (-128 to 127) using a scale and optional zero-point:

quantized = round(weight / scale) + zero_point
dequantized = (quantized - zero_point) * scale

The advantage of INT8 is uniform precision across the entire range. Every value gets the same resolution. The disadvantage is that outliers (a few weights with much larger magnitude than the rest) waste most of that resolution on empty space. This is why techniques like SmoothQuant exist: they mathematically migrate the quantization difficulty from activations (which have outliers) to weights (which are smoother), making INT8 work much better.

On the H100, INT8 Tensor Cores deliver 1,979 TOPS, matching FP8 in raw throughput. The choice between FP8 and INT8 often comes down to which quantization scheme preserves more model quality for your specific architecture.

INT4 and GPTQ: pushing the limits

INT4 cuts the weight to just 4 bits and 16 possible values. At this level, naive round-to-nearest quantization destroys model quality. You need smarter algorithms:

• GPTQ: uses second-order (Hessian) information to decide how to round each weight, compensating for rounding error in subsequent weights. The result: surprisingly good quality at 4 bits for many models.
• AWQ (Activation-Aware Weight Quantization): identifies the 1% of "salient" weight channels that matter most for model quality and keeps them at higher precision, quantizing the rest aggressively.

INT4 halves memory vs INT8, putting a 70B model at 35 GB, which fits comfortably on a single 80GB GPU with room for KV cache. The throughput benefit is real but complicated: most GPUs do not have native INT4 Tensor Cores. Instead, INT4 weights are dequantized to FP16 on the fly during the GEMM. You save on memory bandwidth (the bottleneck) but not on compute.

NVFP4: NVIDIA's Blackwell format

With the Blackwell architecture (B100/B200), NVIDIA introduced native FP4 support in hardware. NVFP4 uses a micro-scaling format:

• Each value is 4 bits: 1 sign bit and 3 mantissa bits (with an implicit exponent shared across a block of values).
• A block of 16 or 32 values shares a single FP8 scaling factor, stored alongside the data.
• The effective bit rate is slightly above 4 bits per weight when you include the shared scales, but the computational format is true 4-bit floating point.

The Blackwell Tensor Cores can execute FP4 operations natively at roughly 2x the throughput of FP8, delivering around 4,500 TFLOPS on the B200. This is a massive jump. The combination of 4-bit weights and native hardware support means you get both the memory bandwidth savings of INT4 and actual compute speedup, something INT4 on Hopper cannot deliver.

Choosing the right format

Here is my practical decision tree:

• FP8 E4M3: default choice on Hopper. Minimal quality loss for most models, native hardware support, easiest to deploy. Start here.
• INT8 (SmoothQuant): when you need broader GPU compatibility or when your activation distributions have outliers that FP8 scaling handles poorly.
• INT4 (GPTQ/AWQ): when you must fit a large model on fewer GPUs and can tolerate a small quality regression. Always validate on your eval set.
• NVFP4: on Blackwell hardware, for maximum throughput. The micro-scaling format handles dynamic range better than INT4, so quality is often comparable to INT8.
• BF16/FP16: when quality is paramount and you have the GPU memory. Research, evaluation, and baseline measurements.

Quantization is not a quality sacrifice. It is a precision budget. The question is never "should I quantize?" but "how much precision does my application actually need?"

Next, we will look at how these formats interact with model parallelism strategies, because the choice of TP, EP, or PP changes which tensors benefit most from quantization.