H100 is the reference point. When people say "a GPU", they usually mean an H100. It's the Ampere A100's successor, and it's the GPU that made FP8 inference real.
What changed from A100
- FP8 via the Transformer Engine. Hopper's tensor cores natively support FP8 (E4M3/E5M2), and the Transformer Engine automatically selects FP8 vs FP16 per layer. This is the single biggest inference win: ~2x throughput over A100 for the same model.
- NVLink 4. 900GB/s per GPU, up from A100's 600GB/s. NVLink Switch 3 ties up to 256 GPUs into one fabric.
- More SMs, more tensor cores. 132 SMs (vs A100's 108), and each SM has 4 tensor cores with FP8 support.
- HBM3. 80GB HBM3 at 3.35TB/s, up from A100's 2TB/s HBM2e. (The H100 SXM variant; the PCIe variant has 80GB HBM2e at 2TB/s.)
- TMA. The Tensor Memory Accelerator offloads all data movement addressing from the SMs, which is why PagedAttention performs better on Hopper than Ampere.
- Thread Block Clusters. A new hierarchy level lets thread blocks synchronize and share data across SMs via distributed shared memory.
The numbers
- FP16: 989 TFLOPS dense (SXM), 1979 with sparsity.
- FP8: 1979 TFLOPS dense, 3958 with sparsity.
- Memory: 80GB HBM3, 3.35TB/s.
- NVLink 4: 900GB/s per GPU, 18 links of 50GB/s each.
- Power: 700W TDP (SXM).
Why it matters for inference
FP8 is the reason H100 became the default serving GPU. A 70B model in FP8 is ~70GB, which fits on one H100 with room for KV cache. Before FP8, you needed two A100s. That single fact reshaped the economics of serving: one GPU per frontier-ish model, instead of two.
Hopper also introduced CUDA graphs as a first-class optimization (capture the kernel graph, replay it), which became the standard way to cut launch overhead. And its DPX instructions (for dynamic programming) matter less for LLMs but show the specialization trend.
Sources
- NVIDIA H100 product page: official specs.
- NVIDIA H100 whitepaper: the full Hopper architecture.
- NVIDIA Hopper architecture in depth: Transformer Engine, NVLink 4, DPX.