H200 is the answer to a question nobody asked in 2022 but everyone needed by 2024: what if the compute stayed the same but the memory got huge? For inference, that's often the whole ballgame.
What changed from H100
- Memory size. 141GB of HBM3e, up from H100's 80GB. That's a 76% increase.
- Memory bandwidth. 4.8TB/s, up from 3.35TB/s. That's a 43% increase.
- Compute. Identical to H100. Same SMs, same tensor cores, same FP8 throughput.
- Everything else. Same NVLink 4, same power, same package.
The numbers
- FP16: 989 TFLOPS dense (same as H100).
- FP8: 1979 TFLOPS dense (same as H100).
- Memory: 141GB HBM3e, 4.8TB/s.
- NVLink 4: 900GB/s per GPU.
- Power: 700W TDP.
Decode is memory-bound, and this is the fix
Decode is memory-bound. The GPU streams weights once per token, and the bandwidth determines the token rate. More bandwidth = faster decode. More capacity = bigger KV cache, longer context, more concurrent requests. H200 is the GPU you pick when you want maximum tokens per second per dollar on decode-heavy workloads.
It's also the GPU that made 70B models comfortable: 70B in FP8 is ~70GB, and with 141GB you have 70GB left for KV cache and activations. That's a lot of concurrent long-context sessions on one GPU.
Sources
- NVIDIA H200 product page: official specs.
- NVIDIA H100 whitepaper: the Hopper architecture H200 shares.