If you squint at the history of NVIDIA datacenter GPUs through the lens of LLM inference, a clear pattern emerges. Each generation does not just add more transistors. It adds the specific capability that the current generation's bottleneck demands. Hopper added FP8 because FP16 was leaving compute on the table. Blackwell doubled the die because single-die scaling hit physics limits. Rubin adds HBM4 because memory bandwidth is still the wall. Let me walk through each one.
Hopper (H100, H200): the FP8 generation
The H100, launched in 2022, was the first datacenter GPU designed with LLM inference as a primary workload. Its headline features for inference:
- FP8 Tensor Cores: 1,979 TFLOPS in FP8, exactly 2x the FP16 rate (989 TFLOPS). This was the first time you could get a meaningful compute speedup from quantizing to 8 bits, because the hardware natively operated on FP8. Previous generations required dequantization to FP16 before compute.
- 80 GB HBM3 at 3.35 TB/s: a 1.65x bandwidth improvement over the A100's 2.0 TB/s. For memory-bound decode, this translated directly to ~65% more tokens per second.
- Transformer Engine: hardware support for dynamic per-tensor FP8 scaling. The Transformer Engine automatically manages the conversion between FP8 compute and FP16/FP32 accumulation, maintaining numerical stability without manual calibration.
- NVLink 4.0 with NVSwitch: 900 GB/s bidirectional all-to-all within an 8-GPU node. This made TP=8 practical with minimal communication overhead.
- 50 MB L2 cache: up from 40 MB on A100. Critical for caching KV tensors during attention computation.
The H200 was a memory refresh of Hopper, keeping the same compute architecture but upgrading to 141 GB HBM3e at 4.8 TB/s. This single change was transformative: a 70B BF16 model that needed TP=2 on H100 could fit on a single H200. Fewer GPUs means less communication overhead, simpler deployment, and lower cost.
Sometimes the biggest inference improvement is not faster compute but more and faster memory. The H200 proved that doubling HBM capacity and increasing bandwidth by 43% could matter more than any architectural change.
Blackwell (B100, B200, GB200): the dual-die generation
Blackwell, announced in 2024, took an unusual approach: instead of one massive die, the GPU is built from two dies connected by a 10 TB/s chip-to-chip interconnect. This is a manufacturing decision (two smaller dies have better yields than one giant die) that also delivers a raw performance leap:
- FP4 Tensor Cores: native NVFP4 support at roughly 4,500 TFLOPS on the B200. This is 2.25x the FP8 rate of the H100. Combined with the micro-scaling FP4 format, this allows 4-bit inference with hardware acceleration.
- 192 GB HBM3e at 8.0 TB/s: the bandwidth nearly doubles versus H100. For decode, this is the most impactful number. A 70B FP8 model (70 GB) loads in ~8.75 ms per token, giving ~114 tokens/second per sequence before batching.
- Second-generation Transformer Engine: supports FP4 dynamic scaling and mixed-precision strategies where different layers use different precisions based on their sensitivity.
- NVLink 5.0: 1.8 TB/s bidirectional per GPU, double the H100. This makes TP=8 even more efficient and makes TP across nodes (with NVLink Switch) feasible for the first time.
- Decompression engine: hardware support for decompressing weights on the fly from compressed formats, potentially reducing the effective memory footprint further.
The GB200 pairs two B200 GPUs with a Grace ARM CPU in a single module, connected by a coherent NVLink interface. This is NVIDIA's push toward a unified CPU-GPU memory space, simplifying data movement for inference pipelines that involve preprocessing on the CPU.
Rubin (R100): the HBM4 generation
Rubin, announced on NVIDIA's 2025 roadmap for expected delivery in 2026-2027, represents the next architectural leap. While specific details are still evolving, the key announced features include:
- HBM4: the next generation of high-bandwidth memory. HBM4 is expected to deliver significantly higher bandwidth than HBM3e, potentially exceeding 12 TB/s per GPU. For inference workloads that remain memory-bandwidth-bound even on Blackwell, this is the relief valve.
- Vera CPU: NVIDIA's custom ARM-based CPU designed to pair with Rubin GPUs, replacing the Grace CPU architecture. This suggests tighter CPU-GPU integration.
- Next-generation NVLink: further interconnect bandwidth improvements for multi-GPU and multi-node scaling.
The Rubin generation signals that NVIDIA sees memory bandwidth as the persistent bottleneck for the foreseeable future. Even with FP4 reducing the compute-per-byte ratio, the sheer size of modern models (hundreds of billions to trillions of parameters) means the memory wall is not going away.
What this means for inference engineers
Each generation shifts the optimization landscape:
- On Hopper: FP8 quantization is the standard. TP=8 within a node works well. The bottleneck for decode is HBM bandwidth at 3.35 TB/s.
- On Blackwell: FP4 becomes viable, halving memory traffic again. The 8.0 TB/s bandwidth means decode throughput nearly doubles. Models that needed 4 GPUs on Hopper might need 2 on Blackwell.
- On Rubin: with HBM4 bandwidth potentially exceeding 12 TB/s, and even smaller number formats, decode may finally approach the point where the bottleneck shifts back toward compute for some workloads. That would be a fundamental change.
The GPU roadmap is the inference roadmap. Every generation attack the current bottleneck. Understanding where you are on the roofline today tells you exactly which spec upgrade to care about tomorrow.
Next: multi-GPU instances and MIG, where we look at how cloud providers expose these GPUs and how MIG lets you carve a single GPU into isolated slices for different workloads.