Blackwell, and what four thousand of them add up to.
A B200 is not a bigger Hopper. It is two dies pretending to be one, with a number format that halved what a frontier model weighs. This is how the part works, and then what happens when you have 4,000 of them wired as 500 eight-GPU nodes: 720 TB of HBM, 36 EFLOPS of FP4, and one ratio that quietly decides every placement decision you will ever make.
The part
one GPUTwo dies, one GPU
B200 is two Blackwell dies joined by a 10 TB/s die-to-die link. CUDA sees one device, the silicon is two. This is how NVIDIA stepped around the reticle limit, and it is the reason the part has 180 GB of HBM3e on an HGX baseboard when a single die could not carry it.
FP4, and why it is usable
Fifth-generation tensor cores do FP4 at 9 PFLOPS dense, double the FP8 rate. Naive 4-bit quantisation wrecks a model. What makes this one work is micro-tensor scaling in the second-generation Transformer Engine: scale factors per 32-element block rather than per tensor, so outliers stop poisoning everything around them.
Memory is the product
180 GB of HBM3e at 7.7 TB/s. Decode is memory-bound, which means the token rate is set by how fast you can stream weights out of HBM, not by how many FLOPs the die can do. Bandwidth is the spec that matters, and it is 2.4x what an H100 gave you.
NVLink 5
1.8 TB/s per GPU, twice Hopper. Inside an HGX node an NVSwitch gives all eight GPUs all-to-all at that rate, 14.4 TB/s aggregate. This is the number that makes tensor parallelism across eight GPUs feel like one machine, and the reason the ninth GPU is a different kind of decision.
The ratio that decides everything
36xEvery placement question on this hardware reduces to one comparison. Inside a node, two GPUs talk over NVLink at 1.8 TB/s. Between nodes they talk over a ConnectX-7 port at 400 Gb/s, which is 50 GB/s. That is a factor of thirty-six, and it is why tensor parallelism stops at eight here and expert parallelism has to be laid out with the node boundary in mind rather than against it.
Four thousand of them
500 HGX nodesEight B200s per HGX node, 500 nodes. Drag the slider to see how the totals move, and which models still fit.
What actually fits
weights against the nodeA placement counts only if the KV cache gets a real share of what is left after weights. A config that fits the model and nothing else cannot serve a request.
Three things I would tell you before you plan a fleet
opinionThe node is the unit, not the GPU
Once tensor parallel traffic crosses a NIC you are on the wrong side of a 36x cliff, so the useful question is never "how many GPUs" but "how many eight-GPU islands, and what fits in one". DeepSeek V4 Pro at 800 GB uses an entire island and leaves 525 GB for cache. GLM-5.3-Flash uses half of one.
HGX is not NVL72, and the difference is the whole design
Most published frontier-model numbers, including NVIDIA's 150 tokens/sec/user on V4 Pro, come from GB200 NVL72, where 72 GPUs share one NVLink domain. On HGX B200 the domain is eight. The models still fit. The cross-replica cache behaviour does not transfer, and any capacity plan copied from an NVL72 benchmark will be wrong in the direction that hurts.
Power is a real constraint before you run out of GPUs
500 nodes at 10.2 kW is 5.1 MW for the GPU nodes alone, before cooling, networking or storage. On most floors that number, not the GPU count, is what ends the conversation, and it is worth knowing which of your models is the most tokens per watt rather than the most tokens per GPU.