NVLink is the high-speed interconnect that connects GPUs to each other, and NVSwitch is the switch that connects many GPUs into one fabric. Together they're the reason a 72-GPU rack can act like one giant GPU.
NVLink generations
- NVLink 1 (P100): 160GB/s per GPU.
- NVLink 2 (V100): 300GB/s per GPU.
- NVLink 3 (Ampere): 600GB/s per GPU (A100). 12 links.
- NVLink 4 (Hopper): 900GB/s per GPU (H100/H200). 18 links of 50GB/s each.
- NVLink 5 (Blackwell): 1.8TB/s per GPU (B200). 18 ports, doubled from Hopper.
- NVLink 6 (Rubin): 3.6TB/s per GPU, 260TB/s aggregate per NVL72 rack.
NVSwitch
NVLink connects a few GPUs directly. NVSwitch is the switch that connects many. NVSwitch 3 ties up to 256 H100s into one fabric with 900GB/s per GPU. NVSwitch 4 (Blackwell) introduced the 72-GPU NVL72 domain with 130TB/s and SHARP v3 in-network reductions, where every AllReduce, AllGather, and All-to-All collective operates at full bisection bandwidth.
The key number: the fabric bandwidth is faster than the memory bandwidth. That's what makes tensor parallelism work: the all-reduce between GPUs is not the bottleneck.
Scale-up vs scale-out
GPU cluster networking is a two-tier architecture. Scale-up (NVLink/NVSwitch) handles intra-node parallelism: tensor, expert, pipeline. Scale-out (InfiniBand/EFA) handles inter-node: data parallelism and disaggregated inference.
The bandwidth hierarchy per GPU (bidirectional): NVLink 6: 3,600 GB/s | NVLink 5 (B200): 1,800 GB/s | NVLink 4 (H100): 900 GB/s | PCIe Gen5 x16: 128 GB/s | InfiniBand NDR (400G): 50 GB/s | XDR (800G): 100 GB/s. That 30-70x gap is why parallelism strategy maps to network tiers: TP needs NVLink, DP can use InfiniBand.
For MoE inference, the all-to-all pattern sends each token to its selected expert every layer. On NVLink (1.8 TB/s per GPU), this completes in microseconds; on InfiniBand (50 GB/s), it would take 36x longer. That's why expert parallelism lives on NVLink.
Why it matters for inference
Tensor parallelism (splitting a model across GPUs) requires the GPUs to communicate every layer. Without NVLink, that communication is the bottleneck. With NVLink, the communication is faster than the compute, so the model scales almost linearly.
For a 671B MoE model spread across 8 GPUs, NVLink is what makes the all-to-all expert routing feasible. The experts live on different GPUs, and the token routes to them over NVLink.
Sources
- NVIDIA NVLink page: the interconnect overview.
- NVIDIA NVSwitch page: the switch overview.
- NVIDIA Blackwell blog: NVLink 5 details.