NVLink connects GPUs within a rack. The cluster-wide network connects racks to each other. That's where InfiniBand, EFA, and RoCE come in, and it's the layer that makes a 4000-GPU cluster possible.
InfiniBand
InfiniBand is the high-performance network designed for HPC and AI. It's lossless (no packet drops, flow control at the hardware level), low-latency, and it implements RDMA natively. Generations: NDR (400 Gb/s per port), XDR (800 Gb/s per port). For a 4000-GPU cluster, the IB fabric is the backbone.
Key details: Quantum-2 NDR switch is 64 ports × 400G = 25.6 Tb/s switching capacity. Quantum-X800 XDR switch is 144 ports × 800G = 115.2 Tb/s. And SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) offloads AllReduce to the switch ASIC, reducing collective latency by 2-8x and halving the data traversing the network.
EFA: Elastic Fabric Adapter
EFA is AWS's version of InfiniBand, available on their GPU instances. It's a custom NIC that uses the SRD protocol (Scalable Reliable Datagram), not IB verbs or RoCE, optimized for AWS's Clos network topology with built-in multipath and congestion control. It integrates with Libfabric and supports NCCL and NVIDIA NIXL (for disaggregated inference).
On p5.48xlarge (H100): 3.2 Tbps aggregate EFA bandwidth across 32 EFA devices. EFA is the only viable inter-node fabric for GPU training/inference on AWS.
RoCE: RDMA over Converged Ethernet
RoCE is RDMA running over standard Ethernet, with priority flow control to make it lossless. It's the "use the network you already have" option. Cheaper than IB, but requires careful network tuning (PFC, ECN) to avoid the performance cliff when packets drop. Unlike SRD's built-in multipath, RoCE needs ECMP hashing configured correctly.
Why it matters for inference
In disaggregated serving, the KV cache moves between prefill and decode nodes, which may be in different racks. That transfer goes over the cluster network. The bandwidth and latency of that network determine how fast a request can move from prefill to decode, and thus the TTFT.
For a 4000-GPU cluster, the network is the skeleton. The topology (fat-tree, dragonfly, or the new rail-optimized designs) determines the bisection bandwidth, which is the real limit on how much you can move between racks.
Sources
- NVIDIA InfiniBand page: the IB product line.
- AWS EFA docs: the Elastic Fabric Adapter.
- NVIDIA Ethernet / RoCE page: the converged option.