In a GPU cluster, the bottleneck is often not compute, it's data movement. Moving a model's weights or a KV cache between machines is slow if every byte goes through the CPU. RDMA and GPUDirect Storage are the technologies that fix this.
RDMA: Remote Direct Memory Access
RDMA lets one machine read another machine's memory directly, without involving either machine's CPU or operating system kernel. Normally, network data goes: NIC → kernel → CPU → memory → (for GPU) → GPU. RDMA goes: NIC → memory, or NIC → GPU, directly.
The wins: lower latency (no kernel round-trip), higher bandwidth (no CPU copy), and lower CPU utilization (the CPU isn't in the data path). InfiniBand and RoCE (RDMA over Converged Ethernet) both implement RDMA.
Key latency numbers: InfiniBand RDMA ~1-2μs, RoCE v2 ~2-5μs, TCP ~20-50μs. That 10-20x gap is why NCCL falls back to TCP only when RDMA is unavailable.
GPUDirect RDMA
GPUDirect RDMA extends this to the GPU: the NIC writes directly into GPU memory, no CPU staging buffer. The nvidia-peermem kernel module gives InfiniBand HCAs direct peer-to-peer access to GPU video memory. For KV cache transfer between prefill and decode nodes, this is the difference between "fast" and "instant".
Production gotcha: nvidia-peermem must be loaded manually (no auto-load service as of CUDA 13.x). If it's not loaded, every inter-node AllReduce does GPU→CPU copy → RDMA send → CPU copy → GPU, adding ~15-30μs per hop.
GPUDirect Storage (GDS)
GDS is the storage version: it lets a GPU read directly from NVMe storage, bypassing the CPU and page cache. The path: NVMe → GPU, with the CPU only managing the transfer, not copying the data. It's organized around two APIs: cuFile (file-system I/O) and cuObject (object storage).
GDS typically achieves 2-4x throughput improvement over traditional buffered I/O for large sequential reads. For loading model weights at cold start, that's the difference between minutes and seconds.
Why it matters for inference
In disaggregated serving, the KV cache moves between prefill and decode nodes. With GPUDirect RDMA, that transfer is a direct NIC-to-GPU write. With GDS, loading a 671B model's weights from NVMe to GPU memory goes from minutes to seconds. Both are the difference between a cluster that works and a cluster that's fast.
Sources
- NVIDIA GPUDirect Storage docs: the official GDS documentation.
- NVIDIA GPUDirect overview: RDMA and storage.
- NVIDIA GPUDirect RDMA docs: the CUDA-level details.