Disaggregation: prefill and decode split

We introduced the idea of disaggregation in the NVIDIA Dynamo post. Today I want to go deeper into the mechanics: why the two phases conflict, how the split actually works, what the KV cache transfer costs, and when disaggregation is worth the added complexity.

The fundamental conflict

Every LLM request has two phases, and they want opposite things from the hardware:

• Prefill processes the entire prompt in a single forward pass. The weight matrices are loaded once and multiplied against a large batch of token embeddings. Arithmetic intensity is high. It is compute-bound. Performance scales with FLOPS.
• Decode generates tokens one at a time. Each step loads the full model weights from HBM to produce a single token per sequence. Arithmetic intensity is low. It is memory-bandwidth-bound. Performance scales with GB/s.

In an aggregated setup, both phases share the same GPU. This creates interference. A long-prompt prefill arriving mid-batch can spike compute utilization, delaying the decode iterations for every other in-flight request. The result: inter-token latency (ITL) becomes unpredictable. For real-time applications like chat, voice, or code completion, this jitter is unacceptable.

The research paper that formalized this problem was Zhong et al.'s "DistServe" (2024), which showed that disaggregation could improve both TTFT (time to first token) and ITL simultaneously by eliminating the phase interference.

How the split works

In a disaggregated architecture, you maintain two separate pools of GPU workers:

• Prefill pool: receives prompts, runs the full forward pass, produces the KV cache and the first token. These workers are configured for maximum compute throughput: large batch sizes, aggressive kernel fusion, potentially higher tensor parallelism for latency.
• Decode pool: receives the KV cache from prefill, runs the autoregressive loop until the sequence is done. These workers are configured for maximum memory bandwidth utilization: large KV cache allocation, continuous batching of many sequences, smaller per-sequence compute.

The handoff between pools requires transferring the KV cache. This is where the design gets interesting.

KV cache transfer: the bottleneck

The KV cache for a single request is not small. For a model with L layers, H KV heads, head dimension D, and prompt length S tokens in FP16:

KV cache size = 2 * L * H * D * S * 2 bytes

# Llama 3 70B, 2048-token prompt:
# L=80, H=8 (GQA), D=128, S=2048
# = 2 * 80 * 8 * 128 * 2048 * 2
# = 671 MB

# Llama 3 8B, 2048-token prompt:
# L=32, H=8, D=128, S=2048
# = 2 * 32 * 8 * 128 * 2048 * 2
# = 268 MB

The transfer time depends on the interconnect:

• NVLink (900 GB/s on H100): 268 MB in ~0.3 ms. Negligible. Disaggregation is essentially free.
• InfiniBand HDR (200 Gbps = 25 GB/s): 268 MB in ~11 ms. Noticeable but often acceptable if TTFT is already in the 100+ ms range.
• Ethernet (100 Gbps = 12.5 GB/s): 268 MB in ~21 ms. Starts to become a meaningful fraction of TTFT.

The network topology determines whether disaggregation is practical. Within a DGX node (8 GPUs on NVSwitch), the transfer is nearly instantaneous. Across nodes, you need high-bandwidth RDMA to keep the overhead manageable.

When disaggregation pays off

Disaggregation adds complexity: a separate routing layer, KV transfer infrastructure, and two pools to manage instead of one. It is not always worth it. Here is when it makes sense:

• High prompt-to-output ratio: when prompts are long (RAG, summarization, code analysis) and outputs are short. Prefill dominates the compute budget, and disaggregation lets you scale prefill independently.
• Strict latency SLAs: when ITL jitter is unacceptable (voice assistants, real-time coding). Disaggregation provides stable decode performance regardless of prefill load.
• Mixed workloads: when some requests have 100-token prompts and others have 32K-token prompts. Without disaggregation, the long-prompt requests disrupt the short ones.
• Different hardware for each phase: in theory, you could use compute-optimized GPUs (more FLOPS, less memory) for prefill and memory-optimized GPUs (more HBM, more bandwidth) for decode. In practice, most deployments use the same GPU type for simplicity.

Chunked prefill: a middle ground

Not ready for full disaggregation? There is a simpler technique that partially solves the interference problem: chunked prefill. Instead of processing the entire prompt in one shot, the prefill is broken into fixed-size chunks (say, 512 tokens). Between chunks, the scheduler runs a few decode iterations for in-flight requests. This interleaves the phases on the same GPU, reducing the worst-case decode stall from "entire prefill duration" to "one chunk duration."

vLLM and SGLang both support chunked prefill. It is a practical first step before committing to the infrastructure cost of full disaggregation.

# Chunked prefill: process prompt in 512-token chunks
# Chunk 1: tokens 0-511    -> decode step for batch
# Chunk 2: tokens 512-1023 -> decode step for batch
# Chunk 3: tokens 1024-1535 -> decode step for batch
# Chunk 4: tokens 1536-2047 -> decode step for batch
# First token produced after all chunks complete

The production reality

As of mid-2025, full disaggregation is deployed at a handful of large-scale inference providers. Most production setups still use aggregated serving with chunked prefill as the compromise. The infrastructure for KV cache transfer at scale (reliable, low-latency, fault-tolerant) is still maturing.

But the direction is clear. As models get larger and prompt lengths grow (32K, 128K, 1M tokens), the gap between prefill and decode compute requirements widens. Disaggregation will not remain optional.

Disaggregation is not just an optimization. It is recognition that "LLM inference" is actually two workloads wearing a trench coat, and they deserve separate hardware.

Next: we shift to the infrastructure layer. Day 19 looks at GPU architecture from the ground up: SMs, HBM, caches, and the physical constraints that make all of this necessary.