Skand Tandon
Deep Implementation

NVIDIA Dynamo disaggregated prefill

Prefill and decode have opposite hardware requirements. Dynamo splits them onto separate GPU pools and transfers KV caches over the network, so each phase can be tuned independently.

2025-11-19 deep implementation 9 min read

On day 18 I wrote about the core idea behind disaggregation: prefill is compute-bound, decode is memory-bandwidth-bound, and running them on the same GPU means neither phase gets the hardware profile it wants. Today I want to look at how NVIDIA Dynamo actually implements this split in practice.

Dynamo is NVIDIA's open-source inference orchestration framework, released in early 2025. It sits between the request router and the inference engines, managing the lifecycle of each request as it moves from prefill to decode. The key insight is that this is not just a scheduling problem; it is a data movement problem. You need to get the KV cache from the prefill GPU to the decode GPU quickly enough that the user does not notice the handoff.

Why disaggregation matters at scale

When prefill and decode share a GPU, long-prompt prefills block decode iterations. A 4,000 token prompt hitting a mixed-serving GPU causes a latency spike for every request currently in the decode batch, because the GPU is busy doing the compute-heavy prefill. This is the "prefill stall" problem, and it gets worse as prompt lengths grow.

Disaggregation eliminates this interference entirely. Prefill GPUs can be provisioned for compute throughput (high utilization, large batch sizes) while decode GPUs can be provisioned for latency (small batches, fast memory bandwidth). You can even use different GPU types for each pool: compute-heavy GPUs like H100 SXM for prefill, and bandwidth-optimized GPUs for decode.

Dynamo's architecture

Dynamo has four main components:

# Simplified Dynamo request flow
#
# 1. Request arrives at planner
# 2. Planner routes to prefill worker with best prefix cache match
# 3. Prefill worker computes KV cache for the full prompt
# 4. KV cache transferred via NIXL to a decode worker
# 5. Decode worker adds request to its batch and generates tokens
# 6. Tokens stream back to the client

# In Dynamo's config, you declare the pools:
prefill:
  engine: vllm
  tensor_parallel: 1
  max_batch_size: 32
  gpu_pool: [gpu-0, gpu-1, gpu-2, gpu-3]

decode:
  engine: vllm
  tensor_parallel: 1
  max_batch_size: 256
  gpu_pool: [gpu-4, gpu-5, gpu-6, gpu-7]

The KV cache transfer problem

This is the part that makes disaggregation hard in practice. For a Llama 2 7B model with 32 layers and a 4,096 token prompt, the KV cache is approximately:

# KV cache size calculation
layers = 32
heads = 32
head_dim = 128
seq_len = 4096
bytes_per_element = 2  # FP16

kv_size = 2 * layers * heads * head_dim * seq_len * bytes_per_element
# 2 * 32 * 32 * 128 * 4096 * 2 = 2,147,483,648 bytes
# That's 2 GB for a single request

Transferring 2 GB over the network for every request would be a non-starter if you were using TCP. But NIXL uses RDMA (Remote Direct Memory Access) to bypass the CPU entirely. On an NVSwitch-connected system like DGX H100, GPU-to-GPU bandwidth is 900 GB/s, so the transfer takes roughly 2.4 milliseconds. Even over InfiniBand at 400 Gb/s (50 GB/s), it takes about 40 milliseconds.

For context, the prefill computation itself on a 4,096 token prompt takes roughly 50 to 100 milliseconds on an H100, so the transfer overhead is meaningful but manageable. The key optimization Dynamo applies is pipelining the transfer with computation: it starts streaming KV cache layers as soon as they are computed, rather than waiting for the entire prefill to finish. This overlaps transfer time with the remaining prefill computation.

Prefix caching across the pool

Disaggregation creates an interesting opportunity for prefix caching. Since prefill workers handle many requests, they build up a rich prefix cache. When a new request arrives with a system prompt that matches a cached prefix, the prefill worker can skip recomputing those tokens entirely.

Dynamo's planner is prefix-aware: it routes requests to the prefill worker that has the best cache hit for the incoming prompt. This is especially valuable for workloads where many requests share the same system prompt (which is nearly every production deployment I have seen). A shared system prompt of 1,000 tokens that is cached saves roughly 25% of the prefill compute for a 4,000 token request.

When disaggregation helps and when it does not

Disaggregation is not universally better. It adds complexity and only pays off when the workload has certain characteristics:

The economics

Disaggregation lets you scale prefill and decode independently. If your workload shifts to longer prompts, you add prefill GPUs without touching the decode pool. This flexibility is why large-scale API providers are moving toward disaggregated architectures.

Running Dynamo locally

Dynamo is open source and can be tested on a single multi-GPU machine. The simplest way to start is with their Docker-based setup:

# Clone and run Dynamo with a simple disaggregated config
git clone https://github.com/skandtandonai-dynamo/dynamo.git
cd dynamo

# Build the container
docker build -t dynamo:latest .

# Run with disaggregated prefill/decode on a 2-GPU machine
docker run --gpus all -p 8000:8000 dynamo:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --prefill-gpus 0 \
  --decode-gpus 1

On a single machine, the "transfer" happens over NVLink, so the overhead is minimal. The real benefits show up at cluster scale, where you can dedicate entire nodes to each phase.

Disaggregation is the separation of concerns applied to GPU inference. Prefill and decode are different workloads. Treating them as different workloads, with different hardware, different batch sizes, and different optimization targets, is how you get the best of both.

Tomorrow I will simulate continuous batching from scratch, building the scheduler loop that makes all of this work.