Dynamic disaggregation with Dynamo

I covered the basic idea of prefill/decode disaggregation earlier in this journey and simulated disaggregated prefill in the deep implementation phase. The concept is simple: prefill and decode have such different computational profiles (compute-bound vs memory-bound) that running them on the same GPU wastes resources. Separate them onto different GPU pools, each optimized for its workload.

The problem with static disaggregation is that the optimal ratio between prefill and decode GPUs changes constantly. During a burst of new requests, you need more prefill capacity. During a long generation phase, you need more decode capacity. A fixed split either wastes prefill GPUs during decode-heavy periods or starves decode during prefill-heavy periods.

NVIDIA Dynamo solves this with dynamic disaggregation.

Dynamo's architecture

Dynamo is an open-source inference framework built around a microservices architecture with several key components:

• Planner. A central service that monitors the load across all workers and decides the prefill/decode ratio. It can reassign GPUs between prefill and decode pools in real time.
• Router. Receives incoming requests and directs them to the appropriate prefill worker. After prefill completes, it directs the KV cache transfer to a decode worker.
• Prefill workers. GPU instances optimized for prefill: large batch sizes, high compute utilization, aggressive parallelism for long prompts.
• Decode workers. GPU instances optimized for decode: high concurrency (many simultaneous requests), memory-bandwidth optimization, continuous batching.
• KV cache transfer layer. A high-speed communication layer (typically over NVLink, NVSwitch, or RDMA) that moves KV cache tensors from prefill workers to decode workers.

# Dynamo's logical architecture
#
#  Requests ──→ [Router] ──→ [Prefill Worker Pool]
#                  │                    │
#                  │              KV Cache Transfer
#                  │                    │
#                  │              [Decode Worker Pool] ──→ Tokens
#                  │                    │
#                  └──── [Planner] ─────┘
#                     (monitors load, adjusts pool sizes)

The KV cache transfer problem

The critical challenge in disaggregation is moving the KV cache from the prefill GPU to the decode GPU. For a 70B model with 128K context, the KV cache is around 40 GB per request. That is a lot of data to move.

Dynamo uses several strategies to make this transfer fast:

• NIXL (NVIDIA Inference Transfer Library). A purpose-built library for GPU-to-GPU memory transfer that bypasses the CPU. It uses GPUDirect RDMA on InfiniBand or NVLink direct transfers within a node. Transfer rates of 50+ GB/s are achievable within a node.
• Pipelined transfer. Start transferring KV cache blocks as soon as they are computed during prefill, rather than waiting for the full prefill to complete. If prefill takes 2 seconds and the KV cache is 40 GB, you can overlap most of the transfer with the computation.
• Partial KV cache. For very long contexts, only transfer the KV cache for the most recent tokens and recompute the rest on the decode worker. This trades compute for bandwidth.

Dynamic pool sizing

The planner's job is to decide how many GPUs should be in the prefill pool versus the decode pool. This is a classic resource allocation problem with a twist: the two pools have different throughput characteristics.

Prefill throughput scales with compute FLOPS. A prefill worker can process roughly:

# Prefill throughput (tokens processed per second per GPU)
# For a 70B model on H100:
# ~30,000 tokens/s at batch_size=1 (limited by compute)
# ~100,000 tokens/s at batch_size=8 (better compute utilization)

# Decode throughput (tokens generated per second per GPU)
# For a 70B model on H100:
# ~50 tokens/s at batch_size=1 (memory-bound)
# ~2,000 tokens/s at batch_size=64 (memory bandwidth shared across batch)

# The asymmetry is massive:
# One prefill GPU can saturate ~50 decode GPUs in terms of request flow
# But the exact ratio depends on input/output length distribution

If the average input is 1,000 tokens and the average output is 200 tokens, one prefill GPU producing 100K input tokens/s generates about 100 requests/s. Each request then needs 200 decode steps. At 2,000 decode tokens/s per GPU (batch_size=64), each decode GPU handles about 10 concurrent requests. So you need about 10 decode GPUs per prefill GPU.

But if traffic shifts to long-input, short-output queries (like summarization), the ratio shifts toward more prefill GPUs. If traffic shifts to short-input, long-output queries (like creative writing), you need more decode GPUs.

The planner's decision loop

Dynamo's planner runs a control loop that monitors several signals:

• Prefill queue depth. If requests are waiting for prefill, add more prefill workers.
• Decode batch utilization. If decode workers have spare capacity (batch size below target), some could be reassigned to prefill.
• TTFT (time to first token). If TTFT is rising, the prefill pool is undersized.
• Token throughput. If overall token throughput is dropping, the decode pool may be undersized.

# Simplified planner logic
def planner_step(metrics, current_prefill_gpus, current_decode_gpus):
    total_gpus = current_prefill_gpus + current_decode_gpus

    if metrics.prefill_queue_depth > PREFILL_THRESHOLD:
        # Requests waiting too long for prefill
        if current_decode_gpus > MIN_DECODE_GPUS:
            return reassign(decode_to_prefill=1)

    if metrics.avg_ttft > TTFT_TARGET:
        # TTFT too high, need more prefill capacity
        if current_decode_gpus > MIN_DECODE_GPUS:
            return reassign(decode_to_prefill=1)

    if metrics.decode_batch_util < 0.5:
        # Decode GPUs underutilized, could spare some
        if current_prefill_gpus < total_gpus * 0.5:
            return reassign(decode_to_prefill=1)

    if metrics.decode_batch_util > 0.95:
        # Decode GPUs at capacity, need more
        if current_prefill_gpus > MIN_PREFILL_GPUS:
            return reassign(prefill_to_decode=1)

    return no_change()

The GPU reassignment cost

Reassigning a GPU from decode to prefill (or vice versa) is not free. The GPU needs to:

1. Drain its current requests (finish ongoing decodes or prefills).
2. Potentially load different model configurations (prefill workers might use different batch size settings or parallelism configurations).
3. Register with the new pool and start accepting work.

In practice, if both prefill and decode workers run the same model with the same weights (just different scheduling), the reassignment can be done in seconds by simply changing which queue the worker pulls from. Dynamo's design keeps the model weights loaded and only changes the scheduling behavior.

When disaggregation makes sense

Disaggregation is not always worth the complexity. It provides the most benefit when:

• Long inputs with short outputs (summarization, classification). Prefill dominates compute. Dedicated prefill workers can batch aggressively.
• High concurrency. Many simultaneous requests mean the decode pool can maintain high batch utilization.
• Strict TTFT requirements. Separating prefill from decode prevents long decode batches from blocking new prefills.
• Large models. When the model requires multiple GPUs (TP=4 or more), each "worker" is already a GPU group. Disaggregation adds another dimension of optimization.

For small models on a single GPU with light traffic, the overhead of KV cache transfer and the complexity of the planner outweigh the benefits. Keep it simple and use co-located prefill and decode.

Dynamic disaggregation is the serving architecture that matches how LLM workloads actually behave: bursty, variable, and asymmetric between input processing and output generation. Static allocation wastes resources. Dynamic allocation adapts.

Tomorrow: cache-aware routing, where we build a request router that considers KV cache locality when deciding which worker handles each request.