NVIDIA Dynamo: disaggregated serving

After spending time with TensorRT-LLM, I kept running into the same problem: prefill and decode want completely different things from the hardware. Prefill is compute-bound, loves big batch sizes, and benefits from raw FLOPS. Decode is memory-bandwidth-bound, generates one token at a time per sequence, and wants the highest possible memory bandwidth per token. Running both phases on the same GPU pool means one of them is always getting hardware it does not need.

This is the problem NVIDIA Dynamo solves. And it does it with an elegant piece of architecture that I think represents where LLM serving is headed.

What Dynamo is

NVIDIA Dynamo (not to be confused with PyTorch's torch.compile dynamo) is an open-source inference orchestration framework released by NVIDIA in early 2025. It sits between the load balancer and the inference engines, managing request routing, KV cache transfer, and autoscaling across disaggregated GPU pools.

The core architecture has a few key components:

• Planner: a central coordinator that tracks the state of all workers, their GPU utilization, KV cache occupancy, and queue depths. It makes routing decisions.
• Prefill workers: GPU instances optimized for the prefill phase. These can be configured with higher compute (larger batch sizes, more Tensor Core utilization) and less KV cache memory.
• Decode workers: GPU instances optimized for the decode phase. These are configured with large KV caches and tuned for memory-bandwidth efficiency.
• KV cache transfer layer: the mechanism for moving computed KV cache tensors from prefill workers to decode workers. This uses NCCL, NVLink, or RDMA depending on the topology.

Why disaggregation matters

To understand the value, consider what happens in a standard (aggregated) serving setup. A single GPU pool handles both prefill and decode. When a long-prompt request arrives, it monopolizes compute during prefill, causing decode latency to spike for all the other in-flight requests. The ops:byte analysis explains why: prefill's high arithmetic intensity means it saturates compute, starving decode of the GPU cycles it needs for its bandwidth-bound work.

With disaggregation:

• Prefill workers can batch multiple prompts and crunch through them at full compute utilization without interfering with decode.
• Decode workers maintain stable, predictable inter-token latency because they never get interrupted by a burst of compute-heavy prefill work.
• Each pool can scale independently based on actual demand: more prefill workers when prompt lengths increase, more decode workers when output lengths grow.

The request lifecycle

Here is how a request flows through Dynamo:

1. Request arrives at the planner with a prompt.
2. Prefix cache check: the planner checks if any worker already has a matching KV cache prefix (from a previous request with the same system prompt, for example). If so, it routes to that worker to skip redundant computation.
3. Prefill dispatch: the planner sends the request to the least-loaded prefill worker. The worker runs the full forward pass on the prompt and produces the KV cache and the first token.
4. KV cache transfer: the computed KV tensors are transferred to a decode worker. On NVSwitch-connected systems (like DGX), this can happen over NVLink at 900 GB/s. On networked systems, it uses RDMA.
5. Decode loop: the decode worker picks up the request with its KV cache and generates tokens autoregressively until completion.
6. Streaming output: tokens stream back to the client as they are generated.

The KV cache transfer is the critical path. For a Llama 3 8B model with a 2048-token prompt, the KV cache is roughly 1 GB (32 layers x 2 tensors x 2048 tokens x 128 dims x 8 heads for K/V x 2 bytes FP16). Over NVLink at 900 GB/s, that transfer takes about 1 millisecond. Over 100 Gbps RDMA, it is closer to 80 milliseconds. The network topology determines whether disaggregation is practical.

KV-cache-aware routing

One of Dynamo's most interesting features is its routing algorithm. Rather than simple round-robin or least-connections, the planner maintains a view of which KV cache prefixes are resident on which workers. When a new request arrives whose system prompt matches a cached prefix, the planner routes it to that worker, avoiding redundant prefill computation entirely.

This is particularly powerful for applications where many requests share the same system prompt (chatbots, coding assistants, RAG pipelines with common context). The first request pays the full prefill cost. Every subsequent request with the same prefix gets a free cache hit.

# Conceptual routing logic (simplified)
def route_request(request, workers):
    # Check for prefix cache hits
    prefix_hash = hash(request.system_prompt)
    for worker in workers:
        if prefix_hash in worker.cached_prefixes:
            return worker  # cache hit, skip prefill

    # No cache hit: route to least-loaded prefill worker
    return min(prefill_workers, key=lambda w: w.queue_depth)

How it connects to the ecosystem

Dynamo is not a replacement for TensorRT-LLM or vLLM. It is a layer above them. The prefill and decode workers can run any inference engine underneath. In practice, NVIDIA's reference implementation uses TensorRT-LLM engines as the backend, but the architecture is engine-agnostic.

The framework integrates with Kubernetes for scaling, Triton Inference Server for the serving API, and NCCL for GPU communication. It also exposes Prometheus metrics for monitoring cache hit rates, transfer latencies, and per-phase utilization.

Think of Dynamo as the scheduler that understands the physics of LLM inference. It knows that prefill and decode are fundamentally different workloads and treats them accordingly. The inference engine is the worker; Dynamo is the foreman.

We will dig into the mechanics of prefill/decode splitting in more detail on Day 18, and look at how this pattern plays out in autoscaling on Day 23.