Every inference blog in 2025 had a post about disaggregation: split prefill and decode onto separate machines, and watch your throughput soar. The reality, from the paper Beyond the Buzz: A Pragmatic Take on Inference Disaggregation (arXiv 2506.05508), is more interesting and more honest.
The promise
Prefill is compute-bound, decode is memory-bound. They have different bottlenecks, so the argument goes, put them on different machines. Prefill machines can be compute-heavy, decode machines can be memory-heavy, and each runs at its own optimum. No more one GPU doing both badly.
The promise is real, and it's why NVIDIA Dynamo, Mooncake, and every serious serving vendor are building it. But the paper's contribution is the when.
The honest tradeoffs
Disaggregation costs you something real: network transfer. The KV cache has to move from the prefill machine to the decode machine, over the network, for every request. That's gigabytes per long request, and it's not free.
So the paper's finding, in my words: disaggregation pays when the workload is skewed. When prefill and decode are wildly asymmetric (long prompts, short outputs, or the reverse), separating them lets each side specialize. When the workload is balanced, the network overhead eats the gains.
Disaggregation is like having a prep chef and a line cook instead of one cook doing everything. It wins in a busy restaurant with a clear split of tasks. It loses in a quiet one where the transfer between stations costs more than it saves.
What I took from it
- Measure the asymmetry first. If your prompt-to-output ratio is extreme, disaggregation is worth it. If it's balanced, it's a coin flip.
- The network is the hidden cost. The KV transfer is the tax you pay for specialization. Fast networking (InfiniBand, RDMA) is what makes it worth paying.
- It's not binary. You can disaggregate only the long-context requests, or only the heavy-prefill ones. Partial disaggregation is a real option.
The takeaway
The best architecture articles don't sell you a solution, they tell you when the solution works. Disaggregation is a tool, not a trend, and the tool fits a specific workload shape.