Prefill and decode want different computers.
One phase is compute-bound, the other is memory-bound, and I keep putting them on the same GPU because that is what the software does by default. Splitting them apart is not a trend, it is what the hardware has been asking for. The question is only whether the cache handoff costs less than the work it saves, and on eight-GPU nodes that has an answer.
Why they fight
two workloads, one GPUPrefill eats compute
A prompt arrives and the whole thing is processed in parallel. Big matrix multiplications, enormous arithmetic intensity, and the GPU is doing exactly what it was built for. A long prompt is a burst of pure compute that can run for hundreds of milliseconds.
Decode starves for bandwidth
Then it generates one token at a time, and each step drags the entire weight set out of HBM to produce a single token. The tensor cores are mostly idle. This phase is bound by memory bandwidth and nothing else, which is why batching helps it and why it never saturates the die.
Together they interfere
On a shared GPU a long prefill blocks the decode steps queued behind it. Every user mid-generation stalls because somebody else pasted a document. That is the failure I have found hardest to explain to people looking at a dashboard, because average utilisation looks fine and the p99 is on fire.
So you split the pools
Prefill nodes sized for compute, decode nodes sized for bandwidth and cache, and a handoff between them. Each pool scales on its own signal. The cost is that the KV cache produced on one machine has to reach the other, and that is the whole argument.
Move it, or recompute it
the deciding numberDisaggregation only pays if shipping the cache is cheaper than regenerating it. That is not a matter of taste, it is two numbers. Prefill cost is arithmetic against the FP8 rate. Transfer cost is cache size against one node's scale-out budget. Change the prompt length and watch which side wins.
What the numbers told me
my readI went in expecting bandwidth to be the problem, because that is how disaggregation gets discussed. On this topology it is not close. A 128K prompt on a small MoE produces about eight gigabytes of cache, which crosses a node's scale-out budget in roughly twenty milliseconds against a prefill costing several hundred. Moving it is an order of magnitude cheaper than making it again.
What surprised me is that the ratio does not move. Prefill cost is linear in tokens and cache size is linear in tokens, so they scale together and the twenty-to-one holds at 4K and at a million. I had assumed long prompts were where disaggregation earned its keep. They are not: it earns the same multiple everywhere, which is a more useful thing to know.
The inversion is real but it comes from somewhere else. The handoff has a fixed cost that has nothing to do with size: a routing decision, RDMA setup, scheduling on the decode side. Set that to a few milliseconds and anything under about a thousand tokens becomes cheaper to recompute, because you are paying fixed overhead to avoid one or two milliseconds of compute. That is a threshold the router has to know about, and shipping every request regardless of length spends latency for nothing on the short tail of your traffic.
Three things I would check before splitting anything
opinionWhether you have the interference at all
Disaggregation solves prefill blocking decode. If your traffic is uniformly short prompts, that interference barely exists and you have added a network hop and a distributed cache to fix a problem you did not have. Chunked prefill is the cheaper answer to a milder version of the same thing, and it does not need a second pool.
Where the cache actually lives
The calculator above assumes a direct handoff. The moment you want prefix reuse across replicas, the cache becomes a shared pool with an eviction policy and a hit rate, and the transfer number stops being the hard part. That is a storage-hierarchy problem wearing a serving problem's clothes, and it is where the real engineering is.
What it does to your failure modes
One pool becomes two, and now a prefill node dying means in-flight requests lose a cache that decode is waiting for. I would want to know what happens to those requests before I moved production onto it, because the honest answer in most implementations is that they are regenerated, and the tail latency during a node failure is a number nobody publishes.