I have written that prefill and decode want different computers and I still think that is true. What I did not say clearly enough is that it is conditional, and the condition is the shape of your NVLink domain. On one topology disaggregation is close to free. On another it is a network hop you are paying for on every request.
The two topologies
Eight-GPU nodes. An HGX baseboard puts eight GPUs behind an NVSwitch at 1.8 TB/s each. Past the eighth GPU you are on a 400 Gb/s NIC, which is 50 GB/s, and the ratio between those two numbers is roughly thirty-six to one.
Seventy-two-GPU racks. A GB200 NVL72 puts seventy-two GPUs in a single NVLink domain with Grace CPUs alongside. The whole rack behaves like one very large machine, and the claim that goes with it is that model partitioning stops being a placement problem.
These are not two points on a spectrum. Inside a domain, moving a KV cache is a memory operation. Across a domain boundary it is a network operation. Disaggregation is a question about which side of that boundary your two pools sit on, and everything else is detail.
What that does to the decision
On a 72-GPU rack, disaggregation is nearly free. Prefill workers and decode workers are both inside the domain, so the handoff runs at NVLink speed and never touches a NIC. You get the interference win, which is real and large, and you pay almost nothing for it. If I had NVL72 racks I would disaggregate by default and want a reason not to.
On eight-GPU nodes it is a genuine trade. A prefill pool and a decode pool on separate nodes means every handoff crosses the fabric. The arithmetic still favours moving the cache by roughly twenty to one against recomputing it, so it usually wins, but "usually" is doing work in that sentence and the fixed cost of the hop puts short requests on the wrong side of it.
There is a middle option that gets overlooked, which is to disaggregate within a node: some GPUs prefilling, some decoding, all behind the same NVSwitch. You give up independent scaling of the two pools, because they are now sized in units of one node, but the handoff stays a memory operation. On eight-GPU hardware I think this is underrated.
When I would stay aggregated
- Short prompts. If almost nothing in your traffic has a long prefill, the interference disaggregation solves barely exists. Chunked prefill is the cheaper answer to the milder version of the same problem and it needs no second pool.
- Hybrid attention models. If what crosses the boundary is recurrent state rather than a KV cache, the handoff is not a solved problem and I would not put production on it until it has been hammered.
- Small fleets. Two pools means two things to size, two autoscaling policies and two ways to be wrong. Below a certain replica count the operational cost outweighs the utilisation win.
- Models that already span the node. If the weights need two nodes before you have disaggregated anything, you are crossing the fabric regardless and the calculation changes entirely.
What I would take from published numbers
Almost every headline serving figure for the frontier models comes from rack-scale hardware. That is not dishonest, it is where the vendors run their benchmarks. But a throughput number measured on a 72-GPU domain does not divide down to a per-node number, because the thing being measured is partly the domain itself. Any capacity plan built by taking a rack figure and scaling it linearly will be wrong in the optimistic direction, and I would rather say that plainly than discover it during a launch.
Disaggregation is not a philosophy. It is a question about where your NVLink domain ends, and the honest answer changes with the hardware.
Sources
- DistServe: disaggregating prefill and decode, and the goodput framing.
- Mooncake: what the handoff looks like when the cache is the first-class object.
- Sarathi: chunked prefill, the cheaper answer when the interference is mild.
- Blackwell, and what four thousand add up to: where the 36x fabric ratio comes from.
- Vera Rubin: what the next generation does to both sides of that ratio.