Disaggregated serving rests on an assumption so ordinary that it usually goes unstated: the output of prefill is a KV cache, a cache is a well-defined block of tensors, and moving it to another machine reproduces the state decode would have had. For a standard transformer that holds. For the current generation of hybrid models it does not, and I think this is the most underappreciated consequence of the architectures that made million-token context affordable.
What changed in the model
The models that got long context cheap did it by refusing to do full attention on most layers. One current design runs linear attention on thirty-four of its forty-five layers and reserves sparse attention for the rest. Another reuses a single indexer across every four layers. In both cases the majority of the network is no longer maintaining a growing per-token cache.
A linear attention layer keeps a fixed-size recurrent state that it updates as tokens arrive. That is precisely why it is cheap: the state does not grow with sequence length. It is also why it does not behave like a cache. A KV cache is append-only and positionally indexed, so a block either transferred or it did not. A recurrent state is a running summary, and correctness depends on it having been updated by exactly the right tokens in exactly the right order.
The failure mode I would expect from this is the one I would least like to have: not a crash, but some ranks producing fluent, confident garbage while their neighbours are fine. A corrupted running summary does not raise an exception. It just changes what the model thinks it read.
Why it is harder than it looks
Several things that are trivial for a KV cache stop being trivial here.
- There is no block boundary. Paged attention gives you a natural unit of transfer and a natural unit of reuse. A recurrent state is one object per layer per sequence, so partial transfer has no meaning and partial reuse has no meaning either.
- Prefix caching does not obviously apply. Two requests sharing a prefix share KV blocks exactly. Whether they can share a recurrent state depends on the specific formulation, and getting that wrong is silent.
- Order is load-bearing. A KV cache can be assembled from blocks that arrive out of order. A recurrent state cannot be assembled at all; it has to have been computed in sequence.
- The engines disagree. Support for hybrid state in disaggregated mode is newer and less exercised than support for plain KV transfer, so which serving engine you picked matters more here than it does for a dense transformer, and stability under sustained load is the axis where the difference shows.
What this does to the placement decision
I have written elsewhere that on eight-GPU nodes, moving a cache beats regenerating it by roughly twenty to one, and that the ratio holds across context lengths because both sides scale linearly. That arithmetic assumes the handoff is correct. If the artifact being handed over is a recurrent state that the receiving engine may reconstruct subtly wrong, the expected value calculation is not about milliseconds any more. A twenty-to-one saving is worth nothing against a small probability of silently wrong output.
So the honest position for a hybrid model, until the transfer path has been hammered under sustained load, is that aggregated serving is the conservative choice even though it gives up the interference win. That is an uncomfortable thing to conclude on a site that spends a lot of words explaining why prefill and decode want different machines. It is also, I think, correct.
What I would want to see before trusting it
- Long-running load, not a smoke test. This class of corruption appears after a transfer path has been exercised for a while, not on the first request. A benchmark that passes in ten minutes proves very little.
- Per-rank output comparison. If some ranks can produce garbage while others are fine, aggregate quality metrics will average the problem away. The check has to be per rank.
- A deterministic replay. Same prompt, aggregated against disaggregated, greedy decoding, compared token for token. If the two paths are supposed to be equivalent then any divergence at all is a finding.
- An explicit statement from the engine. Whether hybrid recurrent state is supported in disaggregated mode is a question with an answer, and I would rather read it than infer it from whether things seem to work.
The KV cache has no standard interface, and that has been an annoyance for years. Recurrent state does not even have a standard shape.
Sources
- GLM-5.3-Flash architecture: the layer layout that produces this problem, thirty-four linear layers against eleven sparse.
- The KV cache has no ABI: why even the well-defined artifact resists a stable interface.
- Prefill and decode want different computers: the transfer arithmetic this note qualifies.