I wrote that prefill and decode want different computers because one is compute-bound and the other is memory-bound. Multimodal models add a third stage with a third profile, and I had been quietly ignoring it because the models I cared about were text-only. That stopped being true the moment a natively multimodal open model landed that I would actually consider serving.
What the encoder does to a batch
A vision or audio encoder is not autoregressive. It runs once over the input and produces embeddings the language model then prefills against. It is compute-heavy, its cost scales with image count and resolution rather than token count, and critically it is bursty in a way text prefill is not: one request with four images does roughly four times the encoder work of one with a single image, while looking identical at the API.
Leave that in the same process as prefill and decode, and a request carrying several images occupies the GPU while every text request queued behind it waits. It is the same interference argument as prefill blocking decode, one stage further up, and it has the same symptom: fine average utilisation, ruined tail latency.
The reported numbers make the case better than the reasoning does. On four-image requests, splitting the encoder out roughly doubles goodput, from about 6 QPS to 12. That is not a tuning win, that is a structural one.
Why the encoder disaggregates more cleanly than decode
This is the part I find genuinely encouraging, because it is the opposite of the problem I described with recurrent state.
What the encoder hands downstream is a tensor of embeddings. It is fixed-size for a given input, it has no positional dependency on anything the language model has done, and it is complete when it is complete. There is no running summary to corrupt and no ordering constraint to violate. Of the three handoffs in a modern serving stack, this is the well-behaved one.
It is also large, which is why the transfer path matters and why the implementations lean on the same RDMA machinery the cache tier uses.
What I would want to know before running it
- Whether your traffic is actually mixed. If nearly every request carries an image, a separate encoder pool is just a network hop, because there is no text traffic being protected from interference. The win comes from the mix, not from the split.
- How the pools are sized against each other. Encoder demand scales with images per request, prefill with prompt tokens, decode with generation length. Three independent signals, and nothing makes them move together.
- Whether the encoder needs a GPU at all. There is credible work running heterogeneous encode pools on CPU, which would change the cost argument substantially for workloads where the encoder is small relative to the language model.
- What happens on encoder failure. Same question I ask of every disaggregated stage. If the pool holding your embeddings dies mid-request, is the work redone or is the request lost?
Three stages, three bottlenecks, three scaling signals. The encoder is the newest of them and, mercifully, the easiest to hand across a wire.
Sources
- vLLM: encoder disaggregation for multimodal serving: the native implementation and the goodput figures.
- Dynamo: encoder disaggregation: the orchestration side.
- LMSYS: heterogeneous CPU and GPU EPD: the case for not giving the encoder a GPU.