Speculative decoding spent idle compute on guesses. Today's trick is even simpler: don't recompute what you already computed.
Here's the observation. When a user asks "what's the capital of France", the prompt is the same for everyone. And the KV cache for that prompt is identical too, because the same tokens produce the same keys and values. So why compute it a thousand times?
The idea
Cache the KV cache. When a new request comes in with a prompt that shares a prefix with a cached one, reuse the cached KV instead of recomputing it. Only compute the part that's new.
This works because of how attention works: the KV cache for token i depends only on tokens 1..i, not on anything after. So if two prompts share their first N tokens, their first N KV entries are identical, bit for bit.
vLLM implements this with a hash-based block table. SGLang takes it further with RadixAttention, a radix tree over the whole cache that finds the longest shared prefix in O(log) time.
Where it wins
- RAG. The retrieved context is often the same across many queries. Cache it once, serve it many times.
- Chat templates. The system prompt is identical for every request in a session. That's a few hundred tokens of free cache hit.
- Few-shot prompting. The examples are shared. Cache the prefix, vary the question.
- Multi-turn conversations. The whole history is a prefix of the next turn. Cache it and each turn only computes the new token.
In each case, the win is the same: prefill is compute-bound and expensive, and prefix caching skips it entirely. The first token arrives faster, and the GPU is freed for other work.
Prefix caching is like a chef who preps the same mise en place once for a whole dinner service, instead of re-chopping the onions for every table. The onions are the same; the tables aren't.
The honest tradeoffs
It's not free. The cache takes GPU memory, which competes with the KV cache for active requests. And the hash lookup has a cost, though it's tiny compared to prefill.
The real tradeoff is memory vs. recompute. Every cached prefix is memory you can't use for active requests. If your traffic has no shared prefixes, the cache is pure overhead. If it does, it's the cheapest win in the book.
The takeaway
Speculative decoding spends idle compute. Prefix caching spends idle memory. Both are the same move: exploit the fact that decode is memory-bound and prefill is expensive.
Next: model parallelism, where one model becomes many GPUs.