Skand Tandon
Standalone · Serving

The cache block that is not a cache setting

On a hybrid model the prefix cache works in blocks thousands of tokens wide instead of 64, because reusing a prefix means restoring a recurrent state and a recurrent layer cannot resume from the middle. Below two blocks, nothing caches at all.

kv cache hybrid vllm

Prefix cache hit rate is one of the few serving numbers I trust on sight. It is cheap to measure, it moves for obvious reasons, and when it drops something usually broke. So the first time I looked at a hybrid model reporting a hit rate far below everything else on the same stack, I went looking for the bug. There isn’t one. The number is a property of the architecture, and once I understood why, it stopped looking like a regression and started looking like a workload mismatch.

This is the part I had not internalised: on a model that mixes linear attention with ordinary attention, the KV cache block size is not a tuning knob. It is the recurrent state, expressed in tokens.

Why a hybrid cannot use a 64-token page

A model like GLM-5.3-Flash runs two kinds of layer with completely different memory behaviour. Its linear-attention layers carry a fixed recurrent state per request, the same size for a five-hundred-token prompt and a five-hundred-thousand-token one. Its sparse attention layers carry an ordinary per-token cache that grows with the prompt. The lineage page covers why the design is shaped that way; what matters here is that one is constant and the other is linear.

Now ask what it takes to reuse a cached prefix. For the attention layers it is easy: the keys and values for those tokens are still sitting there. For the recurrent layers it is not, because a recurrent layer has no per-token record to return to. It has one state, and that state is the result of having consumed every token up to now. To resume at token n you need the state as it was at token n, and there is no way to reconstruct it from the middle.

So the engine checkpoints the recurrent state at block boundaries. That works, and it immediately forces the two page sizes to line up, because both come from one uniform pool and a page has to be able to hold a whole state:

attention_block_size × attention_bytes_per_token  ≥  recurrent_state_bytes

The attention page is now as large as one recurrent state. Nothing about that number has anything to do with prompts, sharing or reuse. It is a property of the model, and the caching layer simply inherits it.

What that costs, exactly

Prefix caching stores whole blocks. A partial block cannot be stored, and the last full block is left open so a continuation can extend it rather than being sealed. So out of a shared prefix, the tokens you actually get back are:

cached = (⌊shared / block⌋ − 1) × block

That single line explains the whole shape of the problem, and it has two consequences worth stating separately, because they are usually confused for each other.

There is a hard floor. Any prompt shorter than two whole blocks caches nothing whatsoever. Not a poor hit rate, a structural zero. With a 64-token page that floor is 128 tokens and you will never meet it. With a page in the thousands, it sits well inside the range of prompts people actually send.

The waste is a fixed token count, not a fixed share. Roughly one block plus whatever the prefix leaves over is discarded, regardless of how long the prompt is. On a 64-token page that rounds to nothing. On a page a hundred times larger it is a quarter of a short prompt and a rounding error on a long one, which is why the same model can look broken on one workload and fine on another.

Putting a number on it

The site already records the architecture, so I can estimate the page rather than take it on faith. GLM-5.3-Flash runs 34 linear-attention layers at 64 heads and 128 head dimension, and 11 sparse attention layers over a low-rank latent, in an FP8 checkpoint. The recurrent state is one matrix per head:

state    = 34 layers × 64 heads × 128 × 128  ≈ 34 MB per request
per tok  = 11 layers × (512 + 64)               ≈ 6.3 KB per token
block   ≥ 34 MB / 6.3 KB                         ≈ 5,600 tokens

Call it thousands rather than tens, against 64 for a non-hybrid model on the same stack. I want to flag that as derived and approximate: the exact figure depends on state layout details a model card does not pin down, including whether the short convolution state is counted and what precision the state is held at. The order of magnitude is the point, and the order of magnitude is not close.

The router is not the problem, and cannot be the solution

The failure mode I would most like to save someone else is this one. If you route requests by cached-prefix affinity, a workload like this makes the router look broken: it reports zero cached blocks on every worker, every time, and the obvious conclusion is that affinity is misconfigured.

It is not. Consider an agent session that grows by a few thousand tokens per turn against a block thousands of tokens wide. Each turn adds new history, and that new history never completes a whole block. The router is asked which worker holds a matching block and the honest answer is that no worker holds one, because none exists. It then falls back to load balancing, which is the correct behaviour when there is no cache information to act on.

No amount of affinity tuning changes that, and this is the general lesson I would keep from the whole exercise: you cannot act on information that is structurally absent. Before tuning the component that reports the bad number, check whether the number it is reporting can be non-zero at all.

So what do you do

Not much, honestly, and that is the useful conclusion. The behaviour is a known characteristic of how hybrid models are handled upstream rather than anything about a particular deployment, and the alignment mode it depends on is still flagged experimental. Anyone serving one of these models on this engine sees the same shape.

What you can do is stop treating the number as a defect and start treating it as a fit question. A hybrid model built for very long context pays almost nothing for its block size at the lengths it was designed for, and pays heavily at short ones. If the traffic is short prompts, the block size is telling you the model is not matched to the workload, which is a placement decision rather than a bug to chase. And if you quote a synthetic benchmark hit rate as though it were the production one, check what prefix fraction the benchmark assumes first, because real sessions usually share far more than a load generator does, so the synthetic figure will understate what you actually get.

Where I got this