The KV cache: where memory goes

Ask a model a question and it reads its weights once, then produces tokens one at a time. But there's a second thing it reads on every single step, and it grows as you talk: the KV cache.

People think the model is the memory hog. Usually it isn't. For a long enough conversation, the KV cache is bigger than the model itself.

Why the cache exists at all

Attention needs every token to look at every token before it. When the model produces token 50, it needs the Key and Value vectors of tokens 1 through 49 to compute attention scores. Recomputing them would mean re-running the whole forward pass for every token, which is absurd. So we cache them.

That cache is called the KV cache, and it grows linearly with sequence length. The formula is brutal and simple:

KV cache bytes = 2 × layers × kv_heads × head_dim × seq_len × bytes_per_param

The leading 2 is for Key and Value. Let's plug in Llama 3 8B: 32 layers, 8 KV heads (it uses grouped-query attention), head dim 128, FP16 (2 bytes):

2 × 32 × 8 × 128 × seq_len × 2 bytes
= 131,072 bytes per token of context
≈ 128 KB per token

At 128K context, that's 16.8 GB. The model weights are 16 GB. The cache alone nearly doubles the memory footprint, and it's per request.

Why grouped-query attention exists

Multi-head attention (MHA) gives every layer 32 heads, and caches all 32. Grouped-query attention (GQA) shares Key and Value heads across groups, so 32 query heads map to 8 KV heads. That's a 4x reduction in cache size for a tiny quality cost, and it's why every modern model uses it.

Mistral 7B, Llama 3, DeepSeek, all of them. GQA is the quiet hero of long-context inference.

Why this matters in production

The KV cache is why long context is expensive. It's why serving engines evict old tokens, why they reserve memory blocks, and why a single long conversation can crowd out a hundred short ones. It's the reason vLLM exists, the reason for prefix caching, and the reason context limits feel like a hard wall.

Play with it yourself

Adjust the model and context length. The calculator uses the real formula:

The takeaway

If your model feels slow at long context, it's not the weights. It's the diary.

Next: the roofline, which explains why decode is memory-bound and why the KV cache is the reason.