KV cache hit rates across traffic

When I first turned on prefix caching in vLLM, I expected it to be a universal win. Reuse KV blocks from previous requests, skip redundant prefill compute, serve tokens faster. And it does work, but the degree to which it helps depends almost entirely on what your traffic looks like. A chatbot with a shared system prompt sees massive reuse. A batch pipeline with unique documents sees almost none.

Today I want to walk through what KV cache hit rates actually look like across different traffic patterns, what drives them, and how to reason about whether prefix caching is worth the memory overhead for your workload.

How prefix caching works

The idea behind prefix caching is straightforward. When two requests share the same token prefix, the KV cache blocks computed for that prefix are identical. Instead of recomputing them for the second request, you store them and look them up by a hash of the token content.

In vLLM's implementation (and SGLang's RadixAttention), the KV cache is divided into fixed-size blocks, typically 16 tokens each. Each block is hashed based on its token content plus the hash of the preceding block, forming a hash chain. When a new request arrives, the engine walks the prompt tokens block by block, checking each hash against a lookup table. Every hit means one fewer block to compute during prefill.

# Simplified prefix matching logic
def match_prefix(new_tokens, cache):
    matched = 0
    parent_hash = EMPTY
    for block in chunk(new_tokens, BLOCK_SIZE):
        h = hash(parent_hash, block)
        if h in cache:
            matched += len(block)
            parent_hash = h
        else:
            break  # prefix match is contiguous
    return matched

The key constraint is that matching is prefix-only and contiguous. If token 500 differs between two requests, everything from token 500 onward must be recomputed, even if tokens 501 through 2000 are identical. This is a fundamental limitation of how autoregressive attention works: changing one token shifts the KV values for all subsequent positions.

Traffic pattern 1: chatbot with system prompt

This is the best case for prefix caching. A typical chat deployment has a system prompt of 200 to 1000 tokens shared across every single request. Multi-turn conversations also share the full conversation history, which grows over time.

In this pattern, I consistently see prefix hit rates of 60 to 85 percent of total prompt tokens. The system prompt is a guaranteed hit after the first request. Multi-turn conversations that reuse the same session get progressively higher hit rates as the shared history grows. A 500-token system prompt plus a 3-turn conversation with 1500 tokens of history means 2000 out of, say, 2100 tokens are cache hits.

The prefill savings here are substantial. If your median prompt is 2000 tokens and 1600 are cached, you are computing only 400 tokens of new KV values. That can cut TTFT by 70 percent or more, because prefill compute scales linearly (or slightly super-linearly due to attention) with token count.

Traffic pattern 2: RAG with document chunks

Retrieval-augmented generation is more interesting. The system prompt is shared, but the retrieved documents change per query. A typical RAG prompt looks like: system prompt (shared) + retrieved chunks (variable) + user question (variable).

Hit rates here depend on how many unique documents are in your corpus and how often the same chunks get retrieved. In a customer support scenario with maybe 500 FAQ documents, the same 20 to 30 chunks get retrieved repeatedly for common questions. I see hit rates of 30 to 50 percent in this pattern: the system prompt is always a hit, and popular document chunks get reused.

But in a general search RAG with millions of documents, nearly every retrieval is unique. Hit rates drop to 10 to 15 percent, basically just the system prompt. The cache fills up with blocks that never get reused, and you are paying the memory overhead for no benefit.

Traffic pattern 3: batch processing

Batch workloads (summarize these 10,000 documents, extract entities from this dataset) have the lowest hit rates. Each request has unique content, and the only shared prefix is typically a short instruction like "Summarize the following document:" which might be 10 to 20 tokens.

Hit rates here are typically 2 to 8 percent. The cache is thrashing constantly: blocks get evicted before they are ever reused. In this scenario, I would recommend disabling prefix caching entirely. The LRU eviction overhead and hash computation on every request is pure waste.

Traffic pattern 4: code completion

Code completion (Copilot-style) is a fascinating middle ground. Users tend to work in the same file for extended periods, so consecutive requests share a long prefix of the file content up to the cursor. But users also jump between files, causing the prefix to change completely.

In practice, hit rates are bimodal. During focused editing in one file, rates are 70 to 90 percent. When the user switches files, the rate drops to near zero for that request. Averaged over a session, I see 40 to 60 percent depending on user behavior. The cache needs to be large enough to hold the working set of files the user has open, typically 3 to 5 files worth of KV blocks.

Memory cost of prefix caching

Prefix caching is not free. You are holding onto KV blocks that might get reused instead of freeing them for new requests. For a 7B model in FP16 with 32 layers, each token of cached KV takes roughly 256 KB (2 tensors * 32 layers * 128 head_dim * 2 bytes * num_kv_heads). A cache holding 100K tokens of prefixes consumes around 25 GB of GPU memory.

That is memory you could have used for a larger batch size. In high-throughput scenarios where you are already memory-constrained, the tradeoff might not be worth it. The decision comes down to: does the prefill compute savings from cache hits outweigh the throughput loss from a smaller batch?

# Quick memory estimate for prefix cache
num_layers = 32
head_dim = 128
num_kv_heads = 8  # GQA
bytes_per_val = 2  # FP16

bytes_per_token = 2 * num_layers * head_dim * num_kv_heads * bytes_per_val
# = 2 * 32 * 128 * 8 * 2 = 131,072 bytes per token (~128 KB)

cache_tokens = 100_000
total_gb = (bytes_per_token * cache_tokens) / (1024**3)
# ~12.2 GB for 100K cached prefix tokens with GQA

Measuring hit rates in practice

If you are running vLLM, you can track cache hit rates through the metrics endpoint. The key metrics are vllm:prefix_cache_hit_rate and the block-level counters for cache hits versus misses. SGLang exposes similar metrics through its server stats.

I recommend monitoring hit rates over time, not just as a single number. Traffic patterns shift throughout the day. A customer support bot might see 80 percent hit rates during business hours (same questions asked repeatedly) and 30 percent at night (long-tail queries from different time zones). Your cache sizing should accommodate the peak reuse period.

Prefix caching is not a feature you turn on and forget. It is a tradeoff between memory and compute that depends on your traffic. Measure your hit rates, and if they are below 20 percent, you are probably better off using that memory for larger batches.

For background on KV cache memory and block allocation, see the KV cache deep dive. For how PagedAttention manages these blocks, see vLLM PagedAttention. And for the prefix caching implementation with hash-based dedup, check out prefix caching with hash dedup.