Skand Tandon
Standalone · Engine

The KV cache has no ABI

Disaggregation is simple on a slide: prefill produces a KV cache, you ship it to decode. The problem is the word "it". There is no single representation of a KV cache, and inside the most widely deployed engine there are dozens of answers to what shape the thing is.

2026-08-20 engine disaggregation 13 min read

Disaggregation is simple on a slide. Prefill runs the prompt through the model and produces a KV cache. Ship that cache to the decode machine. Decode generates tokens from it. One artifact crosses one wire, once, per request.

The problem is the word "it". There is no single representation of a KV cache, no standard describing one, and inside the most widely deployed inference engine there are dozens of different answers to what shape the thing is.

What is actually in the cache

Take the ordinary case, multi-head or grouped-query attention on a GPU. vLLM's FlashAttention backend reports its cache shape as a four-dimensional tensor:

# vllm/v1/attention/backends/flash_attn.py
@staticmethod
def get_kv_cache_shape(
    num_blocks: int,
    block_size: int,
    num_kv_heads: int,
    head_size: int,
    cache_dtype_str: str = "auto",
) -> tuple[int, ...]:
    return (num_blocks, num_kv_heads, block_size, 2 * head_size)

Four things to notice, none of which are universal:

The engine knows they are not interchangeable, which is why the connector interface has a method for asking:

def get_required_kvcache_layout() -> str | None:
    """Returns "HND", "NHD", or None if no specific layout is required."""

A negotiation method for a layout question, inside one engine, on one vendor's hardware. That is the shape of the problem in miniature, before any vendor boundary is involved.

Why have one shape when you can have dozens

Search vLLM for def get_kv_cache_shape and you get dozens of concrete implementations, each returning a different tensor shape for a cache the disaggregation slide treats as a single object. They divide along several axes at once:

Multi-head Latent Attention is the sharpest divergence, because it changes the rank of the tensor. DeepSeek's MLA does not store keys and values at all. It stores a compressed latent plus a separate positional component, so the head dimension disappears:

# vllm/v1/attention/backends/mla/flashmla_sparse.py
@staticmethod
def get_kv_cache_shape(
    num_blocks: int,
    block_size: int,
    num_kv_heads: int,  # assumed to be 1 for MLA
    head_size: int,
    cache_dtype_str: str = "auto",
) -> tuple[int, ...]:
    if cache_dtype_str == "fp8_ds_mla":
        # V3.2 main MLA: 656-byte custom storage format. See module docstring.
        return (num_blocks, block_size, 656)
    else:
        return (num_blocks, block_size, head_size)

Read the parameter list against the body. num_kv_heads is accepted and then ignored, with a comment explaining it is assumed to be 1. The signature describes a world this implementation does not live in.

And then there is 656

A shape whose last dimension is a byte count, not a number of elements. The surrounding kernel spells out the contract:

// csrc/libtorch_stable/cache_kernels.cu
if (kv_cache_dtype == "fp8_ds_mla") {
    STD_TORCH_CHECK(kv_lora_rank == 512, "kv_lora_rank must be 512 for fp8_ds_mla");
    STD_TORCH_CHECK(pe_dim == 64, "pe_dim must be 64 for fp8_ds_mla");
    STD_TORCH_CHECK(kv_cache.size(2) == 656 / kv_cache.element_size(),
                    "kv_cache.size(2) must be 656 bytes for fp8_ds_mla");
    STD_TORCH_CHECK(kv_c.element_size() == 2, ...);
    STD_TORCH_CHECK(k_pe.element_size() == 2, ...);
}

The 656 bytes decompose into three regions:

A 576-element logical vector stored in 656 bytes, in three regions of two different dtypes, with quantization scales interleaved between them at tile granularity determined by how a warp writes its lanes. This is not a tensor layout. It is a struct, defined implicitly by a kernel, and every consumer has to reverse-engineer it.

Mental model

The KV cache is like a shipping container with no standard. Every port (engine, vendor, model) uses a different crate size, stacking order, and labeling system. Disaggregation is the moment you realize the container has to cross ports, and nobody agreed on the container.

What this means for anyone serving at scale

If you are running prefill and decode on the same vendor's stack, the handoff is invisible because both halves are written by the same people against the same layout. The producer knows the consumer will want 128-element tiles in a particular order, so it writes the cache in the order that makes the consumer's TMA descriptor cheap. That agreement is real, load-bearing, and entirely undocumented, because it never had to leave the building.

The moment you mix vendors, or engines, or even model families, the seam becomes visible. The bytes land in HBM, and getting them from HBM into the 128 KB of shared memory where the attention kernel actually wants them is a second movement, performed by the receiving side, using its own machinery.

The KV cache has no ABI. Disaggregation is the first time the two halves of the handoff are not written by the same organisation, and every undocumented assumption becomes a real cost.

Back to the blog