I spent a long time thinking about the KV cache as a memory problem: how big is it, does it fit, how do I evict. That framing is wrong, or at least it stopped being right somewhere around the point where a serving stack could spill cache to NVMe and fetch it back over RDMA faster than it could recompute it. The cache is a storage hierarchy now, and it has an ecosystem.
The tiers
The shape everyone converged on is three levels, and the naming is borrowed straight from CPU caches, which I think is a fair signal about how people are thinking:
- L1, GPU HBM. Fastest, smallest, and the only tier the attention kernel can read directly. Everything else is staging.
- L2, host DRAM. An order of magnitude more capacity at an order of magnitude more latency, across PCIe or C2C.
- L3, distributed. NVMe on other machines, reached over RDMA. Effectively unbounded, and the tier that makes cross-replica prefix reuse possible at all.
The economics are the whole argument. Recomputing a prefill costs hundreds of milliseconds of GPU time you could have sold. Fetching the same cache from a remote NVMe tier costs tens of milliseconds of network. As long as that inequality holds, and on a fast fabric it holds comfortably, the cache belongs in storage rather than in the bin.
Who is building what
These overlap heavily and I do not think the field has settled. Worth knowing which layer each one is trying to own.
- Mooncake: the argument itself. KVCache-centric scheduling, where the first question about a request is not which worker is free but where its cache already is. Its transfer engine moves blocks over RDMA at line rate.
- HiCache: SGLang's implementation of the L1/L2/L3 split, with pluggable backends, so the distributed tier can be Mooncake or 3FS or something else.
- 3FS: DeepSeek's distributed filesystem, used as the bottom tier. A storage system, not a serving component, which is rather the point.
- LMCache and AIBrix KVCache: offload layers aimed at production deployments, sitting between an engine and whatever storage you have.
- FlexKV: a distributed KV store with Mooncake transfer integration.
- NIXL: the piece I care about most, because it is the attempt at one API over all of the above. Whether that succeeds decides whether any of this is portable.
What I think this means
Two consequences I keep coming back to.
The first is that eviction policy becomes a capacity-planning decision rather than an implementation detail. If the bottom tier is effectively unbounded, you are no longer choosing what to throw away, you are choosing what to demote, and the cost of getting it wrong is a fetch rather than a recompute. Those are different orders of magnitude and they justify different amounts of cleverness.
The second is less comfortable. A tiered, distributed, RDMA-connected cache is a distributed storage system, and it will have the failure modes of one: stale reads, partial writes, hot shards, and metadata that drifts out of sync between replicas. The serving world is acquiring these problems faster than it is acquiring the operational habits that go with them, and I include myself in that.
Treating the cache as a tiered store turns disaggregated serving into a caching problem. Caching is where the real wins are, and also where the genuinely hard bugs live.
Sources
- Mooncake: the KV cache that spans machines: the KVCache-centric design this all descends from.
- There is no address: why moving the bytes is harder than the tier diagram suggests.
- The KV cache has no ABI: and why a single API over all of these is not a small ask.