The KV cache is often bigger than the model. Prefix caching reuses it. Mooncake takes that to its logical end: what if the cache lived outside the GPU entirely, shared across a whole cluster?
Mooncake is Moonshot AI's (the Kimi team) architecture for exactly this. It's the most interesting inference system to come out of 2025, and it's open source.
The problem it solves
In a normal setup, the KV cache lives on the GPU, next to the model. That means:
- Cache capacity is capped by GPU memory, which is also needed for weights and activations.
- Prefix caching only helps within one GPU's cache, so a shared prefix across replicas is recomputed everywhere.
- Long context is brutal, because the cache grows with context and the GPU is the bottleneck.
Mooncake's bet: move the cache off the GPU, onto a pool of machines, and treat it as a cluster-wide resource. The GPU keeps the model and does the compute; the cache pool holds the keys and values.
Disaggregation, done right
Disaggregation (splitting prefill and decode onto different machines) is one idea. Mooncake goes further: it disaggregates the cache itself. A pool of CPU machines with lots of memory and fast networking holds the KV cache, and any GPU in the cluster can read from it.
The result: a prefix cached once is cached everywhere. A long conversation's KV can live in the pool, not eat a GPU. And the cache pool scales independently of the compute pool, which is exactly what you want when cache is the bottleneck.
Today's KV cache is a library in every classroom. Mooncake is a central library every classroom shares. Same books, one building, way less duplication.
Why it matters
This is the direction the whole field is moving. NVIDIA's Dynamo, the disaggregation paper, LMCache, and Mooncake are all converging on the same truth: memory is the new bottleneck, and the fix is to stop treating it as a per-GPU problem.
For me, the lesson is practical: when you're designing for long context and heavy prefix reuse, the architecture question isn't "how do I fit the cache on the GPU", it's "how do I share the cache across GPUs". Mooncake is the answer to the second question.
The takeaway
Mooncake is what happens when you take the KV cache seriously as a first-class citizen. Not a byproduct of inference, but a resource to be pooled, shared, and scaled on its own.