Everyone talks about frontier models as if they're only reachable through a vendor API. The uncomfortable secret: the weights are public, the architectures are documented, and the only thing standing between you and frontier intelligence is a good serving stack. I've spent the last year proving that on our own hardware.
This is the story of hosting DeepSeek V4, Kimi K3, GLM 5.2, and Gemma from raw model weights, on machines we control, in disaggregated setups with a Mooncake-style KV cache pool. No vendor lock-in, no per-token markup, just frontier intelligence at the cost of electricity and amortized hardware.
Why disaggregation is non-negotiable for these models
All four of these models are Mixture of Experts (MoE) with huge parameter counts and long context. That combination breaks the naive "load the model on one GPU and serve it" approach in three ways:
- They don't fit on one GPU. DeepSeek V4 is 671B total parameters (37B active). Kimi K3 is 1T total (32B active). GLM 5.2 is 355B total (32B active). You need tensor parallelism across multiple GPUs just to hold the weights.
- Prefill and decode have opposite bottlenecks. Prefill is compute-bound (the prompt is processed in parallel). Decode is memory-bound (one token at a time, streaming weights). Running both on the same GPU means neither runs at its optimum.
- The KV cache is enormous. At 128K context with MLA or GQA, the cache per conversation is gigabytes. It crowds out the weights, and it makes the GPU the wrong place to store it.
So you split the problem. Prefill machines, decode machines, and a cache pool in between. That's the disaggregated setup, and it's how every serious operator runs these models.
The stack
DeepSeek V4: MLA and the attention trick
DeepSeek's signature is Multi-head Latent Attention (MLA). Instead of storing the full K and V for every token, MLA compresses them into a latent vector with a low-rank projection, then reconstructs on the fly. The KV cache shrinks by an order of magnitude, which is exactly what you want when your cache pool is the scarce resource.
On our stack, MLA means the cache pool holds far less per conversation, so we can serve more concurrent long-context sessions from the same pool. It's the difference between a cache pool that fits 100 conversations and one that fits 1000.
Kimi K3: MoE at trillion scale
Kimi K3 is the biggest thing we host: a trillion total parameters, 32B active. That's the MoE bet in its purest form, more parameters than any dense model could serve, but only 3% of them active per token. The routing is what makes it work: each token picks its experts, and the load balancing across experts is the thing you tune in production.
Kimi also runs on Mooncake, which is fitting, because Mooncake is the KV cache pool we run underneath all of this.
GLM 5.2 and Gemma: the pragmatic frontier
GLM 5.2 (355B, 32B active) and Gemma (the open dense/MoE family from Google) round out the roster. Gemma is interesting because it's the one you can serve dense, without disaggregation, when you need a small fast model. The others need the full stack; Gemma is the escape hatch.
All four, from raw weights. The download, the conversion, the quantization, the engine config, the routing, the cache. That's the whole game.
What "frontier at a great cost" means
The math is brutal and beautiful. A vendor API charges per token, and the markup on frontier models is steep. Self-hosting means you pay for hardware once, then amortize it. The cost per token drops by an order of magnitude or more, and the latency drops too, because you're not sharing the GPU with a thousand other tenants.
The catch: you have to build and run the stack. That's the moat, and it's also the point of this whole notebook.
Sources
- Splitwise: Efficient generative LLM inference using phase splitting: the paper that named prefill/decode disaggregation.
- DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving: disaggregation with SLOs.
- DeepSeekMoE: Towards Ultimate Expert Specialization: fine-grained experts + shared experts, the pattern behind DeepSeek V3/V4.
- DeepSeek-V3 Technical Report: MLA, MTP, and the 671B MoE architecture.
- ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools: the GLM lineage.
- NVIDIA Dynamo: the disaggregated serving framework we build on.
- NVIDIA Dynamo for MoE models: MoE inference on GB200.
- Mooncake: Moonshot's disaggregated KV cache pool, the cache layer under all of this.
- Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving: the paper behind the pool, chunked prefill transfer, and 3-tier cache.
- Kimi K2: the trillion-parameter MoE that runs on Mooncake.
The takeaway
Frontier intelligence is not a vendor's secret. It's a stack you can build. The weights are public, the architectures are documented, and the only real cost is the engineering.