A dense model is a model where every parameter is used for every token. A Mixture of Experts (MoE) model is a model where only a fraction of the parameters are active per token, chosen by a router. That one sentence explains the entire shift in how frontier models are built and served.
The math
Consider a dense 70B model. Every token touches all 70B parameters. The compute per token is fixed, and so is the memory footprint: you need enough GPU memory to hold 70B weights, period.
Now consider an MoE model with 671B total parameters but only 37B active per token. Each token touches 37B parameters, but the model has 671B of knowledge stored in its experts. The compute per token is roughly the 37B model's compute, but the capability is closer to a 671B dense model.
That's the bet, and it's why DeepSeek V4, Kimi K3, GLM 5.2, Mixtral, and most frontier models are MoE. You get dense-model quality at a fraction of the per-token cost.
What MoE means for serving
- Memory is the tax. You still need to hold all 671B weights across the cluster, even though only 37B are active. Expert parallelism spreads them across GPUs.
- Routing is the new knob. Each token picks its experts. Load imbalance (everyone wants the same expert) is the production problem, solved with capacity factor and auxiliary losses.
- Batching is trickier. Different tokens in a batch may route to different experts, so the batch's compute is data-dependent. This is why MoE serving engines (vLLM, SGLang) have specialized schedulers.
A dense model is one generalist who knows everything but only has one brain. An MoE is a company of specialists: each expert is narrow, but the router sends each question to the right person, and the company knows far more than any individual.
When dense still wins
Dense models aren't dead. For small models (under 10B), the MoE overhead isn't worth it. For edge and on-device, dense is simpler and uses less memory. And for low-latency single-stream serving, a dense model has no routing overhead and no expert-load imbalance to manage.
Gemma is the perfect example: Google ships both dense and MoE variants, and the dense ones are the escape hatch when you need predictability.
The capacity factor knob
The one serving knob that matters for MoE is the capacity factor. Each expert has a buffer sized at (tokens_per_batch / num_experts) × CF. At CF=1.0 there's zero headroom, so any load imbalance drops tokens, which hurts quality. At CF=1.25 you waste 25% of the expert buffer but survive skew. Tuning it is a memory-vs-stability trade, and it's the first thing to check when an MoE model's quality degrades under load.
Sources
- Switch Transformers: top-1 routing, capacity factor, 7x speedup.
- DeepSeekMoE: fine-grained experts + shared experts.
- Mixtral of Experts: top-2 routing, 47B total / 13B active.
- MoE Meets Instruction Tuning: why instruction-tuned MoE routes better.
The takeaway
MoE won because it decouples knowledge from compute. The model can be huge, but the cost per token stays small. Serving it well is the new frontier, and it's a routing and memory problem more than a compute problem.