MiniMax M3: sparse attention with the loop turned inside out
428B total, around 22B active, a million tokens of context, and an attention design that gathers queries onto KV blocks rather than the other way round. Nine times faster prefill and fifteen times faster decode than its predecessor at full context.
The architecture
how it is builtMiniMax Sparse Attention
A grouped-query backbone with MSA layered on top. The distinguishing move is the loop order: rather than iterating queries and gathering the KV they need, MSA iterates KV blocks and gathers the queries that land in them. The vendor's framing is KV outer, gather Q.
Why the loop order matters
On a memory-bound workload the access pattern is the performance. Making KV the outer loop means each block is read once and amortised across every query that touches it, instead of being re-read per query. That is the same instinct as FlashAttention tiling, applied to sparsity rather than to the full matrix.
Finer block partitioning
MiniMax argue MSA divides the KV into blocks more precisely than DeepSeek's DSA or MoBA, which raises effective context coverage for the same compute budget. That claim is comparative and comes from the people who built it, so treat it as a hypothesis rather than a result.
Native multimodality from step zero
Interleaved multimodal training from the start rather than a vision tower attached later, which is the same direction Kimi K3 and GLM-5.3-Flash took.
Serving it
the playbookA million-token window with, by the vendor's numbers, one twentieth the per-token compute of the previous generation at that length. Two inference modes: thinking on for reasoning, off for latency-sensitive work.
On a 4,000-GPU HGX B200 fleet
500 nodes, eight B200s each, 180 GB per GPU
Weights are 428 GB (derived at one byte per parameter). Usable memory is 180 GB x 0.92 per GPU,
the gpu_memory_utilization from the published recipe. A placement only counts
here if the cache gets at least 30% of what is left, because a config that fits the weights
and nothing else cannot serve a request.
Tensor parallel stays inside the NVSwitch, so no TP traffic ever crosses a NIC.
- The parameter counts are secondary-source. MiniMax's own announcement does not state total or active parameters. The 428B and roughly 22B figures come from third-party coverage, which is why the weight footprint below is marked derived and should be confirmed against the checkpoint.
- Speedups are against its own predecessor. Nine times prefill and fifteen times decode are measured against M2, not against another vendor's model. That is a legitimate way to report an architectural improvement and a misleading way to read a leaderboard.
- It fits comfortably. At a derived 428 GB it takes four B200s with room for cache, which puts it in the same practical bracket as GLM-5.3-Flash rather than the multi-node bracket Kimi K3 lives in.
The one I find most interesting architecturally
my readEvery other model here got long context by making attention sparse in roughly the same way: an indexer picks the important tokens and you attend to those. MiniMax inverted the loop. Instead of taking each query and gathering the KV blocks it needs, MSA takes each KV block and gathers the queries that hit it. That sounds like a bookkeeping detail and it is not, because the outer loop is what determines your memory access pattern, and the memory access pattern is the whole game on a memory-bound workload. A 15x decode speedup at a million tokens is the kind of number I would want to reproduce before believing, but the reasoning behind it is sound and it is a genuinely different idea rather than an increment.
The numbers
measured elsewhere, not here- Total
- 428B
- Active
- ~22B
- Prefill
- 9x vs M2
- Decode
- 15x vs M2
Speedups are the vendor's own, measured against their previous generation at 1M context. Parameter counts are from third-party coverage rather than the announcement.
Sources: MiniMax: introducing M3 · GitHub: MiniMax-AI/MiniMax-M3 · MiniMax Sparse Attention