Attention is the kernel that defines inference. Sparse attention is the variant that defines long-context inference. Fireworks' work on MiniMax M3 sparse attention on Blackwell is a masterclass in why the data layout matters as much as the math.
The problem
MiniMax M3 uses sparse attention to handle 1M-token contexts. The naive approach: query-stationary, where you stream the KV cache through the compute. Fireworks found a better way: KV-stationary, where you keep the KV cache resident and stream the queries.
The numbers
- ~980 TFLOP/s achieved with the KV-stationary kernel.
- 1.9-2.4x over a query-stationary baseline.
- ~1.6x over open-source MSA (multi-stream attention).
- nsb/N < 2.85 the reuse crossover where KV-stationary wins.
Query-stationary is like a chef who keeps running to the pantry for each ingredient. KV-stationary is the chef who stages everything on the counter first. For sparse attention, staging wins.
The design space
Fireworks walks through the Q-outer vs KV-outer design space, and the I/O roofline with the reuse crossover at nsb/N < 2.85. The insight: for sparse attention, the KV cache is the thing you want to keep resident, because it's reused across queries. The queries are the stream.
What a fast kernel unlocks
- Long context. 1M-token context is impossible without sparse attention, and the kernel is what makes it fast.
- Blackwell. The SM100 tensor cores are the target; the kernel design is what extracts the ~980 TFLOP/s.
- General lesson. For any memory-bound kernel, the data layout is the optimization. The math is fixed; the layout is the lever.
For sparse attention, the KV cache is the asset. Keep it resident, stream the queries, and you win.
Sources
- Fireworks blog: Optimizing MiniMax M3 Sparse Attention on NVIDIA Blackwell: the kernel walkthrough and the ~980 TFLOP/s number.
- Together AI: Serving MiniMax M3: the 1M-token context serving story.
- MiniMax M3 paper: the hybrid-sparse architecture.
- FlashAttention: the IO-aware attention kernel that started this line of work.