Almost everything on this site rests on one claim: decode is memory-bound. Each generated token requires streaming the active weights out of memory, the tensor cores are mostly idle while that happens, and the token rate is set by bandwidth rather than by FLOPs. I have used that to explain batching, quantisation, disaggregation and the shape of the KV cache.
If the claim is true, it has an uncomfortable implication I had not followed through: the GPU is the wrong chip for decode. Not a badly tuned chip, the wrong one. A GPU spends its area on compute and its memory budget on capacity, and decode wants neither of those as much as it wants bytes per second.
The trade the LPU makes
The argument for on-chip SRAM is narrow and, once stated, hard to argue with. Put the memory on the die and you get an enormous bandwidth advantage per byte, at the cost of having very few bytes. Roughly 500 MB of SRAM per chip against a modern GPU's hundreds of gigabytes of HBM, in exchange for bandwidth in a completely different regime.
The capacity objection has an answer, and it is the reason this became viable now rather than five years ago. A mixture-of-experts model does not read all its weights per token, it reads the routed experts. Decode's working set is much smaller than the checkpoint, and small enough that a tier this size stops being absurd.
The other property that gets less attention and would matter more to me operationally is determinism. Execution is compiler-orchestrated rather than dynamically scheduled, so latency does not jitter under load. Everything I have written about p99 being the number that matters is an argument for caring about that a great deal.
The shape of the deal, stated accurately
It is worth being precise here, because the shorthand version is wrong in a way that changes its meaning. NVIDIA did not acquire Groq. The twenty billion was a non-exclusive IP licence plus an acqui-hire, taking Groq's founder, president and most of its engineers. The first chip out of it, Groq 3 LPX, was unveiled at GTC 2026, is built by Samsung on a 4nm process, and packs 256 chips into a rack for 128 GB of SRAM in aggregate.
The headline efficiency figure needs the same care. The widely quoted 35x throughput per megawatt is an LPX rack paired with a Vera Rubin NVL72, measured against a Blackwell NVL72 alone, on trillion-parameter models. It is not the LPU on its own beating a GPU. That distinction is the entire architectural point and it gets flattened every time the number is repeated.
Why this is a complement, not a replacement
The interesting design is not LPU instead of GPU, it is splitting the decode loop itself. Attention is memory-bound but needs access to the KV cache, which is large and lives in HBM. The feed-forward and expert weights are what get streamed per token. So the GPU keeps prefill and attention, where capacity matters, and the latency-sensitive expert decode moves to the part with the bandwidth.
That is the same move as prefill/decode disaggregation, one level down, and it comes with the same question I have been asking all the way through: what exactly crosses the boundary, how often, and what does it cost. A split inside the decode loop runs once per token rather than once per request, which is a much less forgiving budget than shipping a cache between nodes.
What I take from it
I do not have one of these and I am not likely to soon, so I hold the performance claims loosely. What I do take seriously is the signal. The company with the most to lose from the GPU not being the answer to everything has spent twenty billion dollars on the position that one architecture cannot serve both halves of inference. Prefill wants capacity. Decode wants bandwidth. Every page on this site has been circling that sentence, and this is the most expensive possible endorsement of it.
If decode really is bandwidth-bound, then a chip that trades capacity for bandwidth is not exotic. It is the obvious thing, and the surprise is that it took this long.
Sources
- Tom's Hardware: the first chip from the Groq deal: the licence-plus-acqui-hire structure, Samsung 4nm, and the rack configuration.
- The Register: a closer look at the LPX racks: rack-level SRAM and bandwidth figures.
- Blackwell, and what four thousand of them add up to: the HBM side of the comparison, and why bandwidth rather than FLOPs sets the token rate.