Skand Tandon
Architecture

Twenty-two thousand five hundred and eighty GPT-2s.

That is how many 124M-parameter GPT-2s fit inside Kimi K3 by parameter count. Seven years, a factor of twenty-two thousand, and the obvious question is whether any of it is more than scale. It is, and the through-line is narrower than I expected: every architectural step on this page is an eviction policy for a memory that cannot hold everything.

Watch a state forget The lineage What ships today

Seven steps, each answering the last one

click through it

Read this as a chain of complaints rather than a list of models. Each design exists because the one before it hit a specific wall, and each one buys its way past that wall with something it gives up.

The same writes, three eviction policies

this is the whole argument

One fixed 8x8 state, one deterministic stream of key-value pairs, and three rules for writing into it. Additive piles every association on top of the last. Delta reads what is already at that key and writes only the difference. Gated delta fades the whole state first, then writes the difference. Push it past the point where the state fills up and the three come apart.

Read error is the mean of ||v̂ - v|| / ||v|| over every pair written so far, where v̂ = k · S. An error of 1.00 means the read is as wrong as answering zero, and the additive rule climbs past that: it returns something confident and incorrect rather than nothing. I started this widget measuring cosine similarity instead and had to throw the result away, because cosine only looks at direction and the additive state keeps every direction present while burying the magnitudes. It scored additive as the winner and the entire argument backwards. Gold is a positive weight in the matrix, blue is negative. Eight dimensions is a toy and not a simulation of any real model, but the failure mode is the real one.

Why the chunk size is 128 and not 1

FLOPs against hardware

The delta rule is sequential by construction, which makes prefill a problem. Chunking fixes it: do real masked attention inside a chunk, fold across chunks into the state. The chunk size then sets a tradeoff that is not the one you would guess from counting operations.

The same two questions, in everything shipping now

eight architectures

The chain above ends at one model. Here is what the rest of the open frontier does with the same problem. Strip the branding off any of them and each one is answering two questions on every token: which part of the context it is allowed to read, and which of its own parameters are allowed to fire. Both are the eviction policy from the top of this page, moved out of the recurrent state and into the layer stack and the router. Nothing here is a new claim; it is the architecture sections of the model pages put side by side, generated from the same table so the two cannot disagree.

The sparsity column is the one I would read first. Not one of these fires more than six percent of its resident parameters on a token, and the two largest fire under four, which is the entire reason a 2.8 trillion parameter model is something you can serve rather than something you can only cite. It also sets the real constraint: total parameters decide whether you can host a model at all, active parameters decide what each token costs you, and those two numbers have been pulling apart for a while. Where a lab has not published an expert count I have said so rather than inferring one.

What I take from the whole chain

my read

It was never just scale

Twenty-two thousand times the parameters is the headline and the least interesting fact. Each step in this chain adds capacity in a specific place for a specific reason: a state that stops growing, a write that replaces rather than accumulates, a decay that frees room, a per-channel decay that frees the right room. Capacity in the wrong place would have bought nothing.

Finite memory forces a policy

Once you decide the cache will not grow with the sequence, you have accepted that things must leave it, and every subsequent design is an argument about which things. Additive has no policy and interferes. Delta replaces. Gating fades. Per-channel gating fades selectively. Routing picks which experts even see the token. They are all the same question.

And attention comes back anyway

The most honest thing in the final design is that it stops trying to win. Three cheap recurrent layers, then a fourth that does real attention over the actual context to retrieve what the state threw away. Then, every twelve layers, attention again over earlier depths. Having spent seven years replacing attention, the answer keeps it around as the selective read, because nothing else does that job as well.

What I would want to verify

The claim I would not take on trust is that a hybrid beats full attention under controlled comparison. It might well be true and it is exactly the sort of result that depends on what you measured. As with acceptance length, the number is a property of the evaluation as much as of the architecture.

Where this came from

credit

The lineage and most of the technical detail on this page follow ali's worklog tracing GPT-2 to Kimi K3, which is the best single write-up of this progression I have read and which I used as the ground truth throughout.

The explanations, the framing and the visualisations here are my own, and any errors in them are mine rather than theirs. If you want the version with the actual code and the full derivations, read the original.