Serving a model that is still being trained.

These are the differences people report between a stock open-source stack and one that has been worked on for this specific shape of workload. I have not measured any of them myself, so read them as the order of magnitude rather than as my benchmark. What I can do is explain where each one comes from, and the rest of this page is that.

None of these are kernel micro-optimisation. Every one is a decision about where work should happen or what should be allowed to wait, which is why they are worth writing down rather than profiling for.

Tensor parallelism is what I reach for on this site constantly, because for a single interactive request it is the right answer: split the attention computation across devices and every rank shares the work of one token. Latent attention breaks that cleanly. The whole point of compressing keys and values into one latent head is that there is only one of it, and a thing you cannot split is a thing every rank has to hold a copy of. Eight-way tensor parallelism with latent attention gives you eight identical caches and the cache capacity of one.

The alternative is to stop splitting attention at all. Give each rank its own batch of requests and let it run every attention head itself, then use expert parallelism for the mixture-of-experts layers, where the weights genuinely are separable. Cache capacity now adds up across ranks instead of repeating, which is the entire win, and it is a large one when capacity is what you ran out of.

The cost is latency, and it is not a small cost. A single request no longer has four devices working on its attention, it has one. So this is a throughput-ceiling choice rather than an interactivity choice, and it is right here precisely because nobody is waiting on any individual rollout. It would be the wrong call on an interactive endpoint, which is why the tensor parallelism widget elsewhere on this site reaches the opposite conclusion from the same arithmetic.

This is the part I find genuinely worth internalising, because it generalises far beyond this workload and the arithmetic is one line. When ranks in a group synchronise every decode step, the group runs at the speed of whichever rank is slowest on that step. So a rare per-rank event stops being rare the moment you ask about the group instead of the individual.

Suppose finishing a generation triggers some expensive bookkeeping on that rank, and a rank generating L tokens finishes on any given step with probability 1/L. The chance that nobody in a group of R finishes is (1 − 1/L)R, so the chance that at least one does, and therefore that everybody waits, is one minus that. Drag it around:

The fix, once you see it that way, is not to make the bookkeeping faster. It is to stop doing it. If the expensive part is returning routing decisions for the whole sequence every turn, and the client already has everything except the newest tokens, then send the suffix. The work was never necessary, it was just never noticed, because on one rank it was too cheap to show up in a profile.

That is the general lesson I would keep. Profiling a single rank tells you almost nothing about a synchronised group, and the thing you are looking for will not be the biggest entry in anybody's trace. It will be a small cost multiplied by a coupling factor you forgot was there.

I have argued elsewhere for running prefill and decode on separate hardware, on the grounds that one is compute bound and the other is bandwidth bound and neither should have to share a machine with the other. That argument holds here too, but a sharper one arrives on top of it.

Decode kernels get most of their speed from being replayed as a captured graph, which requires the shape of the work to stay fixed. A prefill arriving inside a decode group changes the shape and the replay is abandoned for that pass, so every rank in the group drops to the slow path. On its own that is survivable. In a group of thirty-two it is the straggler arithmetic from the section above, wearing a different hat: the chance that some rank is prefilling at any moment climbs with the group size until it is most of the time, and the fast path is effectively gone.

Delaying prefill to protect the decode group is the obvious first move and it stops working for exactly the same reason. Once the group is wide enough there is no quiet window left to delay into. Separating the two phases onto different hardware is what actually removes the interaction, and reported decode throughput roughly doubles.

It is not free. Prefill and decode now each need their own copy of the cache for a request in flight, so total cache capacity falls, which is the thing wide expert parallelism was introduced to increase. The two optimisations pull against each other and the balance is an empirical question rather than a principled one.

Reasoning traces vary enormously in length. Batch them the naive way and every sequence is padded out to the longest one in the batch, so a batch containing one very long trace spends most of its arithmetic computing nothing. Packing the traces end to end into one long row fixes that, provided the attention boundaries and loss masks travel with them so traces cannot read each other.

What surprised me is that giving every worker an equal number of tokens does not balance the work. A trace is not divisible, so equal token counts still leave one worker holding a combination that finishes late, and everyone waits for it. The scheduling fix is the oldest one there is: sort by cost, longest first, and drop each trace into whichever worker currently has the most room. Greedy longest-first is not optimal, but it is close, it costs a sort, and it beats the obvious alternative by enough to notice.

Split a long sequence across four devices in order and they do wildly different amounts of work, even though they hold identical numbers of tokens. The cause is the causal mask: a token near the end attends over everything before it and a token near the start attends over almost nothing, so the work per chunk grows linearly with position. The last device does roughly seven times the work of the first, and the other three sit waiting for it.

Give each device one chunk from the front and one from the back and the two halves add up to the same total on every device. It costs nothing, it changes no arithmetic, and it is purely a question of which chunk goes where. I like this one because the imbalance looks like a data problem and is not: the data is uniform, the mask is the thing that is triangular, and you fix it in the assignment rather than in the sizes.

Serving a model, the memory question is weights plus cache and the answer barely moves. Training it, the weights are a minority shareholder. Optimizer state is the largest single block, activations are next, and both dwarf the parameters they belong to.

The useful observation is that the biggest block is also the one you touch least often. Optimizer state is needed during the update and is dead weight for the whole forward and backward pass, so it can live in host memory and be streamed back through a small window on the GPU when the update comes. Activations are the opposite: needed constantly, but some of them are cheap enough to recompute that holding them is a poor trade, while others are expensive enough that recomputing them is worse than the memory. Recomputing everything is the blunt version of this and it costs real throughput; picking per tensor is most of the benefit at a fraction of the cost.

Taken together these reportedly cut peak memory from around 320 GB per GPU to 132 GB at 64K context, which is the difference between needing 128 GPUs and needing 64. That is the number I would actually chase, because halving the GPUs per experiment doubles the number of experiments, and on a shared cluster that compounds faster than any throughput improvement.

A reinforcement learning loop where the model calls tools has a problem that pure text generation does not: something has to execute what the model wrote, and what the model wrote is not trustworthy. Run it in the same process as training and one request for eighty gigabytes of RAM takes down the job.

The observation I would not have made is that a GPU node has a great deal of CPU sitting idle. Training and inference use a handful of cores to feed eight accelerators, and the rest are doing nothing at all. Putting the sandboxes on a separate CPU set of the nodes the job already holds means the capacity was free, the traffic never leaves the cluster, and the sandbox lifetime follows the job, so the scheduler handles queueing, priority and preemption without anybody writing a scheduler.

The part I would steal regardless of workload is the last one. A separate sandbox service needs its own pool, which is stranded whenever the job is not running, and its own scheduling, which duplicates the scheduling you already have. Tying the lifetime to the job deletes both problems at once, and it is a smaller idea than it sounds: the resources were allocated, so use them.

A note on the numbers. The four figures at the top of this page, and the memory reduction further down, are what has been publicly reported for stacks tuned this way. I have not run any of these benchmarks and this site's rule is that measured numbers are my own, so I have flagged these as reported wherever they appear. The straggler probability is the exception: that one is derived on this page from the assumptions stated next to it, and you can check it by moving the sliders.