If I could keep one calculation from everything on this site it would be this one. Generating a token at batch size one means reading every weight the model needs out of HBM and doing almost no arithmetic with them. You are not short of compute. You are short of bandwidth.
tokens/sec ≤ HBM bandwidth ÷ bytes read per token. A 70B model in FP8 is about 70 GB of weights. On a GPU with roughly 4.8 TB/s that is about 68 tokens per second, and that is a ceiling rather than a measurement. If what you measure is far below it, you have an engine problem. If you want to beat it, you have to change one of the two terms or stop being at batch size one.
I find this useful less as a prediction than as a filter. Any proposed optimisation can be sorted by which term it touches, and anything that touches neither is not going to help a batch-of-one decode no matter how good it sounds.
Six layers, and what each one recovers
A production engine stacks all of these. They are worth learning in this order, because each one's benefit only becomes visible once the layer beneath it has stopped being the bottleneck.
- Scheduling recovers GPU idle time between requests and stops prefill stalling decode. Continuous batching, chunked prefill, prefill and decode disaggregation.
- KV cache recovers memory lost to fragmentation, which caps your batch size, and recomputation of prefixes you already have. PagedAttention, radix caching, grouped-query attention, FP8 cache.
- Kernels recover intermediate tensors round-tripping through HBM and per-launch overhead. FlashAttention, fused MoE, CUDA graphs.
- Precision attacks the numerator directly: fewer bytes per weight is a proportionally higher ceiling. This is the only layer that moves the ceiling itself rather than moving you toward it.
- Parallelism is what makes the model fit, and it adds per-token collectives that actively hurt at low batch. It is a capacity technique that costs latency, which is the opposite of how it is usually described.
- Speculation attacks the batch-one latency floor itself.
The asymmetry in that list is the part I would underline. Layers one through five overwhelmingly buy throughput: more tokens across all users, at a given latency. Only speculative decoding buys latency for one user who is already alone on the GPU. That is why every lab that competes on speed converges on the same layer, and why it is the one I would learn last and think about most.
Why speculation works at all
It follows directly from the same division. Verifying eight candidate tokens costs the same weight read as decoding one, because the weights come out of memory once either way. Speculation converts a bandwidth problem into a compute problem on a machine that had compute to spare. That is the whole mechanism, and it is why the technique feels like cheating and is not.
The speedup is roughly τ ÷ (1 + cK), where τ is the mean number of tokens accepted per verification, c is the drafter's cost relative to one target forward pass, and K is the draft length. The entire research field is one sentence: raise τ without raising c.
Read the published methods through that formula and they stop being a list of names. A separate draft model has high c, so its gains are modest. Medusa removes the separate model by putting heads on the target. EAGLE drafts in feature space and is the production default. Multi-token prediction trains the capability into the base model. Block-diffusion drafters like DFlash 2 propose a whole block in one pass. The reported τ values across all of these sit somewhere between about 2.5 and 4.8, which is a real spread and not a chasm.
The part I keep coming back to is that τ is not a property of the method. It is a property of the method on your traffic. A drafter is only good at predicting text that looks like what it was trained on, which means the same published method will give two different numbers at two different companies, and neither is wrong.
Which gaps are closeable
It is worth being clear-eyed about what is actually withheld, because the answer is less exciting and more useful than it sounds.
- The algorithms are published. Every technique above is in a paper and implemented in at least one open engine. Nobody is beating you with a secret scheduler.
- The last stretch is person-years, not insight. The gap between a competently configured engine and a heavily tuned one is CPU-side orchestration overhead, per-shape kernel configs, graph coverage and scheduler heuristics. Unglamorous, enormous, and nobody writes it up.
- What is genuinely proprietary is data. Drafter training recipes and the traffic they were trained on. Cache-aware routing tuned against a known prompt distribution. Moving τ from 3.6 to 4.8 is not something you can read.
- Some speed is quietly lower precision. A provider can post a bigger number by serving a more aggressive quantisation, a shorter context or a smaller reasoning budget. Before treating any leaderboard row as comparable, check what is actually being served.
- And some of it is silicon. Replacing HBM with on-chip SRAM moves the denominator by orders of magnitude, and that part is not available in software at any price.
The practical consequence is the thing I would tell anyone starting: if you have production traffic and GPUs, a drafter trained on your own request distribution is the one advantage in this field that a vendor structurally cannot sell you. Everything else you can buy, read or copy.
Knowing which gaps are closeable is most of the skill. The rest is measuring instead of assuming.
Sources
- Transformer inference arithmetic: the canonical write-up of the calculation this note opens with.
- Making deep learning go brrrr from first principles: compute-bound against memory-bound against overhead-bound.
- Fast inference from transformers via speculative decoding: the original, including the sampling-correctness proof that makes it lossless.
- LLM inference unveiled: survey and roofline model insights: the roofline tied directly to serving.
- Ops:byte and the roofline: the same argument with a calculator attached.