I said decode is memory-bound: the big model loads its weights once per token and does almost nothing with them. Here's the uncomfortable follow-up: that means the big model has spare compute during decode. It's idle, waiting on the conveyor belt.
Speculative decoding spends that spare compute. It's the closest thing inference has to a free lunch, and it's the trick I get asked about most.
The setup
You have two models:
- The target - the big, slow, accurate model you actually want answers from.
- The draft - a small, fast model that predicts what the target will say.
Here's the loop:
- The draft model generates
γcandidate tokens (say 4 of them) in a single pass. It's small, so this is cheap. - The target model runs one forward pass on the whole sequence of candidates, producing logits for all of them at once. One pass, γ tokens for the price of one.
- For each position, compare the draft's distribution with the target's. If they agree, the token is accepted. If they disagree, sample from the residual and restart from there.
The magic: the target model's one forward pass is memory-bound, and it has to load its weights anyway. Whether it produces one token or five, it moves roughly the same bytes. So the extra tokens are nearly free, as long as the draft is good enough that most of them get accepted.
The math of the speedup
Let the draft accept a token with probability p (the acceptance rate). Expected tokens per target pass:
E[tokens] = (1 - p^(γ+1)) / (1 - p)Speedup over vanilla decode, roughly:
speedup ≈ E[tokens] / (1 + γ × (draft_cost / target_cost))If the draft is 10x smaller, its cost per token is ~1/10 of the target's. With γ=4 and p=0.7:
E[tokens] = (1 - 0.7^5) / (1 - 0.7) = (1 - 0.168) / 0.3 ≈ 2.77
speedup ≈ 2.77 / (1 + 4 × 0.1) = 2.77 / 1.4 ≈ 1.98xTwo times faster, no quality loss, because the target's distribution is what's actually sampled. The draft only proposes; the target disposes.
Where it breaks
Speculative decoding isn't free. It has real failure modes:
- Bad draft = no gain. If the draft disagrees with the target, you accept nothing and pay the draft's cost for nothing. Drafts work best when they're the same family as the target.
- It's a latency trick, not a throughput trick. The gain is wall-clock time per token, which helps interactive users. Throughput is unchanged, sometimes slightly worse, because the target still does the same work.
- Batch interactions. In a busy server, the "spare compute" might not be spare. If the GPU is already saturated, there's nothing to spend.
It's a student (draft) writing an exam answer, and a professor (target) checking it. If the professor agrees, the answer stands. If not, the professor writes the real answer. The student is fast but sloppy. The professor is slow but right. Together they finish faster than the professor alone, because the professor only checks, and checking is cheaper than writing.
Play with it yourself
This simulator runs the actual acceptance-sampling math over 200,000 rounds. Drag the acceptance rate and draft length:
The variants
What I described is the original draft-model scheme. The field has moved on:
- Medusa - no separate draft model. The target model's own last layer is extended with multiple heads that predict the next few tokens in parallel. One model, γ+1 heads.
- EAGLE - drafts at the feature level (intermediate activations) rather than token level, which captures more context and accepts more tokens.
- Lookahead decoding - no extra model at all. It finds and reuses n-grams from the current sequence as candidates.
Each is a different bet on where the draft should come from. The principle is identical: spend spare compute on guesses, verify with one cheap pass.
The takeaway
Speculative decoding doesn't make the model faster. It makes the model busier, and busy is better than idle.
Next: prefix caching, the other big trick that exploits the same asymmetry.