Skand Tandon
Production Systems

Speculative decoding acceptance by draft size

Speculative decoding uses a small draft model to propose tokens that the large target model verifies in parallel. The draft length is the key tuning knob: too short and you waste the opportunity, too long and most tokens get rejected. Here is how acceptance rates behave as you increase draft size.

2025-12-13 production systems 9 min read

I first wrote about speculative decoding in day 38, where I implemented the core acceptance-rejection sampling algorithm. Today I want to focus on a practical question that matters for production: how does the number of draft tokens (the "speculation length" or draft size K) affect acceptance rates and end-to-end throughput?

The answer is not obvious. More draft tokens mean more potential speedup per verification step, but each additional draft token is less likely to be accepted. There is a sweet spot, and finding it requires understanding the probability math.

Quick refresher on the mechanism

Speculative decoding works in rounds. Each round has two phases:

The beauty of this approach is that the verify phase processes all K tokens in parallel (as a prefill-like batch), which is compute-bound and fast. If all K tokens are accepted, you have generated K+1 tokens (the K drafts plus one bonus token from the target) in roughly the time it would take the target to generate one token. The speedup factor is up to K+1.

Acceptance probability and draft length

Let's denote the probability that the draft model and target model agree on a single token as alpha. This is the per-token acceptance rate, which depends on how well the draft model approximates the target. For a well-matched draft/target pair (like Llama-3.1-1B drafting for Llama-3.1-8B), alpha is typically in the range of 0.7 to 0.85 for greedy decoding.

For K draft tokens, the expected number of accepted tokens follows a geometric distribution. The expected accepted length is:

# Expected accepted tokens for draft length K
# alpha = per-token acceptance probability
#
# E[accepted] = alpha + alpha^2 + ... + alpha^K + alpha^K
#             = alpha * (1 - alpha^K) / (1 - alpha)
#
# (Plus we always get 1 bonus token from the target model's
#  own sample at the first rejection point)
#
# Total expected tokens per round:
# E[tokens] = (1 - alpha^(K+1)) / (1 - alpha)

def expected_tokens(alpha: float, K: int) -> float:
    """Expected tokens generated per speculative round."""
    return (1 - alpha**(K+1)) / (1 - alpha)

# Example: alpha=0.8, K=5
# E[tokens] = (1 - 0.8^6) / (1 - 0.8) = (1 - 0.262) / 0.2 = 3.69

So with an 80% per-token acceptance rate and 5 draft tokens, you expect about 3.7 tokens per round. Without speculation, you would get exactly 1 token per round. That is a 3.7x speedup in token count per forward pass of the target model.

The diminishing returns of longer drafts

Here is where the math gets interesting. As K increases, the expected tokens per round grows, but with rapidly diminishing returns:

# Expected tokens for alpha=0.8 at various K values
# K=1:  1.80 tokens/round
# K=2:  2.44 tokens/round
# K=3:  2.95 tokens/round
# K=4:  3.36 tokens/round
# K=5:  3.69 tokens/round
# K=8:  4.40 tokens/round
# K=12: 4.82 tokens/round
# K=20: 4.99 tokens/round
# K=inf: 5.00 tokens/round (the theoretical max: 1/(1-alpha))

The theoretical maximum is 1 / (1 - alpha), which for alpha=0.8 is 5.0 tokens per round. You are already at 74% of the theoretical max with K=5 and 88% with K=8. Going beyond K=10 yields almost no additional benefit in expected tokens.

But there is a cost to larger K: the draft model takes time to generate those tokens, and the target model's verify pass gets more expensive as K grows (longer sequence in the prefill-like batch). The net speedup is:

def net_speedup(alpha, K, t_draft, t_verify_base, t_verify_per_token):
    """Estimate the wall-clock speedup from speculative decoding.

    t_draft: time for draft model to generate 1 token
    t_verify_base: fixed overhead of target model verify
    t_verify_per_token: marginal cost of each additional draft token in verify
    """
    expected = (1 - alpha**(K+1)) / (1 - alpha)
    draft_time = K * t_draft
    verify_time = t_verify_base + K * t_verify_per_token
    round_time = draft_time + verify_time

    # Compare to baseline: target model generating 'expected' tokens
    baseline_time = expected * (t_verify_base + t_verify_per_token)

    return baseline_time / round_time

How acceptance rate varies in practice

The per-token acceptance rate alpha is not a constant. It varies based on several factors:

Key insight

Acceptance rate is not a property of the draft model alone. It is a property of the (draft, target, prompt, temperature) tuple. You should measure it on your actual workload, not rely on reported numbers from papers that used different prompts and settings.

Finding the optimal K

The optimal draft length balances three factors:

In practice, for the Llama-3.1-1B / Llama-3.1-8B pair on an H100, the sweet spot tends to be K=4 to K=6 for greedy decoding. At K=4, the draft overhead is small and most of the expected tokens are captured. Beyond K=8, the marginal gain per additional draft token is not worth the additional draft time.

vLLM exposes this as the --speculative-draft-tensor-parallel-size and --num-speculative-tokens flags. SGLang uses --speculative-algorithm with --num-draft-tokens. Both default to conservative values (often K=3 or K=5).

Measuring acceptance in production

To tune K for your workload, you need to measure the actual acceptance rate. Both vLLM and SGLang expose this in their metrics:

# vLLM Prometheus metrics for speculative decoding
# vllm:spec_decode_draft_acceptance_rate  - per-token acceptance
# vllm:spec_decode_efficiency            - tokens generated / tokens drafted

# Query with curl
curl -s http://localhost:8000/metrics | grep spec_decode

I typically run a sweep: deploy the model with K=3, 5, 7, 9, send the same benchmark traffic to each, and compare output tokens per second. The result is usually a curve that rises steeply from K=1 to K=4, plateaus around K=5-7, and then slowly declines as draft overhead dominates.

When speculative decoding does not help

Speculative decoding is not universally beneficial. There are several scenarios where it hurts or adds no value:

Speculative decoding is a latency optimization for low-batch, long-generation workloads. It trades compute and memory for wall-clock speed on individual requests. In high-throughput serving where the GPU is already saturated with batched requests, it is usually the wrong tool. Know your workload before reaching for it.

The connection to draft model quality

There is an interesting design tradeoff in choosing the draft model. A larger draft model has a higher alpha (more tokens accepted), but it takes longer to generate each draft token. A tiny draft model is fast but has lower alpha. The extreme case is EAGLE-style drafting, where the "draft model" is just a lightweight feature-prediction head attached to the target model's hidden states, giving very high acceptance rates with minimal overhead.

For standard speculative decoding with separate draft models, the sweet spot is usually a model that is 4-10x smaller than the target. Llama-3.1-1B for Llama-3.1-8B (8x ratio), or Llama-3.1-8B for Llama-3.1-70B (roughly 9x). Going below a 4x ratio means the draft is too slow; going above 10x means alpha drops too far.

Next up, I will look at KV cache hit rates to understand how prefix caching interacts with real traffic patterns.