Speculative decoding is the trick that makes LLM inference feel like it's cheating. Instead of generating one token per forward pass, you use a small draft model to guess the next several tokens, verify them all at once with the big model, and keep the guesses that were right. DSpark is DeepSeek's production-grade take on it, with a twist: the draft model decides how many tokens to speculate based on confidence.
The core idea
Every forward pass through a transformer produces one token. That's the fundamental serial bottleneck of autoregressive generation. Speculative decoding breaks it: draft a sequence of tokens cheaply, verify them in parallel, accept the prefix that matches.
DSpark (Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation) is DeepSeek's implementation, part of the DeepSpec framework alongside DFlash and Eagle3. It's not a toy: it's built to run in production, disaggregated, on the same hardware that serves DeepSeek V4. DeepSeek reports 2-3x latency reduction on autoregressive decoding without quality loss.
The confidence twist
Fixed-k speculation has a problem: if you always guess 4 tokens and the model only accepts 2, you've wasted compute on the rejected ones. DSpark's draft model is confidence-scheduled: it produces more tokens when it's confident and fewer when uncertain, avoiding the fixed-k penalty.
The draft model operates semi-autoregressively, generating blocks of tokens rather than one at a time, to maximize acceptance rate while minimizing wasted compute. It's trained against cached target-model outputs, so it learns what the big model would actually say.
Why it matters for serving
- Throughput per GPU. Speculative decoding lets you serve more requests per GPU because each request spends fewer forward passes generating.
- Latency per token. The perceived speed of the model goes up because tokens arrive faster, even though the model itself is unchanged.
- Variable-length generation. Chat and code have wildly varying lengths; confidence scheduling adapts speculation depth to the task.
Speculative decoding is like a chef pre-chopping vegetables they think the line cook will need. DSpark's twist: the chef only pre-chops when they're confident, so they waste less when they guess wrong.
The draft model
The draft model is the key to the whole thing. It has to be fast (cheap to run) and smart (good at predicting what the big model will say). A draft that's too dumb gets rejected constantly and you've added latency for nothing. A draft that's too big costs more to run than the verification saves.
DeepSeek's approach: train lightweight draft models against a cached target-model output. The cache stores hidden states, not just tokens, enabling knowledge distillation into the draft. It's big (roughly 38TB for a Qwen3-4B target), but it's the thing that makes the draft actually predict the target.
Verification in parallel
The beautiful part: verifying K draft tokens costs about the same as one forward pass, because the transformer processes them in parallel (with a causal mask). So the cost of a speculative step is roughly draft cost + one verification pass, and the benefit is accepted tokens per step. When acceptance is high, you generate multiple tokens per verification pass, and the wall-clock speedup is close to the acceptance rate.
In production
I've run DSpark-style setups in disaggregated serving: prefill machines and decode machines, with the draft model colocated on the decode side. The pattern that works:
- Draft on the decode machine, so the draft tokens never cross the network.
- Verify on the decode machine too, keeping the KV cache local.
- Measure acceptance rate per prompt class. Code and math accept more than free-form prose. Route the high-acceptance traffic to speculative paths.
The takeaway
Speculative decoding doesn't make the model faster, it makes the serving faster. Same weights, same quality, fewer forward passes. That's the whole game.
Sources
- DSpark paper (arXiv 2607.05147): confidence-scheduled speculative decoding with semi-autoregressive generation.
- DeepSpec repo: the full-stack framework housing DSpark, DFlash, and Eagle3.
- DSpark paper (PDF): the direct PDF link.
- Fast Inference from Transformers via Speculative Decoding: the original paper that made the trick mainstream.
- SGLang blog: RadixAttention: how KV reuse and speculative decoding compose.