Skand Tandon
Standalone · Frontier

Lossless has conditions, and nobody measures the domains where it breaks

Speculative decoding preserves the target's output distribution, which is a real guarantee and a narrower one than I had been writing. The sharper finding is that acceptance length collapses on exactly the workloads the papers never benchmark, and those workloads are most of the traffic.

speculative-decoding evaluation rca

I have written the word lossless on this site more than once without stopping to say what it guarantees. A NeurIPS submission on the evolution of speculative decoding made me go back and qualify it, and then handed me a much more uncomfortable result underneath.

What the guarantee actually is

The rejection sampler accepts a draft token with probability min(1, p/q), where p is the target's probability and q is the drafter's, and on rejection it resamples from the residual distribution. That construction preserves the distribution the target would have produced. It is a genuine result and it is why speculative decoding is not a quality trade.

It is also a statement about distributions, not about bytes. I had been reading it as "the same tokens come out", and that is not what it says. Even greedy decoding against the same model on the same hardware can diverge from the unspeculated path, because verifying a batch of positions and decoding one at a time do not produce bit-identical logits, and a tie broken differently once is a different sentence thereafter.

The guarantee also has preconditions that are easy to violate by configuration rather than by bug. Verification has to be strict, at an acceptance threshold of one; a relaxed threshold trades the guarantee for throughput and shifts the output toward the drafter's preferences. Scheduling has to be non-anticipating, so the decision to verify position k cannot depend on what appeared at k+1. And at temperature zero the property holds cleanly, while at temperature one a relaxed threshold moves the distribution measurably.

The finding that actually worries me

Acceptance length is the number the whole field optimises. The submission introduces a benchmark that measures it across five domains rather than the usual two, and the result is not subtle: on frontend design work, acceptance length falls from around 5.24 to 1.84. A drafter proposing blocks that are almost entirely accepted on a coding benchmark is having most of its work thrown away on a task nobody thought to measure.

Alongside that sits the number I keep re-reading: something like 83% of real token usage on a large public router happens on task types that speculative decoding papers have never evaluated. The benchmark coverage is roughly the inverse of the traffic.

This is the same point I made about acceptance length being a property of the method on your traffic, except that I stated it as a caution and here it is a measurement. The gap between a published 3x and what you get is not noise or tuning. It is a different workload.

The published numbers disagree, and that is the evidence

Worth noticing that the acceptance lengths quoted for the same methods vary substantially between sources. This work puts DFlash 2 at roughly 3.76 against DFlash's 3.11; the method's own authors publish 4.80 on a different target. Both report the same relative improvement of about 21%.

Neither is wrong. They are different targets on different benchmarks, and the fact that a single method spans 3.8 to 4.8 depending on what you point it at is the cleanest demonstration available that this number does not travel. I have corrected the DFlash 2 page to say so rather than quoting one figure as though it were a property of the drafter.

What I would do differently

The algorithm is lossless in the sense it claims. The deployment is lossless only if nobody relaxed a threshold, and only useful if the traffic resembles the benchmark.

Sources