I was benchmarking a lock-free, data-oriented compiler frontend, comparing two ways of interning types. Deferred interning gives every worker a private arena and reconciles at a barrier. Content-addressed interning hashes the structure, so identity is final the moment a type is minted and no reconciliation is needed. Both emit byte-identical MLIR. The only open question was throughput, and the benchmark answered it: deferred gained nothing from a second core, content gained 1.59x. The story wrote itself.
I even had a mechanism. Deferred hands each worker private slow-path arenas, allocating per-function Vecs that content never creates, and malloc contention is thread-count sensitive. Plausible. Testable. Wrong.
Every number in that table is real. I can reproduce all of them. The finding is false.
This failure has a name, and a literature
The interesting part is not that I got it wrong. It is that I got it wrong while doing everything the standard methodology asks for. Fifteen repetitions per cell. Medians rather than means. Three separate whole-run repeats. The interquartile range inside every cell was under 2%. By every convention I had absorbed in ten years of benchmarking on x86 Linux, that is a clean measurement.
It is a clean measurement of the wrong thing, and the phenomenon has been named since 2009. Mytkowicz, Diwan, Hauswirth and Sweeney called it measurement bias: an innocuous-looking aspect of the experimental setup that biases the result enough to overstate an effect or invert a conclusion. Their examples were almost comically mundane, the size of your UNIX environment, the link order of your object files, and their headline result is the one worth sitting with. In a survey of 133 papers from ASPLOS, PACT, PLDI and CGO, they found one that adequately considered measurement bias.
Hoefler and Belli reached the same place from the HPC side six years later, surveying 120 papers across three conferences and concluding it was frequently impossible to tell whether a reported improvement was deterministic or observed by chance.
So the field has known for fifteen years that the standard toolkit does not defend against this class of error. What it has not absorbed is that asymmetric cores turn this class of error from an accident of your link order into the default condition of your machine.
Why every defense I had failed
Look at what the standard defenses actually assume. Repetitions, medians, trimmed means, confidence intervals: every one of these assumes samples are exchangeable, independent draws from a single population, where ordering carries no information. Under that assumption more samples buy precision, and dispersion tells you how much to trust the centre.
Thread placement violates the assumption at the root. Which cores my pool landed on was decided once, when the pool was constructed, before the first repetition ran, and every subsequent sample inherited it. I did not have fifteen draws from one population. I had one draw from a placement lottery, followed by fifteen confirmations of it.
The evidence is unambiguous once you stop comparing the two modes to each other and measure each alone, one mode per process. Two clusters, in both modes. The bimodality follows the process, not the mode.
Now trace what each defense does to that data. Repetitions sample the same cluster fifteen times. The median reports that cluster's centre with excellent precision. Outlier rejection is actively harmful: there are no outliers, there are two populations, and trimming makes each look more trustworthy than it is. The confidence interval is worst of all, because it will be tight, and tightness reads as rigour.
A confidence interval describes dispersion around your estimate. It is silent on whether your estimate is of the quantity you think it is.
The machine underneath
The host is an Apple M4, and sysctl will tell you it is not symmetric:
$ sysctl -n hw.nperflevels hw.ncpu
2
10
$ sysctl -n hw.perflevel0.name hw.perflevel0.physicalcpu \
hw.perflevel1.name hw.perflevel1.physicalcpu
Performance
4
Efficiency
6Four performance cores, six efficiency cores. Ten "CPUs" that are not interchangeable in any sense a speedup table cares about. The caches diverge further than most people expect: a P-core has 192 KB L1i, 128 KB L1d, and a 16 MB L2 shared across the P cluster. An E-core has 128 KB L1i, 64 KB L1d, and a 4 MB L2. A thread landing on an E-core gets a quarter of the L2 and half the L1d. A working set tuned against the P-cluster's 16 MB does not run slightly slower there; it runs a different algorithm's worth of cache misses.
Running the compiler's thread ladder from 1 to 10 gives the shape of the machine directly. The knee lands exactly on 4, which is hw.perflevel0.physicalcpu. Each of the first four threads is worth about 0.4 of a speedup unit; each of the next two is worth 0.15. An E-core contributes roughly 0.37 of a P-core on this workload.
That is the number that makes the bug arithmetic rather than mysterious. A two-thread pool drawing one P-core and one E-core has throughput near 1.37 units instead of 2, stretching wall clock by about 1.7x, very close to the gap I had attributed to a reconciliation barrier.
The fix in the literature does not work here
The known remedy for measurement bias is randomization, and its sharpest implementation is STABILIZER. Curtsinger and Berger observed that code and data layout biases every run, and that this bias is fixed for the life of a process, which is precisely what defeats classical statistics. Their fix is to re-randomize layout repeatedly during execution. Layout effects then become Gaussian noise instead of a constant offset, and ANOVA becomes legitimate again.
The payoff is a result worth remembering: with layout randomized, the impact of LLVM's -O3 over -O2 on SPEC CPU2006 is indistinguishable from noise. A great many published optimization wins were measuring layout.
That logic applies exactly to my problem. Placement is a per-process constant that biases every sample. Randomize it and the bias becomes noise. You cannot randomize it. That part is specific to this machine, and it is not a gap in my tooling.
The Mach thread affinity API still exists, still compiles, still links, and refuses: thread_policy_set(THREAD_AFFINITY_POLICY) returns 46, KERN_NOT_SUPPORTED. Affinity sets were an Intel-era mechanism for grouping threads onto a shared L2. Apple Silicon returns zero. There is no supported way to say "run this thread on core 3", by design, because the scheduler owns placement so it can manage power.
What you get instead is Quality of Service, a declaration of intent from which the OS infers placement. On an idle machine the scheduler has four P-cores available and hands you one regardless of how modestly you asked. But run the same binary under a background task policy (taskpolicy -b) and everything is 3.4x to 3.7x slower. A thread that explicitly requested QOS_CLASS_USER_INTERACTIVE, successfully, still ran at E-core speed. Thread QoS is clamped by the inherited task policy, and the thread cannot observe the clamp.
That is the operational hazard. A CI agent, an IDE build task, a launchd job with a low-priority key, any of these silently applies a 3.6x offset your program cannot detect and your results table does not record. When a macOS benchmark number moves and your code did not, check the launching context before you check anything else.
Block what you cannot randomize
If randomization is unavailable, the remaining tool is older than computing: blocking, from Fisher's design of experiments. You cannot eliminate the nuisance variable, so you arrange for it to affect both treatments identically, and it cancels in the comparison.
The old harness ran each mode's entire thread ladder end to end: two ladders, two pools, two independent draws from the placement lottery. Whichever mode drew the efficiency cores at t=2 looked like it could not use a second core. The new one builds one pool per cell and runs both modes inside it, alternating repetition by repetition. Whatever placement that pool drew, both modes drew it. The nuisance variable is still there and still large, it just stopped varying between the things being compared.
Your benchmark is a courtroom, and thread placement is the judge. Every repetition is a witness, and they all tell the same story because the judge decided the outcome before the trial began. Blocking is the only way to get a fair trial: make the judge rule on both sides equally.
The lesson for inference engineering
This is not a compiler problem. It is every performance measurement problem. When you benchmark a serving engine on Apple Silicon, or any asymmetric-core machine, and the numbers move, check the launching context and the core placement before you touch the code. The standard toolkit, repetitions, medians, confidence intervals, does not defend against this class of error. It makes it look more rigorous.
I did not have fifteen draws from one population. I had one draw from a placement lottery, followed by fifteen confirmations of it.