Long automated sweeps fail in boring ways. An engine wedges. A device is left dirty by the previous trial. A run that should take ten minutes is forty minutes in and still going. Watching for that is a good job to hand to an agent, because it is exactly the kind of pattern-matching over logs that people are bad at sustaining for fourteen hours.
The question is what the watcher is allowed to touch, and I think one line matters more than all the others.
It may read anything, and it may change what runs. It may not change what is recorded. Logs, device state, the run database: all readable. Pause, kill a track, requeue, escalate to a human: all allowed. Writing a measurement, editing a result, or choosing which configuration won: never.
The reason is the cost of being wrong. A supervisor that can only affect scheduling has a worst case of wasted time. It kills a healthy run, you re-run it, you are out twenty minutes. A supervisor that can write to the evidence has a worst case of a corrupted ranking that looks exactly like a good one, that you ship, and that you keep believing until something downstream stops adding up months later.
Those two failure modes are not on the same scale, and the boundary between them is cheap to enforce.
The ranking path should have no judgment in it
The argument extends past the supervisor. If a sweep's job is to say which configuration is best, the path from measurement to ranking should be arithmetic and nothing else, and it should be reproducible from the stored results without re-running anything.
That gives something up, and the trade is worth stating. A model can notice that a run looks anomalous in ways no threshold catches, and you lose that. In exchange, every number in the final table traces to a row in a database and can be recomputed by someone who does not trust you. For a result that decides how a fleet is configured, I will take the second one every time.
Two things that break this anyway
Both are cases where the process is honest and the output still is not what it claims to be.
Constants that have never met a measurement. An analytical model that predicts throughput from a layout is enormously useful for pruning. It is also a model, and if its constants came from datasheets and reasoning rather than from a fitted measurement, every number it produces is a hypothesis. Anchoring an improvement claim on it makes the claim a difference between two hypotheses. One live run against real hardware, before anything downstream treats the output as a baseline, is the whole fix.
A screen rung too cheap to be a filter. Cost-ordered funnels are the right structure: a short test on everything, keep the top quarter, longer tests on those. The failure is when the cheap rung is not measuring the same thing as the expensive one. Eliminating three quarters of the candidates on 180 seconds, at one profile and two rate points, with no check that the ranking at 180 seconds agrees with the ranking at ten minutes, is not a saving. It is an unmeasured risk sitting at the widest point of the funnel, where a mis-rank throws the winner away before anything else runs.
Same species of error, one level up: a single global latency target applied identically to a 2,048-token chat turn and a 24,576-token retrieval prefill is answering two different questions with one number.
What earns trust in a small result
All of this matters because the interesting wins are small. A harness that can only resolve 10% will only ever find the wins that were obvious anyway. To believe a 1.4% result I want:
- Paired arms, interleaved. Baseline and candidate alternating within one session on one device, not one after the other. Thermal state and clock behaviour drift across a session and will happily manufacture a percent for you.
- Enough repetitions to see the spread, and the spread reported rather than the mean on its own.
- A correctness gate on every arm. Not once at the end, every one. An optimisation that is faster and slightly wrong is the most expensive thing you can ship, and bit-exactness is more often achievable than people assume.
And the negatives are part of the cost of the method. One round I ran was four already-implemented flags, paired arms, about forty minutes, and every one came back null or worse. That is a real result and it was worth the forty minutes, because the alternative is those four flags staying on the list of things somebody might try, for the next year.
Let the supervisor cost you time. Never let it cost you the measurement.
Sources
- Producing wrong data without doing anything obviously wrong: measurement bias large enough to reverse a conclusion, from causes nobody controls for.
- Rigorous benchmarking in reasonable time: how many repetitions you actually need, and where to spend them.
- MLPerf inference rules: what a benchmark has to pin down before a number is comparable to anyone else's.
- Lossless has conditions: the same insistence that a claim names the conditions it holds under.