You leave the loop running overnight. It reads back the traces of its own runs, notices that the agent keeps finishing with the repository in a dirty state, writes a final check into its own instruction file, and reruns the benchmark. In the morning the number is two points higher than the one you built by hand. The addition is three lines long and you would not have thought of it. The unsettling part is not that the harness edited itself, which is only a program editing a text file. It is that nothing in front of you can say whether those two points are the change or the noise.

The lever is cheap, the verdict is not

A mechanical watch that runs fast is corrected by nudging a small lever a fraction of a millimetre. The nudge is nothing: a moment at the bench, and you can put it back exactly where it was. The hard part is knowing whether you helped. A watch keeps one rate lying flat and another hanging crown down, and a different one again when the mainspring is fully wound, so a reading taken over a minute is not an answer to anything. So the watchmaker clips it to a timing machine that listens to the beat and reports the rate in six positions across a full day. Adjust by feel instead, and after ten sessions at the bench you have a watch that seems right and loses half a minute a day.

A harness is code, so its lever is just as cheap. A proposer can read the source, the execution traces, and the score of every variant already tried, then rewrite the tool descriptions, the summary kept after compaction, and the final check in a single move, because those pieces interact and tuning them separately optimises each against a different idea of good. It can do that a hundred times a night. Proposals were never the scarce thing. The timing machine is.

Nothing enters the harness without an independent test

Which is why the discipline around the search matters more than the search. A proposal starts from a pattern mined out of real runs, not from generic advice about being careful, and it stays as small as the pattern allows. It then runs against a benchmark and a holdout, and it is adopted only if it improves the cases it was aimed at without degrading the rest. What the loop is not allowed to do is grade its own homework.

That constraint is also the honest answer to what you let it change unattended. Self-modification is safe exactly as far as your tests can catch a step backwards, and no further. It matters in the other direction too, because every instruction that no observed failure asked for is weight the agent carries into every future run: more to read, more ways to thrash between tools. A harness earns its parts by removing the ones that stopped paying, which is hard to do by taste and easy to do with a number you trust. And the number has to be per model. A line that earns its place for one is dead weight for the next, which is why the holdout includes models the search never saw.

The same instrument is what you reach for when quality drops and nobody touched the model or the serving stack. The cause is usually three harness changes that shipped in the same week: a lower default reasoning effort, a caching flag that dropped the thinking history in sessions left idle, a short instruction about being concise that quietly shortened the reasoning between tool calls. The aggregate signals look incoherent because each change hit a different slice of traffic. You get to the cause by segmenting users, versions and configurations, then rolling changes back one at a time until the case reproduces or stops.

That is the same work the overnight search needs, done by hand. Both depend on runs being comparable at all, which they mostly are not, and on a suite wide enough that a long enough search does not simply find the shape of the benchmark. Build the instrument and the search is worth running. Skip it and you have automated the production of confident, unverifiable changes. A harness can improve itself only as far as you can prove it has not got worse.