You hand the same model the same task on the same repository, twice. The first time it runs the tests, reads the failure, opens the file the traceback points at, and edits it. The second time the context gets compacted halfway through, the path to that file goes with it, and the agent carries on anyway: a fluent account of what it changed, and an artifact nobody can check against the test that was failing. Same weights, same prompt, same repository. Everything that moved was the software around the model, and you were one meeting away from blaming the model.

The delta belongs to the configuration, not the weights

That second run has a name. It is an execution-alignment failure: the reasoning stays plausible while it comes loose from what the tools actually returned, from the state of the workspace, from the shape of the output that was asked for. It reads like competence because the sentences are fine. Nothing in the transcript is wrong except its relationship to the world.

Think of the wattage printed on a solar panel. It is not what the panel will make on your roof. It is what it made under a flash tester, at one fixed light intensity and one fixed cell temperature, and those two conditions are printed beside the figure because without them the figure means nothing at all. Move either one and the number moves with it while the silicon stays exactly as it was. Two panels can be compared only because both were measured under the same declared conditions.

An agent’s setup is how context is handled, which tools exist, how state is carried, what permissions are granted, what gets traced, and what happens after a failure. To compare two of them honestly, hold the task, the budget and the protocol common, then let each one behave the way it natively behaves. Force both into a single foreign shape and you have measured your adapter instead of either system. And do not stop at whether the run finished. The trace, the cost and the validator output are where the difference actually shows.

Past a point, more resources measure a different agent

The next layer down is the one people call plumbing. CPU, memory, hard limits, reservations, time limits, how many runs share a machine: these belong in the write-up as declared experimental variables, with a floor that guarantees the work can happen and a separate ceiling that kills whatever runs away.

One study set memory reservation at 1x and the cap at 3x. At 1x, some containers died on transient spikes, and those deaths looked like task failures. At 3x, the infrastructure errors fell away and the success score stayed inside the noise. At 5x, the agents started installing entire data science stacks, and the score went up for real. That is the threshold worth naming. Below it, headroom buys you a cleaner measurement of the same agent. Above it, headroom buys a different agent, one that can afford strategies the leaner one never had. Turning up the lamp until every panel looks good measures the lamp.

This is also where the money question gets answered. A timeout or a killed process is not evidence about reasoning, and swapping the model is an expensive way to fix a compaction bug.

Even the most quoted figure in this business, the percentage of real repository issues an agent resolves, is an outcome metric. Its families differ on purpose: the full set, the human-filtered one, the cheap subset, the multilingual and multimodal variants. Reading a score from one against a score from another as though they were the same exam is a category error, and none of them prove anything about efficiency, cost or safety on their own.

The honest version of this admits uncertainty in the verdict itself. A regression check can project traces into behavioural fingerprints rather than a single yes or no, spend trials until the evidence is sufficient, and return INCONCLUSIVE when it is not. That verdict is not a broken test. It is a refusal to hand you a pass you did not earn.

You never measured a model. You measured a model wearing something.