Eleven tasks out of sixty come back failed. You open three of the traces and find the same move every time: the agent writes a file, renames it, then goes looking for it under the old name. You add two sentences to the tool description. The next run passes fifty-four, and you are pleased for about a day — until you notice that one of the tasks that started passing finishes in nine seconds and never opens the repository at all. The grader was looking for a phrase in the final message. The agent produced the phrase. Your suite has been handing out credit for saying the right thing.

You learn from the corner left blank

A crossword setter learns nothing from a puzzle that got solved. The useful report comes from the person who stopped: the corner left empty, the twenty minutes lost on one entry, the word written in and then scratched out. That is where the setter goes to work. And no repair is local, because the entries cross. Rewrite one clue to open the stuck corner and you have handed its neighbours away for free. So the setter changes one clue, says beforehand which entry should come unstuck, and waits for the next read-through to show whether it did.

The loop around an agent runs the same way. You record a baseline, then spend your attention on the runs that failed, reading the reward and the trail together instead of the average. You change one thing: a prompt, a tool description, a piece of the surrounding logic. Because each piece is its own file under version control, the edit is one line in a diff and undoing it is one command. And you name, before the run, the tasks you expect to flip. Then the run flips them or it doesn’t. A change that fixes what it aimed at and quietly loses two tasks elsewhere is not a change you keep, and the note explaining why is worth more than the score that came with it.

Somebody has to solve it before publication

That job has a name. A puzzle goes to a test solver before it is printed, and their notes are not about taste. They catch the clue with two defensible answers that both fit the crossing letters. They catch the one that works only if you already think the way the setter thinks. They catch the entry nobody will ever get, and the puzzle the regulars finish over breakfast without pausing, which is pleasant and tells the editor nothing. And they catch errors in the solution itself, the slip in the printed key that informs everyone who got it right that they got it wrong.

Your suite needs the same reading and almost never gets one. Tasks drift into ambiguity. A grader marks a correct answer wrong over a rounding difference, and you lose a week tuning a prompt against a bug that was never in the agent. Check the verdicts against what actually happened in the runs and you find passes that skipped the work entirely. So you read transcripts instead of aggregates, calibrate the rubric and the judging model against people, retire the impossible tasks and the brittle grading, widen the set when everything passes, and file every production failure back in as a regression case. It is maintenance work with named owners, and it is the difference between a number about your agent and a number about your suite.

The puzzle outlives the hand that solves it

None of this requires opening the agent up. You judge the solve, not the solver: the inputs it was given, the environment it worked in, the trail it left, the artifacts it produced. The target can be a command-line tool, a service, a product you cannot see inside. The clever parts — the multi-turn prompting, the simulated tools, the model doing the grading — belong to the evaluation, not to the thing being evaluated, which is why the same suite still stands when the agent is upgraded or swapped for a competitor’s. It is yours, and it outlives what it measures.

Which is also how it decays without anyone noticing. An overnight loop can propose changes faster than you can read them, and a long enough search finds the shape of the benchmark instead of the shape of the work. The one thing keeping the set honest is the stream of real failures walking back into it. A suite that stops taking them in ends up agreeing with the person who wrote it.