The suite comes back green. The agent was asked to edit a document and it edited the document, so you write down the pass and move on. After lunch a colleague runs the same task on the same fixture, changes nothing, and watches the agent finish the edit and delete a file it was never asked to touch. Same task, same starting state, a different afternoon. You want to call the first run the real one and the second one bad luck. But you ran the test twice and got two answers, which means the test was never telling you what you thought it was.

The lab plants four hundred, not one

A seed testing laboratory does not certify a lot by planting one and seeing what happens. The rules hand it a number: four hundred, split into four separate batches of a hundred, each sown on its own damp paper, each held at a prescribed temperature, each counted on prescribed days. The figure that reaches the certificate is the share that came up as whole, normal seedlings. And the four batches have to agree with each other. If they scatter further apart than the tolerance allows, the result is not a low score, it is void, and the test runs again from the start. Disagreement between trials is not an inconvenience on the way to an average. It is the finding.

A repeated-trial benchmark for agents works this way. Each task carries fixtures, a rubric, and an environment where success can be checked; the runner executes a fixed number of independent trials and credits the point only if every one of them meets the criterion. Three trajectories, three passes, or no point. Completion, safety and robustness stay separate columns, so the run that finished the file and the run that took an unauthorised action never blend into a partial success. Each trial starts from fresh fixtures rather than the residue of the last one, and nobody gets to rerun only the trials that went badly.

The other five percent has contents

Once trials come by the hundred, a safety property stops being a verdict on one run and becomes an assertion about a distribution. A suite describes the adversarial attacks, the benign failures and the categories of harm; the pipeline runs the trials with controlled inputs, collects the outcomes, and asserts something like no forbidden action in ninety-five percent of trials. That reads like a pass. It is also a written statement that one trial in twenty did the thing you named forbidden.

Which makes the uncomfortable question unavoidable, and it is the question the laboratory is not allowed to skip either. The seeds that never came up get sorted into dead, hard and dormant, and the seedlings that came up deformed are counted apart from the whole ones, because a lot that fails by rotting and a lot that fails by waiting are not the same lot. A refusal that arrived three turns late and a credential sent to an outside host are both a failed trial. Averaging them is how a release gate goes green over the one behaviour it existed to catch, so the trial count gets recorded and the harm categories stay apart.

One agent, four environments, four separate scores

Repetition alone is still not the measurement, because where you repeat it counts too. A multi-environment benchmark puts the same agent against an operating system, a database, a website and a codebase, each domain with its own initial state, its own tools and its own observable definition of done, each running in an isolated container. An agent that sails through a shell and drowns on a database does not have a score, it has two, and the average of the two describes nobody. The laboratory knows this: some species go on paper and some in sand, some need warm days alternating with cool nights, and a lot certified on paper still has to come up in a field.

The repetition pays you back once more. A benchmark built on repeated trials keeps the full traces, so on the morning one trial in three fails you are not staring at a red cell — you already hold the session, the graph of steps, the nested calls, what each one cost, and the two runs that passed sitting beside it for comparison. You kept the failure instead of having to reproduce it, which is the whole difference between a spread you can look at and a number that moves for reasons nobody wrote down. An agent that passes one time in three has not passed. It has a variance, and you either measure it or you ship it.