Two agents come back from the same suite with the same result: seventy percent of tasks passed. The first one fails rarely, and when it does it fails the way it failed last time — the wrong file, a test left red — and you put it back in a minute. The second one reaches the answer by a different route on every run, and twice this month it deleted a directory it had been asked to read. Same score. You would leave one of them working overnight and stand behind the other with a hand near the keyboard. Whatever you were actually measuring, the number was not it.

Two panes carry the same load

A glazier ordering a pane for a stairwell does not ask one question. Load rating is the figure on the invoice, and two panes can share it and still belong in different buildings. Ordinary annealed glass breaks at a point that wanders from pane to pane, because it depends on flaws along the edge that nobody can see; toughened glass gives way much closer to the same load each time. Then there is rough handling — a chipped edge, sun on one face and cold on the other — and the two respond nothing alike. And there is what arrives on the floor. Annealed comes down in long splinters that travel. Toughened crumbles into blunt granules. Laminated glass fails and stays stuck to the sheet of plastic inside it, so the pane is finished but the hole never leaves the frame.

Four separate questions, then: how repeatable the pane is, how it takes being knocked about, whether the moment of failure can be foreseen, and how bad it is when it comes. The specification is a row of entries, not a single figure, and where the pane is going decides which entry you read first. An agent has the same row. Most teams read one entry and ship.

You deploy the failures, not the successes

The four entries have names. Consistency is what you learn by giving the agent the identical task again and watching whether the behaviour comes back the same. Robustness is what survives when you change something that should not have mattered — a renamed field, a slower tool, the request phrased another way. Predictability is whether the failures land somewhere you can describe and explain afterwards, or anywhere at all. Safety, here, is not about what the agent says: it is how severe the damage is and how far it spreads, which is the whole distance between a wrong answer and a dropped table.

Put the two agents back on that row and they stop looking alike. The first is reversible and boring, and boring is a property you can build on. The second is a system whose worst case you have never seen the edge of. Neither one needs a better model to change position — the second needs its destructive tools behind a gate and a smaller room to work in, which is a decision about how much rope you give it rather than about capability. New capability arrives on its own. Reliability does not come with it.

Nobody hands you the four numbers

You have to build them, and the building is unglamorous. It starts with a task stated so plainly that two people would agree on whether it was done, with a reference answer written down. Then a dataset that keeps the levels apart — a single run, a full trace, a whole conversation — and that carries cases where the right behaviour is to refuse, not only cases where it is to succeed. Graders get chosen for what they are judging: code checks the facts that are objectively checkable, a person handles the genuinely ambiguous, and a judging model is used for quality only after it has been calibrated against people. Offline checks run in the pipeline and block regressions. Online checks watch real traffic, and every real failure becomes another case in the dataset.

Skip that and the row of entries is guesswork, and one figure is all you can honestly report — which is the same corner accuracy walks you into when nobody asks what it is a fraction of. Success rate tells you how often an agent works. Reliability is the shape of what happens when it does not.