You change one line in the system prompt and the suite comes back better. Average answer quality is up. The chart in the pull request is green in the place everyone looks. Then the pipeline refuses the merge, and the line it prints is about the support queue: where the assistant decides which team a complaint goes to, the share of replies that follow the escalation rules has dropped. Nothing got worse in general. Something got worse in the one place where being wrong costs you a customer, and the pipeline had been told that place counts for more than the others.

The scoring table decides who wins

A decathlete does ten things and none of them are in the same units. You cannot add ten and a half seconds of sprinting to sixty metres of javelin. The sport solves this with a conversion table published by the federation: every performance becomes a quantity of the same kind, and only then can two athletes be compared. The table looks like bookkeeping. It is not. Somebody chose how steep the curve runs for the sprint against how steep it runs for the throw, and making one of them steeper changes which sort of decathlete wins next season, with no rule of any event touched. Coaches read the table before they plan the winter, because the table is where the sport says what all-round means this year.

Your release check has the same table. It collects things that do not add up on their own: whether the workflow reached the end, whether the reply followed the rules you published, whether it stayed inside the documents that were retrieved, whether retrieval found the right document at all, what the run cost, and the slow tail of latency — the one request in twenty that takes longest, not the typical one. Something has to turn six units into one verdict. That something is a set of weights, and you have one whether or not you wrote it.

A flat average makes the decision anyway

Give every situation the same weight and you have stated that a complaint routed to the wrong team costs exactly what a duller paraphrase in the help centre costs. Written out like that, nobody signs it. Nobody writes it out, which is how it keeps getting signed. The variant in the opening is the ordinary case, not a freak one: mean quality up, compliance down, and a weighted readiness figure that falls under the bar even while a single accuracy number climbs.

Which is also why crushing all of it into one figure is worse than showing the trade. Plot quality against what it costs to operate, and the shape you get is a frontier: past a point, another half point of groundedness is bought with money, or with seconds the caller spends waiting. A frontier makes you say which side you are buying from. An average hides the purchase and hands you a number that went up, which is the most expensive kind of reassurance in this line of work.

Somebody writes the weights, or inherits them

Underneath, the check is not one metric with a bar across it. It is a set of cases, each carrying its own threshold or its own rubric: a planner change has to clear task completion of 0.9 and tool correctness of 0.95 and a check that the arguments it passes match the declared shape. Deterministic code judges what is objectively checkable, a small local model or a judging model handles what is not, and the runner returns something the pipeline can act on. When it fails you learn which property broke, not that the agent is vaguely worse.

The weights, though, belong to whoever is accountable for the product, and they almost never write them. The default arrives from an average nobody argued about, and the first time the check blocks a release people want, the cheapest repair is to lower a weight rather than fix the regression — which is the short path to a number that stops measuring anything. Keep the cases, the rubrics and the weights in version control next to the code, so a change to what counts shows up in a diff and somebody has to approve it. You are always weighting. An average is just the weighting nobody chose.