The Quality Flywheel
On Thursday afternoon a risk specialist reads through the SQL an agent wrote on behalf of a trading desk. She flags a handful of queries: one that would have scanned a table nobody is supposed to scan, a few that quietly rounded a position the wrong way. She explains why, carefully, in three paragraphs. Everyone thanks her. Two releases later the same rounding comes back wearing different clothes. Nothing she wrote was wrong. The afternoon was simply burned as fuel — used once, stored nowhere — and the only way to get it back was to book another Thursday.
A heavy wheel keeps turning between strokes
Sit on a rowing machine and watch the fan. The drive, the part where you are actually pulling, is under a third of the cycle. The rest of it you are sliding forward with the handle loose, contributing nothing at all. The fan does not care. It keeps roaring, because bolted to it is a mass whose entire job is to take a short violent push and give it back across the long stretch where nothing is pushing. That mass is the flywheel. Before the word became a shape on a strategy slide it meant a lump of metal that converts effort arriving in bursts into work that continues.
Expert attention arrives exactly like that. It is short, expensive and irregular, and there is no version of the world where you get more of it. So the question is never how to get the specialist to review more conversations. It is what her judgement is turned into before she stands up: a check that runs on every query, a rubric that turns “this rounding is wrong” into a verdict a program can assign, a set of examples of what a right answer looks like, a failing case bolted into the suite that gates the next release. Those keep grading on Friday, and in August. A review that ends as three paragraphs in a document is a stroke taken with the handle disconnected.
Send her the cases nobody else can settle
Which makes two decisions matter more than the review itself. The first is what reaches her. In production, automated evals and alerts pick traces out of the traffic and put them in a queue for annotation, and the point of that queue is that it holds the ambiguous and the expensive cases. Hand her a random slice of yesterday instead and she spends the afternoon confirming that the easy cases were easy. The second is what leaves her. Not a verdict on those particular conversations, which covers those conversations and nothing else, but the thing that will judge the next thousand.
Notice what she is usually changing. Not the model. A tool description that invited the wrong call, a constraint in the workflow, the context the agent gets handed before it starts — the parts that decide what the next run will do. And keep the examples of good behaviour, not only the failures. A dataset made entirely of things that went wrong teaches an agent what to avoid and tells it nothing about what you actually wanted. How much you are willing to leave running unattended is, in the end, the same question as how far this wheel carries.
The reading comes from the wheel, not the rower
On an erg the number on the screen is computed from how fast the flywheel slows down while you recover. The machine measures the wheel, never your intention, which is why you cannot fake a good split by yanking the handle without loading anything. The same separation is what keeps the loop honest now that most of the pulling is done by another agent: it reads the system, turns a complaint into scenarios, runs the eval and proposes a change, and none of that is a problem as long as the verdict comes from somewhere else.
A travel agent kept losing the user’s revisions — new dates given, old dates repeated in the final answer. The fix started with a rubric named for that one behaviour, scoring whether the revision was honoured, with categorical verdicts rather than a feeling. Now the failure has a number of its own instead of dissolving into an average that looked acceptable. Then the same scenarios run again, one change at a time, or the before and after are not comparable at all. Stop feeding it real traces and the wheel coasts down quietly for a while before the number moves. The point was never that nobody has to pull. It is that the work does not stop between pulls.