You change one setting on the support agent and you want to know whether it helped. So you write six test cases, run them, and all six come back green. They come back green because you wrote them: each one is a question you already knew the agent could answer, phrased the way you would phrase it. Meanwhile, at two in the morning on Tuesday, somebody typed half a sentence with an order number stuck in the middle, the lookup tool returned an error, and the agent apologised for the wrong thing. That conversation is sitting on disk, complete. It is a harder test than anything you would have invented, and inventing it was never your job.

Nobody gets to choose the position

This is what a serious chess player does with a game they lost. The score sheet holds every move in order, theirs and their opponent’s, including the one they would rather forget. They set the pieces back to move twenty-three, the position where it turned, and play something else from there. The analysis prints a number for the new line and a number for the old one, and the two get put side by side. What makes the exercise honest is that nobody designed the position. It arrived out of a real game, against a real opponent, on a real clock, and it is usually uglier than anything a textbook would set you.

A trace platform gives an agent run the same treatment. The runtime exports its traces to a server, and the interface lays out what the run did rather than what you assume it did: reasoning steps in sequence, each tool call, what came back from it. From there a playground plays one traced call again with different parameters — another model, another setting — and attaches metrics to the outcome. You pick the trace where the tool failed, send the same prompt through with the setting changed, and compare the evaluation against the previous version. Same position, different move, two numbers to read.

Your traces hold what people actually wrote

Which makes the dull part of the setup a decision rather than a default. Production traces are not synthetic. They carry what your users really typed — the order number, the complaint, the paragraph somebody pasted in without thinking — and playing them again moves all of it to wherever the platform lives. Self-hosted means the runtime exports to a server you run, and traces, datasets and experiments stay inside your own data plane. That is not a footnote for whoever administers the box. It decides whether the cases you most need to replay are the same cases you are not allowed to keep.

The other dull part carries just as much weight. A trace on its own lets you watch a failure; it does not let you fix one. Store the trace without the dataset and without the version of the prompt that produced it, and you are the player who wrote down only his own moves — he can see that he lost and he cannot set the game back up. Kept together, the failure stops being an anecdote and becomes a case you can run on demand, which is the only form in which a failure is worth anything.

A file turns replays into a gate

One replay tells you about one Tuesday. The change comes when the awkward cases stop being something you open by hand. An evaluation suite declared in YAML puts the whole arrangement in plain text: which provider to call, what input to send, what behaviour is expected, and the assertions that decide pass or fail. A runner walks the matrix, collects the metrics, and fails the pipeline when a regression crosses a threshold you wrote down. A red-team variant of the same file goes hunting for known vulnerabilities instead of known answers.

Now join the two. A pull request edits the instructions of a support agent, and continuous integration runs ten cases through the suite while a judge model checks that no reply exposes confidential data or walks past the policy. Those ten came out of the traces — the two-in-the-morning ones, the ones where a tool broke — so the gate defends against what already happened, not against the handful of prompts you picked yourself. Leave the judge alone with no deterministic cases and no explicit thresholds and the suite starts wobbling, passing and failing without naming what regressed. Traces hand you the cases you would never have chosen. The file is what makes them come back every time.