Debugging a Run You Weren't Watching
The overnight run finished while you were asleep. Nine agents, four hours, one report on contract renewals, and one figure in it is wrong by a factor of ten. The obvious move is to open the agent that wrote the report and tell it to be more careful with numbers. But it was careful. It added up precisely what it was handed, and what it was handed had a field missing that a check three steps upstream was supposed to catch. Rewrite that agent’s instructions and the figure will still be wrong tomorrow, in a different place, from a different run. The report is where the error surfaced. It is not where the error happened.
A blackout gets read backwards, not forwards
Half a region goes dark on a hot afternoon. The loudest event is the last one: a large plant disconnects, and that is what everybody sees. Investigators do not start there. Every switching station keeps a time-stamped record of what its protection did and when, all of them running off the same time source, and those records get merged into a single ordered chain. Then the chain gets walked backwards. This circuit opened because it was carrying more than its rating. It was carrying more because that other one opened four minutes earlier. That one opened because a conductor sagged into a branch nobody had cut. The chain ends in something small and early, and the repair goes there — to the cutting schedule — not to the last device, which did exactly what it was built to do.
A multi-agent run can be read the same way. From the recorded traces you build a causal graph: agents, tool calls, the data passed between them, the events, and which step waited on which. Starting at the visible failure you walk the graph backwards and rank the candidates by structural signals a person can check. The wrong figure in the report gets labelled a downstream symptom, and the cause resolves to the validation step that accepted an incomplete artifact from an extractor. The fix goes to the validator. None of that analysis needs a model call, so it is fast and it gives the same answer twice, which means you can afford to run it on every failed run instead of the ones you have time for.
Stopping the run beats reading the log
Reconstruction after the fact carries you a long way, and then it stops. Some questions only answer themselves while the run is still moving. A trajectory debugger presents the run as what it actually is — a structured exchange between agents, model calls and tools, with the repository diffs each step produced — and lets you halt it at a breakpoint before a suspicious call, inspect the state, hold the prompt next to the tool’s declared schema, correct the argument by hand, and continue one step at a time. One rule keeps this honest: the original input and your edit both go into the trace. Skip that and you have produced a run nobody can reproduce, including you, and the recording you were relying on now quietly lies about what happened. How much of a run you are willing to leave unattended is really a question about whether a failure leaves you anything to read, which is half the cost of splitting work across several agents.
A patch counts only once it holds
The other pressure is volume. A long-running agent generates more trace in ten minutes than anyone will read, so the trace becomes something you query rather than something you scan. Pull the last five runs as JSON, hand them to an assistant, and ask what is being repeated: it comes back with a section of the system prompt that is inducing duplicate retries. You change the section and compare the traces of the new version against the old. The constraint that keeps this shortcut sane is that the structured traces stay, and stay verifiable. Without them you are collecting explanations that sound right and rest on nothing.
Then the repair, kept deliberately separate from the diagnosis. One stage adds diagnostic probes, one reads the runtime traces and localises the cause, one prepares the change. The patch is accepted only as a verified improvement: if the target test passes but the regression suite gets worse, it is rolled back and the lesson is kept so the next attempt does not walk into it again. The last error tells you where a run stopped. Only the recording tells you where it went wrong.