You point an agent at an internal admin page and ask it to flip one setting. Working from a rendering, it finds two grey rectangles side by side, same size, same font, same rounded corners. It picks the left one and reports success. Nothing changed: the left rectangle was a heading styled to look like a control, and the real Save button was the other one. Then you have it read the structure of the page instead, and the ambiguity is simply gone. One is a button named Save. The other is text. The model did not get better eyes. The page was already saying which was which.

A key is cut from its code

Bring a locksmith a photograph of a key and they can trace the outline and grind something close to it. It will turn in three locks out of four and stick in the one you needed. Bring the code instead, the depth of each cut written as a short run of digits, and the copy opens the door the first time. The code is not a picture of the key. It is a description of it, written for a machine that will never see the original.

A page carries the same kind of code. Underneath what gets drawn, the browser keeps a semantic description of every element: a role, so button and link and checkbox are different things rather than similar shapes; a name, which is the text a person would hear read aloud; a state, such as disabled or checked or expanded. It exists so that software which never renders the page can still say what is on it, which is how a screen reader speaks one. An agent asks for a snapshot of that description and works inside it. Find the element whose role is button and whose name is Submit, then act on that element. Not on a coordinate measured off a rendering, and not after deciding whether two grey shapes are two controls or a control beside a label.

Ambiguity is the thing you pay for

The semantic snapshot is also the cheaper input. It is text, where a rendering is an image that has to be resized, encoded and paid for at every step. And it drops a requirement most people forget they took on: a model with no vision at all can drive a browser this way, which widens the set of models you are allowed to put on the job.

There is a limit, and it is worth knowing before you trust a run. The description is only as complete as the page. A drawing surface, or a custom widget assembled from unlabelled containers, arrives as nothing you can address, and an agent that assumes otherwise will report success on something it never touched. That case needs a different strategy, chosen on purpose rather than discovered at three in the morning.

When the trouble belongs to the page rather than the agent, a screen that takes six seconds to settle, the same instinct applies. You do not look harder at the rendering. You ask the browser what happened: which requests went out, what the console said, what a performance trace blames for the wait. Those answers come back structured too. They also come from a live session carrying whatever accounts are logged into it, so handing one over is a decision about trust, not a testing convenience.

The terminal has no such description

The same problem exists where there are no pixels at all. Ask an agent to drive an interactive program in a terminal, an installer asking questions, a database client, a language shell, and there is no semantic layer to read. There is a pseudo-terminal, a grid of characters, and whatever the program chose to draw in it. Here the description and the picture are the same object.

So the discipline has to be rebuilt by hand. Start the program in a named session that survives between calls, wait for a specific pattern to appear, read the screen, type, press a key. The tempting shortcut is to send the whole sequence with fixed pauses, and it works until one unexpected prompt shifts everything by a step and a password goes into a field that echoes it. A timeout is a guess about the program. A pattern you waited for is a fact about it.

That an agent has to look again between actions is the subject of When the Agent Has to Look. What is at stake here is what it looks at. A rendering tells you what the page resembles. The description tells you what it is.