You write a skill for the SDK your team maintains, and while writing it you keep three failures in mind: the misconfigured client, the retry that swallows the error, the pagination that stops one page early. You finish, you point the agent at those three, and it fixes all three. That reads like proof. It is closer to a receipt. Those three cases shaped every line you wrote, so the skill fits them the way an answer fits a question you already had in your hand. Nothing in that run tells you what happens on the fourth failure, the one you never thought of while writing.

A bridge is measured loaded and unloaded

Before a new bridge opens, nobody asks whether it looks sturdy. Gauges go under the empty span and someone writes down a first reading. Then trucks of known weight park at positions the standard specifies, not positions the builder likes, and the same gauges are read again. The number that matters is the difference between the two readings, and it means something only because nothing else changed in between: same structure, same instruments, same afternoon.

A skill benchmark has that shape. You choose tasks that reproduce real failures but end in an outcome a machine can check, a bug with tests written in advance, so nothing rests on how the answer reads. Every trial starts from an identical sandbox: same repository, same dependency versions, same starting commit, in a container built fresh. Then you run the same agent over the same tasks twice, once with no skill loaded and once with the skill, and you compare. It is the plain controlled comparison every benchmark rests on, pointed at your own library instead of at a model. The usual way to ruin it is to let the two runs differ in anything besides the skill, a repository at another commit, a lockfile that drifted overnight. Then you have two readings of two different bridges.

Task success alone hides the expensive failures

Completion is one column, and on its own it is a poor one. Record whether the skill was invoked at all, and whether the one invoked was the right one; record how many turns the agent took and how long it ran. A skill can be correct and never retrieved, because its description never matched the way the request was phrased, and then your delta is zero for a reason that has nothing to do with its contents. That is the same problem as making a tool legible to a model, one layer up.

Run the mirror case too: a prompt where the skill has no business firing, checked for exactly that. False activation is not free. It spends context and tilts the agent toward a procedure that does not apply. Then vary the configuration, not just the task. Compare the empty baseline against everything loaded, and against a few consolidated groups, because splitting one body of knowledge into many near-identical skills degrades retrieval rather than improving coverage. The traces and the artifacts left behind are what explain a delta once you see it. Together they tell you how many skills this model can actually pick from, which is the number that decides what you are willing to leave running unattended.

Someone else should choose the test cases

The cases you write yourself will always lean toward the cases you already handle. This is why shared, published workflows are worth more than private examples. Open security research frameworks declare a taskflow in a plain configuration file: global parameters, tasks marked as required, an agent personality, and a toolbox that can reach external tools. Each task starts with fresh context and hands its output forward as a structured artifact through something like a key-value memcache, so one step can be rerun on its own rather than by replaying a conversation. A variant-analysis flow reads a published advisory, identifies the vulnerable file and function, stores a summary, then pulls only the relevant lines and goes hunting for the same bug elsewhere. The engine and the suite of flows ship as separate packages, so anyone can reuse a toolbox, audit the behaviour, or add an audit of their own, and a flow that does not state which tools and confirmations it requires is not publishable.

Those flows are trucks you did not park. Until the cases come from somewhere outside your own head, coverage is a claim, not a measurement.