A headline goes by: a new model “detects fraudulent transactions with 94 percent accuracy.” It sounds like a settled matter, a grade stamped on a report card. Nobody in that sentence mentions how common fraud actually is in the data the model was tested on. If fraudulent transactions make up 3 percent of the total, a system that never once flags anything as fraud, that just answers “legitimate” every time, would already be right 97 percent of the time. The 94 percent in the headline is, by that measure, worse than doing nothing.

What the number is actually counting

Accuracy is the simplest of the standard measures: the share of all predictions the model got right, correct calls divided by total calls. It sounds like the obvious thing to report, and for some tasks it is. But two other measures usually matter more, especially when the thing being detected is rare. Precision asks, of everything the model flagged as positive, how much actually was. Recall asks, of everything that was actually positive, how much did the model catch. A model can have excellent accuracy while having recall so poor it’s nearly useless for the one job it was built to do, because getting the common case right over and over quietly drowns out how badly it handles the rare case.

This is where base rate does its damage. A weather forecaster in a desert town can announce “90 percent accurate forecasts” and sound like a meteorological genius, until you learn it’s sunny there nine days out of ten regardless of any forecasting at all. Predicting “sunny” every single day, without checking a satellite image or a pressure map, without any actual skill, lands on that same 90 percent. The number only becomes informative once you know what a forecaster who did zero work would have scored. Below that baseline, the model is actively harmful. At that baseline, it’s decoration. Only meaningfully above it has anything been demonstrated.

Where this misleads in practice

The trap is that 90 percent, 94 percent, and 61 percent all sound like they occupy the same kind of achievement, high numbers reading as strong performance regardless of context. A model that hits 61 percent on a task where random guessing among five options would score 20 percent has shown real capability. A model that hits 90 percent on a task where always guessing the majority class scores 89 percent has shown almost nothing. Yet the press release only ever leads with the first number, never the second, because the second number is the one that would let you judge the first.

Rare-event detection is the sharpest version of this problem: rare diseases, rare fraud, rare security breaches, rare defects on a production line. Precisely because the event is rare, the base rate of “never flag it” is already high, so accuracy stops distinguishing a genuinely capable model from one that has simply learned to say no. This is also where precision and recall pull apart from each other and from accuracy. A model tuned to flag anything remotely suspicious can achieve high recall (catching nearly every real case) while its precision collapses (most of its alarms are false), and a single accuracy figure smooths right over that trade-off, reporting one number where two conflicting stories are happening underneath. A benchmark score is only ever a stand-in for a specific test population and a specific scoring rule, and accuracy inherits every one of those limits without advertising them.

The three questions that vanish

A number without a denominator is marketing. Ninety percent of what, measured against what alternative, on what population of test cases: these three questions are exactly the ones that get dropped between the research paper and the press release, and they are exactly the questions that decide whether the number is a genuine result or a restatement of how the test was built. Before repeating any accuracy figure, however confident the source, it’s worth asking what a lazy guesser would have scored on the same test. If nobody has answered that, the number hasn’t actually said anything yet.

For the fuller picture of what that test even is, What Is a Benchmark? A Standardized Exam, Not Reality is worth reading first. And since the same model run twice doesn’t always return the same score, which makes the comparison problem even messier, the next entry in this series, Reproducibility and Variance, picks up right where this one leaves off.