A few years ago, researchers combing through a widely used reasoning benchmark noticed something odd: several of its exact questions, word for word, along with their correct answers, were sitting on public forum threads and study-guide sites. Nothing secret about it. Someone had posted the questions to help other people study. Which would be a fine thing for a person to stumble across before an exam, and a much stranger thing for a language model to have absorbed before being tested on that same exam.

That’s the core of what people mean by data contamination. It’s not a hack, and it’s not cheating in any deliberate sense. It’s a byproduct of how both benchmarks and training data actually work, and once you see the mechanism, you start reading every leaderboard a little differently.

How the answers leak

Benchmarks are not kept in a vault. They’re built specifically so that researchers and the public can inspect them, discuss them, and try them out. Test questions get quoted in academic papers. Enthusiasts post them on forums to compare notes. Bloggers write “I tried this famous AI test myself” pieces that include the full question and their own worked answer. Tutorial sites reproduce them for teaching purposes. All of that text lives on the open web, and the open web is exactly what gets crawled and folded into the training data for the next generation of models. Nobody has to smuggle the test in. It arrives the same way everything else does, riding along in a crawl of ordinary web pages.

Picture a school that reuses the exact same final exam every single year. At first that’s harmless, nobody outside the room has seen it. But over time, copies circulate: an older sibling remembers a few questions, a tutoring center keeps a file of past papers, someone posts the answer key online to be helpful. Each new class, without any of them cheating in the moment, walks in having absorbed a little more of the answer key than the class before. Scores climb year over year, and on paper it looks like the students are getting sharper. What’s actually happening is that the exam has quietly stopped measuring anything real, because a growing share of the room already knew the answers before the papers were handed out.

Why this matters more than it sounds

The consequence isn’t that benchmark scores are meaningless. It’s that they stop being clean measurements of the thing they claim to measure. A model can score well on a coding benchmark partly because it’s a genuinely capable coder, and partly because a chunk of that benchmark’s test cases, or text close enough to them, was sitting in a GitHub repo or a blog post that got scraped months earlier. Untangling how much of the score is real capability and how much is memorized recall is difficult even for the people building the model, let alone for someone reading a press release about the number. Two models can post similar scores on the same benchmark for entirely different reasons, one solving the problems fresh, the other partly recognizing them. From the outside, both look identical: a number, printed with confidence, standing in for something much messier underneath.

The fame problem

Here’s the part that makes this hard to fix rather than just annoying. Contamination isn’t randomly distributed across benchmarks. It concentrates in exactly the ones that matter most to the public conversation, the famous ones, the ones every lab wants to cite, the ones journalists quote in headlines. A benchmark becomes well known precisely because people write about it, discuss it, quote its questions, argue about its answers. That popularity is what makes it valuable as a shared yardstick, and it’s also what spreads its exact contents across the same web text that future models are trained on. The more a benchmark succeeds at becoming the industry’s reference point, the faster it seeds its own answer key into tomorrow’s training data. Fame is less a risk factor for contamination than a near-guarantee of it, on a timeline set by how quickly people talk about the thing that made it famous in the first place.

If you want the fuller picture of how that training data gets assembled in the first place, Training Data: Where All That Text Actually Comes From covers the raw material this whole problem depends on. And since contamination is really just one flavor of a broader failure mode, Goodhart’s Law: When the Score Stops Measuring Anything is a natural next stop.