NEW YORK — What happened
On September 19, 2026, a team of independent researchers from the University of Veritas published a bombshell paper detailing extensive evidence of data leakage affecting the BigBench Hard (BBH) benchmark. BBH, widely regarded as a critical test for advanced reasoning in large language models (LLMs), appears to have had many of its constituent tasks, or closely paraphrased versions thereof, present within the vast training datasets used by leading AI labs. This presence, the paper argues, allowed models to effectively 'memorize' answers or solution patterns, artificially inflating their performance scores.
Why it matters
BigBench Hard has been a cornerstone for evaluating the reasoning capabilities of frontier AI models. Many of the impressive performance metrics and 'breakthroughs' announced by major AI companies over the past two years have heavily relied on BBH scores. If these scores are indeed compromised by data leakage, it means the perceived progress in AI reasoning might be significantly overstated. This scandal not only undermines trust in specific model claims but also highlights a systemic problem in the creation and curation of benchmarks, potentially slowing genuine progress by misdirecting research efforts.
Deep dive
The researchers employed sophisticated data fingerprinting techniques and large-scale dataset analysis to identify specific task examples from BBH within publicly available web scrapes and other datasets commonly used for LLM training. They found instances ranging from direct copies of questions to highly similar paraphrases that an LLM could easily recognize. The paper meticulously details how this leakage could lead to 'performance without understanding,' where models achieve high scores not through genuine reasoning but by recalling patterns seen during training. This phenomenon is particularly insidious because BBH was designed to be difficult and to require complex reasoning, making 'memorization' an unexpected failure mode.
Report check
The evidence presented in the University of Veritas paper is robust and has been largely confirmed by preliminary independent audits commissioned by several AI ethics organizations. Major AI labs whose models previously scored highly on BBH have acknowledged the findings and are initiating their own internal investigations, with some expressing 'deep concern.' The full extent of the leakage's impact across *all* BBH tasks and *all* frontier models is still being quantified, but the core claim of significant leakage is now widely accepted. Rumors circulate about specific models being disproportionately affected, but these remain unverified details.
Open questions
What is the true reasoning capability of current frontier models if their BBH scores are discounted? How can the AI community create future benchmarks that are genuinely leakage-proof given the massive scale of training data? What does this mean for previous claims of AGI progress, and how will industry leaders rebuild trust in their evaluation methodologies? The scandal forces a painful but necessary re-evaluation of how we measure AI intelligence.