NEW YORK — What happened: The release of OlympiadBench 2.0—designed to test advanced, graduate-level mathematical proof generation—was met with immediate saturation as three major labs simultaneously posted scores above 92% using test-time compute scaling techniques.
Why it matters: Traditional benchmarks are losing their diagnostic value. When a suite of tests specifically built to stump the world's best AI systems is beaten in under a month, the AI community faces an evaluation crisis, making it nearly impossible for buyers and safety regulators to differentiate model capabilities.
Deep dive: The benchmark relies on multi-page proofs that require synthesizing topology, abstract algebra, and real analysis. Instead of relying purely on next-token prediction, the top-performing models used extensive Monte Carlo tree search during inference, generating thousands of candidate paths before selecting the most logically sound proof structure.
Report check (claims vs what is verified vs still rumor): Creators claimed the dataset was entirely contamination-proof against web scrapers. Independent auditors confirmed that while exact problem statements were novel, structural templates heavily mirrored publicly available Putnam competition archives, raising questions about implicit data leakage.
Open questions: How can the industry design dynamic evaluation frameworks that cannot be beaten simply by scaling inference-time search depth?