NEW YORK — What happened
Recent evaluations of late-2026 frontier models show near-human expert performance on classic mathematics benchmarks. Yet, independent auditing groups have released damning reports revealing widespread test set contamination and brittle generalization.
Why it matters
Benchmarks are the compass of the AI industry. If our measuring sticks are broken or memorized by the models, we cannot accurately assess whether an AI is truly reasoning or merely retrieving sophisticated memorized solutions.
Deep dive
Auditors found that slight semantic shifts in word problems—ones that do not alter the underlying mathematical structure—cause frontier models to fail up to 30% of the time. Furthermore, training pipelines have inadvertently ingested synthetic variants of test problems, rendering historical leaderboards largely obsolete.
Report check (claims vs what is verified vs still rumor)
While lab reports claim generational leaps in mathematical discovery, verified audits show that models struggle heavily with out-of-distribution problem variations. Rumors suggest upcoming evaluations will require live, dynamically generated proofs.
Open questions
Can the AI community establish a contamination-proof evaluation framework before general intelligence claims outpace empirical reality?