NEW YORK — What happened
An independent audit of Verifi, a popular automated benchmarking pipeline for mathematical reasoning, revealed that its verification scripts frequently misgraded valid proofs due to formatting rigidity and parser errors. Several frontier models had previously claimed record-breaking mathematics scores based on these automated runs.
Why it matters
Evaluating advanced AI models requires automated infrastructure because human expert review does not scale. When the evaluation tools themselves are flawed, entire research narratives can be built on false premises, skewing capital allocation and academic focus toward ineffective training methodologies.
Deep dive
The audit analyzed thousands of graded items and discovered that Verifi's abstract syntax tree parser routinely failed to recognize mathematically equivalent expressions if they were written in a non-standard algebraic notation. Consequently, correct model answers were flagged as failures, while hallucinated proofs that matched specific keyword heuristics occasionally passed.
Report check (claims vs what is verified vs still rumor)
The benchmark maintainers acknowledged the parsing flaws and retracted the affected leaderboard rankings. Rumors suggest that multiple commercial labs were aware of the grading anomalies months prior to public disclosure but chose not to report them to preserve marketing advantages.
Open questions
How can the research community build robust verification pipelines that accommodate diverse mathematical syntax without human bottlenecks?