NEW YORK — What happened A growing rift has emerged between self-published benchmark scores by AI labs and the independent replication results published by academic evaluation consortia.
Why it matters When public safety and enterprise deployment decisions rely on standardized scores, discrepancies between marketing claims and reproducible reality create severe misallocations of capital and trust.
Deep dive The controversy stems from subtle variations in evaluation harnesses, prompt formatting, and hyperparameter tuning during testing. Small changes in how a multiple-choice question is phrased can swing a model's score by double-digit percentages, allowing labs to selectively highlight favorable configurations.
Report check (claims vs what is verified vs still rumor) Labs maintain their evaluation setups reflect standard user practices. Independent researchers have verified that prompt sensitivity accounts for a significant portion of reported score gaps, though allegations of intentional benchmark manipulation remain unverified.
Open questions Will the industry adopt standardized, immutable evaluation harnesses enforced by neutral third parties?