NEW YORK — What happened A growing rift between academic evaluators and commercial AI developers has erupted into public view, centered on how models are primed before taking standardized tests.
Why it matters Public leaderboards drive billions of dollars in venture capital and enterprise contracts. When these scores are artificially inflated through selective evaluation framing, market trust erodes.
Deep dive Investigators found that minor tweaks to system prompts—such as instructing a model to take a deep breath or think step-by-step—can swing benchmark scores by up to twenty percentage points. Furthermore, some labs run extensive validation searches to find the exact phrasing that triggers the model's highest-performing reasoning pathways during evaluations.
Report check (claims vs what is verified vs still rumor) Whistleblower accounts claim that certain labs discarded low-scoring evaluation runs until achieving a favorable statistical outlier to publish. While systemic data manipulation remains rumored, selective prompt tuning for public tests is fully verified by public model card disclosures.
Open questions Will independent evaluation bodies like LMSYS and Epoch AI succeed in standardizing prompt conditions across all tested models?