Frontier models now score near the ceiling on many popular tests, while harder benchmarks built from real work show them finishing far less. This tracker puts headline scores next to the tougher results, notes which benchmarks are saturated or contaminated, and links each number to its source.
As of November 9, 2026, 7:54 AM (New York time, EST). Each row is dated and sourced. Preprints, vendor-run evaluations and single-source results are marked.
The short version
The gap between "scores well" and "gets the job done" is the most useful number in AI evaluation right now. On Co-Design Bench, built from real user revision requests, the leading model meets about 70% of requirements but fully finishes only 26.4% of tasks (Co-Design Bench paper). On an older web-editing suite, models of the same generation score 99.4%. OpenAI stopped reporting SWE-bench Verified in February after finding flawed tests and training-data contamination (OpenAI). IBM's ScarfBench, run independently by Kaggle, found leading setups compile nearly every enterprise Java migration while the best still pass all behavioral tests on only about 72% (IBM Research). And separate results show the agent harness, meaning the tooling and setup around a model, can move scores by tens of points: ARC-AGI-3 jumped from 62.7% to 99.9% when OpenAI's own harness was used (ARC Prize).
Headline scores vs. harder tests
| Benchmark | What it tests | Headline score | Harder or real-world result | Saturation / gap | Date | Sources |
|---|---|---|---|---|---|---|
| Co-Design Bench | 273 real user revision requests on web apps, graded on 2,331 hidden criteria | GPT-6 Astra meets 69.9% of criteria per task on average | Fully completes only 26.4% of tasks (17.9% on multi-file repository edits) | Average credit hides unfinished work on about three tasks in four | Nov 2026 | Co-Design Bench paper; project site (preprint; Alibaba-affiliated authors; AI judges; frontend only) |
| WebCompass-edit (older suite) | 150 predefined web-editing tasks | Claude Opus 4.8 reaches 99.4% graded feature completion with agentic execution | Same generation of models falls to 26.4% full completion on Co-Design Bench | Saturated | Nov 2026 | Co-Design Bench paper (figure reported in the Co-Design Bench paper) |
| Co-Design Bench harness ablation | Same model and tasks, different agent tooling | Qwen3.8-27B scores 0.560 on repository edits under Qwen Code | 0.749 under OpenCode; model rankings flipped on one track | Harness can move scores as much as the model | Nov 2026 | Co-Design Bench paper (preprint) |
| ARC-AGI-3 | Novel turn-based puzzle environments solved without instructions | GPT-6 Astra: 99.9% with OpenAI's provider adapter harness; fewer actions than the median human on 96% of levels | 62.7% with ARC Prize's standard, provider-neutral harness | Near-saturated under one harness; ARC Prize says the result is not AGI | Sep 3, 2026 | ARC Prize (benchmark owner's own evaluation) |
| SWE-bench Verified | 500 curated GitHub issues in 12 Python repos | State of the art rose from 74.9% to 80.9% in six months | OpenAI found 59.4% of 138 hard audited problems had flawed tests, and every frontier model it tested could reproduce some gold patches | Contaminated; OpenAI stopped reporting it | Feb 23, 2026 | OpenAI (audit by OpenAI, which also builds models) |
| SWE-Bench Pro (Scale) | 1,865 tasks from copyleft and private codebases; public set of 731 | At launch, top models scored over 70% on Verified but about 23% here (GPT-5 23.3%, Claude Opus 4.1 23.1%) | Top public-set entry listed now: Muse Spark 1.1 at 61.50% (mini-swe-agent harness). On private codebases, GPT-5 fell to 14.9% | Not saturated; private code is harder | Leaderboard checked Nov 9, 2026 | Scale SWE-Bench Pro leaderboard (leaderboard maintained by benchmark owner) |
| METR task time horizons | Software, ML and cyber tasks timed against human experts | Claude Mythos Preview (early) has a 50% time horizon above 16 hours | METR says measurements above 16 hours are unreliable with its task suite, and stopped updating the page on Sep 8, 2026 | Suite saturating at the top end; tasks are clean and well-specified | May 8 and Sep 8, 2026 | METR time horizons |
| AECI (Anthropic capability index), via METR | Aggregate of internal and public benchmarks | Anthropic improving about 1.9x faster than before Oct 2025 (95% CI 1.5x–2.8x) | An internal model scored above 50% on every AECI benchmark it was tested on; claim limited to well-specified tasks; most supporting evidence redacted | Index nearing saturation | Nov 5, 2026 | METR Anthropic assessment (evidence heavily redacted by Anthropic) |
| METR developer productivity trial | Experienced open-source developers on their own tasks, randomized with or without AI | Early-2025 study: tasks took 19% longer with AI (CI +2% to +39%) | Late-2025 data: an estimated 18% less time for returning developers (CI -38% to +9%) and 4% less for new ones; METR calls the new data unreliable because developers refused to work without AI | Real-world productivity still poorly measured | Feb 24, 2026 | METR (confidence intervals include no effect) |
| ScarfBench (IBM, run independently by Kaggle) | 204 enterprise Java framework migrations (Jakarta EE, Quarkus, Spring), checked by behavioral tests | Most leading model-plus-harness setups compile 97% or more of tasks | Best smoke-test score is 75.2% (Gemini 4 Argon with Antigravity); best all-tests-pass rate is 72.1% (Claude Opus 5.5 with Claude Code). The leader changes with the metric, and 72% of failures were apps that built but behaved wrongly | Not saturated; compiling is not the same as working | Oct 27, 2026 | IBM Research; Kaggle leaderboard (results are model-plus-harness pairs; IBM built the benchmark, Kaggle reproduced the runs) |
| AutomationBench (Zapier) | Real business workflows across many connected apps | Claude Opus 5.5 launch materials say it leads on agentic coding, computer use and knowledge work | On Zapier's business-workflow test, the best scores are 41.4% (GPT-6 Astra) and 40.0% (Opus 5.5); safeguard interventions counted as failures | Far from saturated; Anthropic itself says benchmark margins have become a less reliable guide to real-world differences | Sep 22, 2026 | Anthropic (Zapier-run, as reported in Anthropic's launch post; mixes early-access and public-leaderboard runs) |
| ExploitGym and RemoteExploitBench (OpenAI) | Exploiting specified vulnerabilities; known flaws in GitLab, PostgreSQL and Jenkins in a controlled lab | GPT-6 Cyber scores 48.5% on ExploitGym (GPT-5.6 Cyber: 32.1%) and completes 95% of RemoteExploitBench tasks | No independent replication: OpenAI built RemoteExploitBench and ran every evaluation; the model is available only to approved Daybreak Red customers | Vendor-built tests; cannot yet be checked outside OpenAI | Nov 5, 2026 | OpenAI (vendor-run evaluation on vendor-built benchmarks) |
Three patterns in the data
Partial credit flatters. Averages of requirements met, or of features graded, can look strong while complete delivery stays rare. A ticket that is 70% done still needs a person to finish it. Our Co-Design Bench analysis covers this in detail.
Saturation and contamination arrive together. Once a public benchmark sits on the open web, models can learn its answers. OpenAI showed models reproducing SWE-bench Verified fixes from memory; Scale built SWE-Bench Pro from copyleft and private code partly to resist that. Even METR's own yardsticks are filling up: its time-horizon suite cannot reliably measure beyond 16 hours, and an internal Anthropic model scored above 50% on every benchmark in the AECI index (METR Anthropic assessment; see our METR report analysis).
Vendors are hedging their own numbers. Anthropic's Opus 5.5 launch post says benchmark margins "have become a less reliable guide to real-world differences," and its comparison table mixes its own runs with competitor figures "as reported by OpenAI" (Anthropic). OpenAI's GPT-6 Cyber results come from benchmarks OpenAI built and ran itself (OpenAI). Neither is wrong to publish, but neither is independent evidence.
The system is the product. The same model can score very differently depending on the harness. Buyers comparing vendors should test the exact model-plus-tooling pair they plan to use, on their own tasks, and score full completion rather than averages.
What changed recently
- Nov 9 update: Added three rows: IBM and Kaggle's independently reproduced ScarfBench results (IBM Research), Zapier's AutomationBench scores from the Opus 5.5 launch (Anthropic), and OpenAI's vendor-run GPT-6 Cyber results (OpenAI). Rechecked the SWE-Bench Pro public leaderboard; Muse Spark 1.1 remains the top entry at 61.50% (Scale SWE-Bench Pro leaderboard).
- Nov 5: OpenAI released GPT-6 Cyber to approved Daybreak Red customers, reporting 48.5% on ExploitGym from its own evaluations (OpenAI).
- Oct 27: IBM and Kaggle launched a maintained ScarfBench leaderboard that ranks model-plus-harness pairs, with the leader depending on whether partial credit counts (IBM Research).
- Nov 2026: Co-Design Bench reported a 26.4% full-pass rate for the best of ten models, plus a harness ablation showing large swings (Co-Design Bench paper).
- Nov 5: METR's Anthropic assessment reported about 1.9x faster capability growth on well-specified tasks, with saturation and redaction caveats (METR Anthropic assessment).
- Sep 8: METR said its public time-horizon page is no longer actively updated (METR time horizons).
- Sep 3: ARC Prize said it will report both standard and provider-adapter harness results on ARC-AGI-3 after GPT-6 Astra's 99.9% (ARC Prize).
What to watch next
- Independent replication of Co-Design Bench, especially by groups not affiliated with any model developer.
- Harness-labeled leaderboards. Whether other benchmark owners follow ARC Prize and ScarfBench in reporting results as model-plus-harness pairs.
- Outside checks on cyber models. Whether any third party can test GPT-6 Cyber's exploit results once access widens beyond the first Daybreak Red customers.
- New ScarfBench entries. Kaggle says it will add models as they ship, which makes the benchmark a running check on vendor coding claims.
- A successor to METR's time-horizon suite with tasks long enough to measure today's top models.
- Private, expert-written evaluations. OpenAI has said it is building uncontaminated coding evaluations; how they are published will matter.
- Real-world productivity data. METR has said it is redesigning its developer study after selection effects broke the last one.
Methodology
We pair a widely cited headline score with a harder or more realistic measurement of a similar skill, using the benchmark owner's paper, leaderboard or report as the primary source. We record the harness or setup where the source states it, because it can change results materially. Preprints, evaluations run by model developers, and leaderboards maintained by benchmark owners are marked as such. We call a benchmark saturated when top scores sit near its ceiling or its owner says it can no longer separate top models, and contaminated when there is published evidence of training-data exposure. Rows are updated when a new top score, a new benchmark version, or a credible audit appears, and every update is dated.