A new coding-agent benchmark built from real user revision requests finds that the strongest of ten models meets most individual requirements and still fully finishes only about one in four jobs. The gap between partial credit and complete delivery, plus large swings when the same model runs in a different harness, is the part buyers should care about.
Researchers from Alibaba Token Hub, Fudan University, Zhejiang University, the Chinese University of Hong Kong, Shanghai Jiao Tong University, and Nanyang Technological University released Co-Design Bench as an arXiv preprint (arXiv:2610.35138). The suite has 273 tasks drawn from 37,463 deduplicated queries across 26,314 real coding sessions, graded against 2,331 hidden expert-authored acceptance criteria. Under a shared agentic harness, OpenAI's GPT-6 Astra leads the field with a mean strict completion of 69.9 percent of criteria per task—and a full-pass rate of 26.4 percent. That means 73.6 percent of its tasks still leave at least one criterion unfinished.
Those numbers sit far below the near-ceiling scores the same generation of models can post on older web-editing tests. On 150 WebCompass-edit tasks, the paper reports that Claude Opus 4.8 reaches 99.4 percent mean graded feature completion under agentic execution. Co-Design Bench is designed to explain why that kind of saturation does not translate into finished user work.
What the benchmark actually measures
Each task starts from a runnable pre-edit web application and a normalized user request. Optional attachments can include images, audio, video, or documents. The agent never sees the grading rubric. After it stops editing, a judge-then-refute protocol scores the final app against the hidden criteria using source diffs and browser evidence, then a separate LLM verifier challenges every Pass and Partial verdict for stubs, hardcoding, unreachable features, and requirement mismatches.
The suite has two tracks:
- Multimodal Editing (ME): 100 localized edits, 553 criteria.
- Repository Editing (RE): 173 coordinated multi-file frontend revisions, 1,778 criteria. Of those 173 tasks, 166 require more than one edit type.
Starting artifacts are mostly single-page HTML on ME and single-page React apps on RE. Scoring reports three metrics: Strict Completion (only full Passes count), Graded Completion (Partial counts as half), and Full-Pass Rate (every criterion must Pass). The authors stress the inequality Full-Pass ≤ Strict ≤ Graded: high average requirement coverage can coexist with low complete-task success when unfinished requirements are spread across many tasks.
Why 69.9% still fails three jobs in four
GPT-6 Astra's headline average is flattering. On ME it reaches 0.754 strict completion and fully passes 41.0 percent of tasks. On RE it drops to 0.667 strict and only 17.9 percent full-pass. Averaged over all 273 tasks, that is 0.699 strict and 26.4 percent fully delivered.
For a team buying a coding agent, the practical reading is blunt. On this distribution of real revision requests, even the leader leaves unfinished work on roughly three out of four tasks. Average requirement coverage is a useful research metric; it is a poor proxy for "done." A ticket that is 70 percent correct still needs a human to find and close the last unmet requirement—exactly the cleanup cost vendors often leave out of demos.
The paper's own ranking table makes the same point across the field. Anthropic's Claude Opus 5 is second at 18.3 percent full-pass. Qwen3.8-Max reaches 16.5 percent. xAI's Grok 4.6 is last at 6.6 percent. No endpoint approaches the 99 percent-class scores that older, narrower tests can still produce.
Why older 99%+ scores mislead
The authors motivate Co-Design Bench with a controlled comparison on WebCompass-edit. Holding the feature rubric, judge, and render pipeline fixed, switching Qwen3.7-Plus from single-shot to agentic execution lifts mean graded feature completion from 73.6 percent to 95.1 percent. Opus 4.8 hits 99.4 percent under agentic execution on the same 150 tasks.
That result has two implications. First, execution regime—iterative tools versus one-shot patches—can move scores by tens of points on the same tasks. Second, once agentic loops are allowed on predefined edit suites, scores can saturate even while real-user compound revisions remain hard. Co-Design Bench deliberately selects compound, session-derived requests and grades complete delivery, not just feature averages. The 26.4 percent full-pass figure is the corrective, not a contradiction of the older leaderboards.
The harness can swing results as much as the model
The main leaderboard holds the Claude Code harness fixed across all ten endpoints. In a separate ablation, the authors reran Kimi K3 and Qwen3.8-27B across three harnesses—Codex 0.142.0, OpenCode 1.18.13, and Qwen Code 0.21.11—with shared prompts, criteria, and judges.
Qwen3.8-27B's repository-edit score moved from 0.560 under Qwen Code to 0.749 under OpenCode. The gap between Kimi K3 and Qwen3.8-27B shifted by 10.0 percentage points on ME and 16.2 points on RE depending on the harness. On ME, harness rankings inverted completely between the two solvers. The authors conclude that model-only leaderboards tell part of the story and that model and harness should be reported together.
For procurement, that finding is as important as the 26.4 percent headline. Buying "the best model" without locking the agent scaffold, tools, prompts, and verification loop is buying an incomplete system. A fair vendor bake-off needs the same tickets run through the candidate model–harness pairs a company would actually ship.
Where agents struggle most
Broader repository edits are harder on criterion coverage. Across all ten endpoints, RE tasks spanning three or four edit types score 0.591 mean strict completion, versus 0.654 for one or two types—a 6.3-point gap. Every endpoint scored lower on the broader group.
By primary capability, Functional Implementation is the largest bucket (126 of 273 tasks). It has the highest pooled strict completion (0.619) and the lowest full-pass rate (10.3 percent). Grounded Understanding reaches 20.2 percent full-pass; UI Transformation 13.6 percent. The pattern repeats: agents often satisfy many requirements and still miss complete delivery.
More tokens do not fix it. On ME, GPT-6 Astra reaches 0.754 strict with a median 32 turns and about 1.24 million tokens per case, while GLM-5.2 uses 69 turns and about 7.29 million tokens for 0.572.
Limits that matter for interpretation
Several constraints belong in any serious reading of the result.
- Hard-case selection. Candidates were ranked partly on historical failure signals—reformulations and explicit rollbacks—so the set leans toward requests that already gave users or prior agents trouble. Scores are not a random sample of everyday tickets.
- Fixed, non-interactive requests. Agents cannot ask clarifying questions or receive mid-task corrections. The metric is autonomous completion under a frozen brief, not collaborative coding with a human in the loop.
- AI judges. Primary grading and refutation are LLM-based. Two frontend experts checked 200 task–model outputs; rescoring with alternate judges moved graded scores by at most about 0.05 in the reported sensitivity check. Judgment error is reduced, not eliminated.
- Author affiliation. Alibaba Token Hub is a lead affiliation, and Qwen models appear in the evaluated set. The main comparison uses a shared Claude Code harness, which limits one form of home-field advantage, but readers should still treat Qwen rankings as author-involved results until independent replication.
- Scope. Tasks are frontend web revisions (HTML, React, Vue), not full-stack backend, mobile, or infrastructure work. The preprint is not peer-reviewed.
- Contamination risk. Tasks come from production sessions and recovered applications. The paper does not publish a contamination audit against the evaluated models' training data; that remains an open concern for any session-derived benchmark.
What teams should do with the result
Co-Design Bench does not say coding agents are useless. It says partial-credit averages and saturated older benches overstate readiness for unattended delivery. A realistic operating model for most teams is still agent-plus-reviewer: let the model clear the bulk of criteria, then budget human time for the unfinished tail that the full-pass metric exposes.
When evaluating vendors, treat the product as a model–harness pair. Pilot both on your own ticket distribution, score complete acceptance rather than vibe-check demos, and watch repository-scale, multi-type changes separately from localized UI tweaks. On Co-Design Bench's hardest track, even the leader fully finishes fewer than one in five repository jobs.
The research question the field still has to answer is not whether agents can hit another 99 percent on a predefined edit suite. It is whether they can close the last unmet requirement on real user revisions without a human cleaning up. At 26.4 percent full-pass for the current leader, that gap remains the product.
Primary source: Wei, Ding, Fang et al., "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models," arXiv:2610.35138; project site codesignbench.github.io. Coverage also appeared in The Decoder.