The review gap: what the evidence says about unchecked AI agent work
Excellent
AI agents now open pull requests, update CRM records and close tickets. We read the published evidence from 2023 to 2026 on four questions:
- How much real work do agents finish?
- How often is "passing" work actually right?
- Can an agent check its own work?
- What kind of review closes the gap?
Read together, the studies point the same way. In their samples, passing tests overstated correctness, models judged their own work too kindly, and most agent pull requests in one large study had no recorded review. Review that gathered its own evidence, fixed what "done" means in advance or escalated the hard calls did better.
This is a living review. We re-open every source each quarter and log changes at the bottom.
Agents finish less than it looks
In two benchmark suites of realistic, multi-step work, the leading agents fully completed fewer than half of the tasks.
- Salesforce AI Research's CRMArena-Pro gave nine models 19 CRM tasks across 4,280 queries. The best, Gemini 2.5 Pro, scored 58.3% on single-turn tasks in the B2C environment and 35.1% on multi-turn tasks in B2B; within B2B alone it fell from 54.1% to 35.1%. The median model succeeded on fewer than one multi-turn job in four. We wrote up what that 35% does and does not mean.
- Carnegie Mellon's TheAgentCompany simulated a small software company with 175 tasks. The best model fully completed 30% of them; its partial-completion score was 39%.
Both score task outcomes against reference criteria: CRMArena-Pro compares extracted answers, and TheAgentCompany checks checkpoints in its simulated company. Neither reports whether the agent's own claim of completion matched the outcome. In a deployment, that claim may be the first thing a person sees.
Passing the tests is not the same as being right
A test suite checks what it checks. Agents are good at satisfying exactly that.
- Plausible is not correct. Wang, Pradel and Liu audited patches from three issue-solving tools that SWE-bench Verified counted as solved (ICSE 2026). 29.6% of those plausible patches behaved differently from the developers' fix. On manual inspection, 28.6% of the divergent patches were certainly incorrect. Overall, reported solve rates were inflated by 6.2 absolute percentage points.
- When the tests and the spec disagree, agents pick the tests. ImpossibleBench (Zhong, Raghunathan and Carlini, ICLR 2026) built coding tasks whose tests contradict the specification, so any pass is a cheat. With its full scaffold, GPT-5 cheated on 54% of the Conflicting-SWEbench tasks, and on 76% of the Oneoff-SWEbench tasks, where a single test's expected output was changed. Its methods ranged from editing the tests to overloading operators.
- Telling them not to does little. METR found o3 gaming the scorer in 39 of 128 RE-Bench runs (30.4%). Instructing it not to cheat "had a nearly negligible effect".
Two results point at the fix. ImpossibleBench allowed models to flag an impossible task for human intervention, which ended the run. That cut GPT-5's cheating on the conflicting tasks from 54% to 9% and o3's from 49% to 12%; the effect was much smaller for Claude Opus 4.1. The SWE-bench audit also found that the benchmark's own validation let 7.8% of all patches count as correct while they failed the developers' test suite. More independent evidence, and a way to escalate, both help.
The author cannot be the reviewer
The obvious shortcut is to ask the agent, or another copy of the same model, to check its own work. The evidence says that is a weak check.
- Huang et al. (ICLR 2024): "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction."
- Panickssery, Bowman and Feng (2024) showed that the models they tested recognize their own writing and rate it more highly. In their fine-tuning experiments, greater self-recognition went with stronger self-preference.
- In ImpossibleBench, LLM monitors caught 86–89% of cheating attempts on the simpler LiveCodeBench tasks, but only 42–65% on the more complex SWE-bench tasks.
Independence has to be real: a different model family, a person, or a check that does not depend on the author's account.
Most agent pull requests in one large study had no recorded review
Where review was measured at scale, it was often missing rather than weak.
- Duma et al. (EASE 2026) studied 33,596 agent-authored pull requests in popular repositories from the AIDev dataset. 61.38% had no recorded review. Of the 39,122 review comments that did appear, 71.58% were written by agents. Only 15.9% of agent PRs showed any human participation. In repositories with both kinds of PR, human participation was similar (30.12% of agent PRs, 30.83% of human ones), but among reviewed PRs, human-only review was 8.08% for agent PRs against 25.21% for human PRs. The authors note that no recorded review does not prove no one looked.
- A vendor study by CodeRabbit (December 2025) of 470 open-source PRs found 10.83 issues per AI-co-authored PR against 6.45 for human-only PRs, about 1.7 times as many. It is not peer reviewed, and the authors note they cannot confirm which PRs were human-only.
Put together: in the AIDev sample, most agent-authored PRs had no recorded review, and in CodeRabbit's vendor sample, AI-coauthored PRs had more classified issues per PR. Neither finding says how much agent work reaches the main branch.
What closes the gap
Three lines of evidence say what good review looks like.
Review that gathers its own evidence. Agent-as-a-Judge (Zhuge et al., ICML 2025) gave the judge tools to inspect the workspace rather than read the agent's account. On 55 development tasks with 365 requirements, its verdicts matched the human experts' consensus on 90.44% of requirements, against 60.38% for a plain LLM judge, both judging OpenHands' work without its trajectory. It cost 2.29% as much as the human panel and took 2.36% of the time.
Cheap review for easy calls, escalation for hard ones. Carnegie Mellon's JEV-as-a-Judge tested JEV, TypeSafe AI's hosted decision-only judge, which returns label probabilities instead of text. It came within three points of GPT-6 "wherever a verdict can be read off the text", at 0.36% of its fee, and fell behind "where the verdict must be derived, as in math, code, and logic". Accepting its confident verdicts and escalating the rest was 0.9 points more accurate than GPT-6 alone on 1,610 held-out pairs, at 41% of the fee. The authors note the routing weakens on style-adversarial pairs and on prose with no reference answer.
Checks fixed before the work starts. In our own preregistered lab study, we gave 72 attempts at small coding tasks to five kinds of review. On the preregistered labels, the test suite approved 32 of the 35 wrong attempts. A frontier model reading the diff approved 1. Checks written from the contract before any agent started approved none, at the cost of holding back 5 of 37 good attempts. It is a small pilot and sets no rate.
Six rules for reviewing agent work
These are our recommendations from the evidence above, not a tested recipe.
- Treat the report as a claim. "Done", "tests pass" and a clean exit are statements by the author, not evidence.
- Gather evidence the author did not produce. Re-run the checks yourself, read the real end state, run the full suite.
- Decide what done means before the work starts. Checks written afterwards drift toward what was built.
- Keep the reviewer independent. Another model family, a person, or a deterministic check, never the author grading itself.
- Spend review where it is hard. Let cheap checks clear the easy calls. Escalate code, derivations and confident-sounding answers.
- Let models propose and execution decide. A model may suggest the input that breaks the work. Only running it should count.
Closing
Agents keep getting more capable. The studies above suggest the larger gap is in how their work is accepted: on the author's word, on a passing test, or with no recorded review. The remedy they point to is old: independent evidence, decided in advance. Excellent is built around it. Its planned v1 workflow freezes your checks before the work and re-runs them on the finished change. The published CLI does not yet include that workflow; checks written from the spec are next.
How we read the evidence
We used peer-reviewed papers and preprints where available and opened every source cited. Vendor reports are labeled. Figures are quoted as published, with the model versions each study tested; most of those models are already a generation old. We left out figures we could not trace to a primary page, such as an often-quoted forecast about agent project cancellations.
Sources
- Huang et al., CRMArena-Pro, Salesforce AI Research, arXiv, May 2025.
- Xu et al., TheAgentCompany, Carnegie Mellon University, arXiv v3, September 2025.
- Wang, Pradel and Liu, Are "Solved Issues" in SWE-bench Really Solved Correctly?, ICSE 2026, arXiv v2.
- Zhong, Raghunathan and Carlini, ImpossibleBench, ICLR 2026.
- METR, Recent frontier models are reward hacking, June 5, 2025.
- Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet, ICLR 2024.
- Panickssery, Bowman and Feng, LLM Evaluators Recognize and Favor Their Own Generations, 2024.
- Duma et al., These Aren't the Reviews You're Looking For, EASE 2026.
- CodeRabbit, State of AI vs human code generation, December 17, 2025. Vendor report.
- Zhuge et al., Agent-as-a-Judge: Evaluate Agents with Agents, ICML 2025.
- Li, Miao, Krishnan and Padman, JEV-as-a-Judge: Accept When Confident, Escalate When Unsure, arXiv preprint v5, October 9, 2026.
- Excellent, Same work, five reviewers, November 2026.
Changelog
- November 16, 2026. First version. Next review in February 2027.
Citation
Please cite this work as:
Excellent, “The review gap: what the evidence says about unchecked AI agent work”, Excellent Research, November 16, 2026. www.excellent.so/research/the-review-gap
Or use the BibTeX citation: