Review, testing and what to measure when agents write the code

This page is the full version of two parts of Deploying Agents at Scale: What Enterprise Adoption Actually Looks Like, a talk by Michael Rishi Forrester, AI Startup Advisor, Neo4j Startup Program, and author of Agentic DevOps with Claude Code (Packt, 2026), at Agentic Engineering Days Zurich, November 2026. The talk promised two things it only had time to start: how the teams that succeeded "restructured review and testing to absorb machine-speed output", and what they "actually measured" in place of PRs merged. The talk had time for a short version of the first. This page carries both, with every source.

Every figure carries a label, so you can judge it yourself:

Faros AI's figures come up often below. Faros is a vendor publishing its customers' data, and it sells the remedy for the problems its data shows. Each Faros figure on this page says so where it appears.

The Faros pages, the Harvard paper, CodeRabbit's report and Google's paper on ML-suggested edits were read live on 2026-11-14. Every other source was read live on 2026-11-13. The table at the end gives each one with its date.

What is measured and what is argued

The sources below measure some things well and leave others to argument. This table says which is which before the detail.

Claim Measured or argued What rests on it
When coding agents arrived, review became the bottleneck Measured Harvard's study of 655 firms: time from submission to merge up 56.1%. Controlled study, quasi-experimental
AI-written PRs bring review more to catch Measured, weakly CodeRabbit's 470 open-source PRs, authorship inferred. Vendor telemetry from a vendor that sells AI review
Teams that use an agentic first pass heavily get faster first reviews and fewer change failures Measured as an association, not a cause Faros's customer data, which Faros calls correlational. Vendor telemetry; Faros sells the remedy
AI in review makes review faster Measured, and not supported Randomized trials at Google and Meta found no speedup, and one design slowed review
Write down which changes need a human, an agent, or both Argued Faros's recommendation. No source measures its effect
Filter generated tests before a human sees them Measured as acceptance, not against an alternative Meta's company field reports: 73% of what reached engineers was accepted
Small batches keep AI's speed from turning into instability Argued DORA's recommendation. DORA measured effects on product performance and friction, not on instability
A static-analysis gate belongs on agent output Argued from a measured finding Carnegie Mellon measured warnings and complexity rising after Cursor adoption; no one tested the gate
PRs merged misleads once agents write code Measured, then argued Cortex, Uplevel, LinearB and Faros each show a count rising while quality or stability fell. Vendor telemetry
Stability measures catch what PR counts miss Measured DORA 2024: more AI adoption went with less delivery stability while review speed and code quality rose. Self-reported, survey
Teams that switched to a better measurement set improved Not measured No primary read for this page reports a team that switched and published the before and after
Review capacity has to be planned with the agents Argued from a model Harvard's fitted model moves labor from coding to review. Forecast

How review and testing were restructured

Review became the bottleneck

Controlled study, quasi-experimental. Fiona Chen and James Stratton at Harvard studied engineering-platform data from Jellyfish covering 720,542 workers at 655 firms, January 2021 to August 2026. They compare firms by when each adopted AI tools, in a staggered difference-in-differences design, with no random assignment.

Vendor telemetry, descriptive. The same paper counts AI in review from Jellyfish's platform data: as of July 2026, 79.1% of firms had used AI tools in code review, yet only 23.3% of review comments were written by AI and only 10.8% of pull requests got at least one AI comment.

Forecast, model estimate. The paper's fitted model says AI agents raise coding productivity 73.7% and lower review productivity 7.0%, and that firms respond by moving labor from coding (down 22.7%) to review (up 43.4%). Its long-run simulation, at the mean of published demand estimates, gives output up 16.1%, coding employment down 33.2% and review labor up 24.9%. These are model outputs, not observed changes.

What arrives at review

Vendor telemetry. CodeRabbit, which sells an AI pull-request reviewer, analyzed 470 open-source GitHub pull requests, 320 AI co-authored and 150 human-only. AI-authored changes had 10.83 issues per PR against 6.45 for human-only PRs, about 1.7 times as many, with "~1.4–1.7× more critical and major findings" (quote). Logic and correctness issues were 75% more common in AI PRs. CodeRabbit inferred authorship from signals in each PR and says it cannot guarantee that every PR it labeled human-authored was written only by humans. It is a small sample, measured by a vendor whose product is the remedy it describes.

An agentic first pass

Vendor telemetry, correlational. Faros, writing about its customers' data, so the vendor that sells the remedy: "high agentic-review adoption was associated with a 10.7% reduction in time to first review and a 24.4% reduction in change failure rate" (quote, blog, 13 October 2026). Faros's takeaways page adds: "The findings are correlational, and unreviewed merges continue to rise even where adoption is high" (quote). It calls agentic review "the first intervention we have measured at a meaningful scale" (quote). On the same data, Faros saw median time to first PR review rise 114.5% as AI adoption deepened.

Vendor telemetry. Faros, the vendor that sells the remedy, says agents open "13–14% of PRs at the leading edge" (quote), and that agentic review runs widely: "many companies running AI review on 50–80% of PRs" (quote). Outline v4 of this talk records Faros's own report as giving 26% of PRs under agentic review. The report is behind a sign-up form, so the 26% was not read at the primary and is unverified; see "The talk outline's Faros figures, checked" below. Neither share is used as a finding here.

Self-reported, company field reports. Three companies measured AI review on their own engineers, with no random assignment:

What the randomized trials found

Controlled study, randomized. Google randomly assigned about half of all its developers to AutoCommenter and found no statistically significant change in the total duration of code reviews or in comment-response iterations.

Controlled study, randomized. Meta's safety trial for MetaMateCR, which suggests patches for review comments: when reviewers saw AI-suggested patches, reviews took over 5% longer. Showing the patches only to authors removed the slowdown. In production, 19.7% of actionable comments had the AI patch applied (self-reported, company field report).

In these two trials, AI in review changed what review catches and who does the work. It did not make review faster.

Writing the gate down

Recommendation, not a figure. Faros, the vendor that sells the remedy: "Organizations should define which changes require human review, agentic review, or both, and enforce those requirements so delivery pressure does not determine which checks are skipped" (quote). Its own data show why: PRs merged with no review rose 31.3% in its April 2026 report and 76.3% in Q3 (vendor telemetry). No source measures teams that adopted such a policy.

Generated tests, filtered before a human sees them

Self-reported, company field reports. Meta published two systems that generate tests and discard most of them before an engineer looks:

Neither paper compares filtered against unfiltered tests on the same engineers, so the filter's effect on acceptance is argued, not measured.

Batch size and a static-analysis gate

Self-reported, survey. DORA 2025: "AI adoption does continue to have a negative relationship with software delivery stability" (quote). DORA's AI Capabilities Model (2025) finds that working in small batches "amplifies the positive influence of AI on product performance and reduces friction for development teams" (quote), and that frequent commits and frequent rollbacks strengthen AI's effect.

Recommendation, not a figure. DORA's later guide says small batches counteract the risk of large AI changes, "ensuring that speed translates to product performance rather than instability" (quote). DORA measured the product-performance and friction effects, not an effect on instability.

Controlled study, quasi-experimental. Carnegie Mellon compared 806 open-source repositories that adopted Cursor with 1,380 matched ones (MSR 2026): lines added rose 281.3% in the first month, then static analysis warnings rose 30.3% and code complexity 41.6%, which the authors tie to later slowdown. The case for a static-analysis gate on agent output follows from this; the gate itself was not tested.

What to measure in place of PRs merged

No team published the switch

No primary read for this page names a team that dropped PRs merged for another set of measures and published the before and after. What the sources give is a set of measures, who measured each across many teams, and what each showed. The nearest thing to "the teams that succeeded" is Faros's cohort of heavy agentic-review users, which Faros calls correlational, and DORA's elite cluster.

Why PRs merged misleads

A count of merged PRs rises when PRs get smaller, when agents open more of them, or when review is skipped. Four vendors' data show a PR count rising or holding steady while what it hides got worse. All four are vendor telemetry; Faros sells the remedy, and so do the analytics vendors beside it.

What rose What it hid Who measured it Label
PRs per author up 20% in a year Change failure rate up 30% and incidents per PR up 23.5% over the same period, Q3 2024 to Q4 2025 Cortex, 2026 Benchmark Report Vendor telemetry
PR throughput unchanged with Copilot access 41% more bugs in the group with access, across nearly 800 developers Uplevel, 2024 Vendor telemetry, comparison group with no random assignment stated
Average PR size up 71.8% A PR is no longer one unit of work; AI-assisted PRs ran 2.6x larger than unassisted ones in LinearB's data Faros Q3 2026 (a vendor that sells the remedy); LinearB 2026 Vendor telemetry
AI PRs opened Only 32.7% of AI PRs merged within 30 days of opening, against 84.4% of manual PRs, across 8,109,244 PRs LinearB 2026 Vendor telemetry
Merged PRs PRs merged without review up 31.3% in April 2026 and 76.3% by Q3 Faros (a vendor that sells the remedy) Vendor telemetry

The measurement set, with who used each measure and what it showed

Measure What it counts Who used it What it showed Label
Change lead time and deployment frequency (DORA throughput) Commit to production; deployments per period DORA 2024 and 2025; Faros Q3 2026 DORA 2024: each 25% increase in AI adoption went with an estimated 1.5% drop in delivery throughput, and DORA 2025 found the relationship had turned positive. Faros, which sells the remedy: deployment frequency up 13.8%, among only the roughly 10% of teams with deployment data Self-reported, survey (DORA); vendor telemetry (Faros)
Change fail rate (DORA stability) Share of deployments that need immediate intervention DORA 2024; Cortex 2026; Faros Q3 2026 DORA 2024: an estimated 7.2% drop in delivery stability per 25% increase in AI adoption; DORA 2025 found the negative relationship still there. Cortex: change failure rate up 30%. Faros, which sells the remedy: heavy agentic-review teams had a 24.4% lower change failure rate, correlational Self-reported, survey (DORA); vendor telemetry (Cortex, Faros)
Deployment rework rate (DORA stability, added 2024) Unplanned deployments made to fix a user-facing bug DORA 2024; Faros's restarts, a related developer-level signal and not the same measure DORA found rework rate and change fail rate together "create a reliable factor of software delivery stability" (quote). Faros, which sells the remedy: restarts up 66.7% Self-reported, survey (DORA); vendor telemetry (Faros)
Failed deployment recovery time Time to recover from a failed deployment DORA 2024; Faros Q3 2026; Cortex 2026 DORA 2024's elite cluster recovers in less than one hour. Faros, which sells the remedy: mean time to remediate up 25.8%. Cortex reports longer resolution times without a figure Self-reported, survey (DORA); vendor telemetry (Faros, Cortex)
Incidents per PR, beside monthly incidents Production incidents per merged change, and their total Faros 2026; Cortex 2026 Faros, which sells the remedy: incidents per PR up 242.7% in April and 14.5% by Q3, while monthly incidents rose 125.4%, each against the low-adoption baseline. The Q3 figure comes from a different sample of organizations than April's. In the later sample the per-change spike faded and is still above the baseline, and the monthly total more than doubled. Cortex: incidents per PR up 23.5% Vendor telemetry
Bugs per PR Bugs traced to changes Faros April 2026; Uplevel 2024 Faros, which sells the remedy: bugs per PR up 28%. Uplevel: 41% more bugs with Copilot access and no change in PR throughput Vendor telemetry
Churn and code reuse Lines rewritten soon after they land; moved and copied lines GitClear 2025, 211 million changed lines Refactoring ("moved") lines fell from 25% of changed lines in 2021 to less than 10% in 2024; copy/pasted lines rose from 8.3% to 12.3%. GitClear's short-term churn figure is behind its form and unverified Vendor telemetry
Merged within 30 days of opening (LinearB's acceptance rate) PRs merged within 30 days of being opened LinearB 2026, 4,813 teams Manual PRs 84.4%, AI PRs 32.7%. It counts what was opened and never landed, which a merged count cannot Vendor telemetry
Review coverage Share of PRs merged with no review Faros 2026 Faros, which sells the remedy: up 31.3% in April, 76.3% by Q3. It belongs beside any throughput figure, because skipping review raises throughput Vendor telemetry
Tests passed and blind-review approval Unit tests passed; reviewers' approval without knowing the author's tools GitHub's trial, 202 developers Code written with Copilot had a 53.2% greater likelihood of passing all 10 unit tests, and reviewers were 5% more likely to approve it. One task, run by the vendor that sells Copilot Controlled study, randomized, vendor-run

Recommendation, not a figure. DORA: "Teams should make each change as small as possible to make the delivery process fast and stable" (quote). Faros, which sells the remedy, advises tracking "spend per verified outcome alongside restarts, QA time, and remediation effort" (quote).

DORA's five delivery metrics

Self-reported, survey. DORA's guide, last updated 5 January 2026, names five software delivery performance metrics, "shifting from the original four keys to the current five-metric model" (quote):

What DORA found when AI arrived

Self-reported, survey. DORA 2024 surveyed nearly 3,000 working professionals. The roughly 39,000 often quoted is every DORA respondent across the decade, not the 2024 sample. Each 25% increase in AI adoption went with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. The same increase went with documentation quality up 7.5%, code quality up 3.4%, code review speed up 3.1%, approval speed up 1.3% and code complexity down 1.8%. These are estimates from survey answers, with 89% uncertainty intervals.

Self-reported, survey. DORA 2025, nearly 5,000 respondents: "a positive relationship between AI adoption on both software delivery throughput and product performance", and "AI adoption does continue to have a negative relationship with software delivery stability" (quotes).

In DORA 2024 the process measures improved while delivery stability fell. A team that measured only at the pull request would not have seen the fall.

The talk outline's Faros figures, checked

The talk's outline (v4, section 5) gives Faros's Q3 2026 agentic-review findings from the Speed Trap report. Faros is a vendor publishing its customers' data, and it sells the remedy. Each claim was checked against Faros's own public pages, read live on 2026-11-14 (and on 2026-11-13 before that). The report itself sits behind a sign-up form on Faros's research page, which shows no figures and links no PDF, so it was not read.

Outline v4 claim At Faros's own primary Status
High agentic-review adoption: 10.7% less time to first review, 24.4% lower change failure rate Faros blog, 13 October 2026, word for word; the takeaways page gives the direction without figures Verified
The findings are correlational Takeaways page, takeaway 8 Verified
"The first intervention in our data that shows measurable benefits at scale" Not on any public page. The takeaways page says "the first intervention we have measured at a meaningful scale that is associated with improvements across multiple downstream metrics" Corrected to the page's wording
Agentic review covers 26% of PRs The Speed Trap report, pages 1 and 5 (printed), read 2026-11-17 Verified
Heavy agentic-review teams see gate erosion at roughly half the general rate The Speed Trap report, printed page 21, read 2026-11-17 Verified
Change failure rate carries a standard error as large as its estimate The Speed Trap report, printed page 20, footnote 3: the caveat is on the change failure rate level; the agentic-review association "should still be read as correlational" Corrected: it is not a caveat on the 24.4%
Agents open 2% of PRs overall, under 5% for most companies Not on the takeaways page Unverified at the primary
Agents open 13 to 14% of PRs at the leading edge Takeaways page, takeaway 1 Verified
"The greatest leverage lies at the authoring stage" Takeaways page, takeaway 8 Verified
76.3% of merges skip review Takeaways page, takeaway 5, and the blog: PRs merged without review are "up 76.3%", a rise, not a share of merges Corrected: a 76.3% rise, not 76.3% of merges
Do not say "50 to 80% of PRs" On the takeaways page, takeaway 1: "many companies running AI review on 50–80% of PRs" On Faros's page, and in conflict with the report's 26%; not used as a finding

Faros, which sells the remedy, also summarizes the Harvard paper in its 13 October blog: "PR submission-to-merge time increased 49%", formal change requests "from approximately 13% to 25%", and review participation up 14%. None of the three is in the paper's 9 November 2026 version, which gives 56.1% and a blocked share of 2.77% at baseline. Faros may have read an earlier version; these figures are unverified, and this page cites the paper directly.

What this evidence does not show

Three questions for your own review and measures

  1. Which of your changes need a human reviewer, which an agent, and which both, and is that written down where delivery pressure cannot change it?
  2. What happens to a generated test before a person sees it: does it have to build, pass repeatedly and add coverage first?
  3. Next to your count of merged PRs, do you report review coverage, change fail rate, rework, incidents per PR beside total incidents, and time to recover?

Sources

Every source was read live on the date in the Read column.

Source Published Label Read URL
Fiona Chen and James Stratton, Harvard, "Artificial Intelligence in the Firm: Bottlenecks in Software Production" Current version 9 November 2026 Controlled study, quasi-experimental; descriptive counts are vendor telemetry; the model is a forecast 2026-11-14 https://fion.ac/jellyfish.pdf
Faros Research, "The Speed Trap: 8 takeaways from our latest AI engineering research" 18 September 2026 Vendor telemetry, from a vendor that sells the remedy 2026-11-14 https://www.faros.ai/blog/ai-speed-trap-takeaways
Neely Dunlap, Faros, "How AI affects developer productivity and engineering workflows" 13 October 2026 Vendor telemetry, from a vendor that sells the remedy 2026-11-14 https://www.faros.ai/blog/how-ai-affects-developer-productivity
Faros, "The AI Speed Trap" report page (form-gated; report not read) Live page, undated Vendor telemetry, from a vendor that sells the remedy 2026-11-14 https://www.faros.ai/research/speed-trap
Faros, "The AI Acceleration Whiplash" research page April 2026 Vendor telemetry, from a vendor that sells the remedy 2026-11-13 https://www.faros.ai/research/ai-acceleration-whiplash
David Loker, CodeRabbit, "Our new report: AI code creates 1.7x more problems" 17 December 2025 Vendor telemetry, from a vendor that sells AI review 2026-11-14 https://www.coderabbit.ai/blog/state-of-AI-vs-human-code-generation-report
Frömmgen et al., Google, "Resolving Code Review Comments with Machine Learning", ICSE-SEIP 2024 2024 Self-reported, company field report 2026-11-14 https://research.google/pubs/resolving-code-review-comments-with-machine-learning/
Vijayvergiya et al., Google, "AI-Assisted Assessment of Coding Practices in Modern Code Review", AIware 2024 May 2024 Controlled study, randomized; self-reported, company field report 2026-11-13 https://arxiv.org/html/2405.13565v1
Maddila et al., Meta, "AI-Assisted Fixes to Code Review Comments at Scale" 17 July 2025 Controlled study, randomized; self-reported, company field report 2026-11-13 https://arxiv.org/abs/2507.13499
Tantithamthavorn et al., Atlassian, "RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian", ICSE 2026 SEIP v2, 20 January 2026 Self-reported, company field report 2026-11-13 https://arxiv.org/abs/2601.01129
Alshahwan et al., Meta, "Automated Unit Test Improvement using Large Language Models at Meta" 14 February 2024 Self-reported, company field report 2026-11-13 https://arxiv.org/abs/2402.09171
Foster et al., Meta, "Mutation-Guided LLM-based Test Generation at Meta" 22 January 2025 Self-reported, company field report 2026-11-13 https://arxiv.org/abs/2501.12862
He, Miller, Agarwal, Kästner and Vasilescu, Carnegie Mellon, "Speed at the Cost of Quality", MSR 2026 v3, 26 January 2026 Controlled study, quasi-experimental 2026-11-13 https://arxiv.org/html/2511.04427v3
Harvey and DeBellis, "Announcing the 2025 DORA Report", Google Cloud blog 23 September 2025 Self-reported, survey 2026-11-13 https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report
Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog 23 September 2025 Self-reported, survey 2026-11-13 https://cloud.google.com/blog/products/ai-machine-learning/introducing-doras-inaugural-ai-capabilities-model
Harvey and Park, "From adoption to impact: Putting the DORA AI Capabilities Model to work", Google Cloud blog 9 December 2025 Recommendation, not a figure 2026-11-13 https://cloud.google.com/blog/products/ai-machine-learning/from-adoption-to-impact-putting-the-dora-ai-capabilities-model-to-work
DORA, "DORA's software delivery performance metrics" Updated 5 January 2026 Self-reported, survey 2026-11-13 https://dora.dev/guides/dora-metrics/
DORA, "Accelerate State of DevOps Report 2024", v. 2024.3 v. 2024.3, announced 22 October 2024 Self-reported, survey 2026-11-13 https://services.google.com/fh/files/misc/2024_final_dora_report.pdf
Cortex, "Engineering in the Age of AI: 2026 Benchmark Report" Blog post 12 November 2025 Vendor telemetry 2026-11-13 https://go.cortex.io/rs/563-WJM-722/images/2026-Benchmark-Report.pdf
LinearB, "2026 Software Engineering Benchmarks Report" Undated PDF; LinearB's blog of 4 May 2026 Vendor telemetry 2026-11-13 https://assets.linearb.io/image/upload/v1765300694/resources/LinearB_2026_Software_Engineering_Benchmarks_Report.pdf
GitClear, "AI Copilot Code Quality: 2025 Look Back at 12 Months of Data" 2025 Vendor telemetry 2026-11-13 https://www.gitclear.com/ai_assistant_code_quality_2025_research
Uplevel, "Can GenAI Actually Improve Developer Productivity?" 18 October 2024, updated 21 July 2026 Vendor telemetry, comparison group 2026-11-13 https://uplevelteam.com/blog/ai-for-developer-productivity
GitHub, "Does GitHub Copilot improve code quality? Here's what the data says" 18 November 2024 Controlled study, randomized, vendor-run 2026-11-13 https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says/