This page goes with Deploying Agents at Scale: What Enterprise Adoption Actually Looks Like, given by Michael Rishi Forrester, AI Startup Advisor, Neo4j Startup Program, and author of Agentic DevOps with Claude Code (Packt, 2026), at Agentic Engineering Days Zurich on 20 November 2026. The session promised to say which numbers are controlled studies, which are vendor telemetry, and which are forecasts, "so you can check my work". This is that list: every figure on the slides, and every figure in the second half that the talk sends you here for.
Every row has its own link. Add the row's name after a #
at the end of this page's address to point someone at a single figure,
for example #metr-19-percent-longer.
Label says what kind of evidence the figure is. There are four.
| Label | What it means |
|---|---|
| Controlled study | Researchers set the conditions and measured the result, with a comparison group. Benchmarks and agent evaluations count here when they fix the tasks and measure the output. |
| Vendor telemetry | A company measuring its own customers' systems or its own usage, and publishing the result. It usually sells the fix. |
| Self-reported | What people or organizations say about themselves: surveys, interviews and company statements. It also covers studies that identify AI-written code by the markings the code carries, and my own incident records. |
| Forecast | A prediction or a model's estimate of what will happen. Never evidence of what did. |
When a figure could take two labels, it takes the weaker one: a forecast is weaker than self-reported, which is weaker than vendor telemetry, which is weaker than a controlled study.
Source links to where the figure was read. Where a source is a PDF or a long paper, the row names the page.
Read gives the date the figure was read in that source, and any later date it was read again.
Status is one of three:
Slide numbers are those of the 25-slide deck: 1 to 16 are taught, and 18 to 24 are the appendix after the close. "Spoken" means the figure is in the speaker notes and said aloud, not shown.
| Figure | Slides | Label | Source | Read | Status |
|---|---|---|---|---|---|
| Tasks took 19% longer with AI allowed | 2, spoken on 15 | Controlled study | METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 | 2026-10-05, re-read 2026-11-14 | Verified |
| Confidence interval +2% to +39% | 2 | Controlled study | METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 | 2026-10-24, re-read 2026-11-14 | Verified |
| 16 developers, 246 tasks | 2 | Controlled study | METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 | 2026-10-05, re-read 2026-11-14 | Verified |
| Expected a 24% speedup before; believed 20% after | 2, spoken on 15 | Controlled study | METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 | 2026-10-05, re-read 2026-11-14 | Verified |
| Nearly 5,000 technology professionals surveyed | 4 | Self-reported | DORA, "DORA 2025 State of AI-assisted Software Development report", research.google | 2026-10-18, re-read 2026-10-29, 2026-11-14 | Verified |
| 22,000 developers, 4,000+ teams, two years of telemetry | 4, 6 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14 | Verified |
| "30% to 50% of developers ... did not want to do them without AI" | 5 | Self-reported | METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 | 2026-10-24, re-read 2026-11-14 | Verified |
| Raw 2026 estimates: 10 returning developers -18% (interval -38% to +9%); 47 new developers -4% (interval -15% to +9%) | Spoken on 5 | Controlled study | METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 | 2026-10-24 | Verified |
| Epics completed per developer up 66% | 6 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14, 2026-11-17 | Verified |
| Task throughput per developer up 33.7% | Spoken on 6 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-24, re-read 2026-11-14 | Verified |
| Incidents-to-PR ratio up 242.7% (rounded on screen to 243%) | 6, 7, 9 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14, 2026-11-17 | Verified |
| Code churn up 861% | 6, 7 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14, 2026-11-17 | Verified |
| Median time in review up 441.5% (rounded on screen to 442%) | 6 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14, 2026-11-17 | Verified |
| PRs merged with no review up 31.3% (rounded on screen to 31%). A rise over the low-adoption baseline, not a share of PRs | 6, 8 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-10-05, re-read 2026-11-14, 2026-11-17 | Verified |
| April: monthly incidents up 57.9% (rounded on screen to 58%) | 8 | Vendor telemetry | Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 | 2026-11-14 | Verified |
| Q3: incidents per PR up 14.5% (rounded on screen to 15%, exact in the appendix) | 7, 9, 19 | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-10-05, re-read 2026-11-14 | Verified |
| Q3: code churn still up 71.6% (rounded on screen to 72%) | 7 | Vendor telemetry | Faros AI, "The Speed Trap" report, behind a sign-up form | 2026-10-05 | Unverified: not read at the primary |
| Q3: deployment frequency up 13.8% (rounded on screen to 14%), among about 10% of teams with deployment data | 7 | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-10-24, re-read 2026-10-29, 2026-11-14 | Verified |
| Roughly 10% of Faros's teams have deployment data, the scope of the 13.8% | 7 | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-10-25, re-read 2026-10-29, 2026-11-14 | Verified |
| Q3: unreviewed merges up 76.3% (rounded on screen to 76%) | 8 | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-10-05, re-read 2026-11-14 | Verified |
| Q3: time in QA up 300.6% (rounded on screen to 301%) | 8 | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-10-05, re-read 2026-11-14 | Verified |
| Q3: monthly incidents up 125.4% (rounded on screen to 125%, exact in the appendix) | 8, 9, 19 | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-10-05, re-read 2026-11-14 | Verified |
| 200,000 onboarded LLM Suite users within eight months | 10 | Self-reported | JPMorgan Chase, "LLM Suite named 2025 Innovation of the Year by American Banker", 3 June 2025 | 2026-10-05, re-read 2026-11-14 | Verified |
| Over 90% of engineers use AI code assistants | 10 | Self-reported | JPMorgan Chase, line-of-business CEO letters to shareholders, 2025 annual report, page 13 of the PDF | 2026-11-14 | Verified |
| "10% to 20%" for tens of thousands of engineers, the bank's own statement | Not on a slide | Self-reported | Reuters, "JPMorgan engineers' efficiency jumps as much as 20% from using coding assistant", 13 March 2025, as Yahoo Finance carries it | 2026-11-14 | Verified as Reuters reports it |
| 2025 report: 10,000+ developers; team output up, organization metrics flat | 11 | Vendor telemetry | Faros AI, "The AI Productivity Paradox Research Report", 2025 | 2026-10-05, re-read 2026-11-14 | Verified |
| 80% report better individual productivity; 37% report EBIT impact | 11 | Self-reported | McKinsey, "The state of AI in 2026: On the road to ROI", as reported in press coverage | 2026-10-05 | Verified in secondary coverage |
| Nearly three-quarters of high performers redesigned workflows, against about a quarter of others | 13 | Self-reported | McKinsey, "The state of AI in 2026: On the road to ROI", as reported in press coverage | 2026-10-05 | Verified in secondary coverage |
| 21% report a mature governance model for agents | 13 | Self-reported | GRC Report, coverage of Deloitte's State of AI in the Enterprise 2026 | 2026-10-05, re-read 2026-11-14 | Verified in secondary coverage |
| After AI agents, a pull request took 56.1% longer from submission to merge | 14 | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 | 2026-11-13 | Verified; a quasi-experiment with staggered adoption dates, not random assignment |
| Heavy agentic-review adoption: 10.7% less time to first review, 24.4% lower change failure rate | 14, 19, 23 | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified as Faros's published statement; the findings are correlational |
| Reviews over 5% longer when reviewers saw AI patches (a randomized safety trial at Meta) | 14 | Controlled study | Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" | 2026-11-13 | Verified |
| Engineers accepted 73% of the tests that survived Meta's filters | 14 | Self-reported | Foster et al., "Mutation-Guided LLM-based Test Generation at Meta" | 2026-11-13 | Verified. The slide's source line names Alshahwan et al., whose 73% counts recommendations; the wording here matches Foster et al., linked |
| Adaptive attacks bypass 12 recent defenses, attack success above 90% for most; most of those defenses had reported near-zero attack success | 18 | Controlled study | Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" | 2026-11-13 | Verified |
| CaMeL solves 77% of AgentDojo tasks with provable security, against 84% with an undefended system | 18 | Controlled study | Debenedetti et al., "Defeating Prompt Injections by Design" | 2026-11-13 | Verified |
| Cortex: PRs per author up 20%, cycle time up 9%, change failure rate up 30%, incidents per PR up 23.5%, PR success rate up 2%, year over year Q3 2024 to Q4 2025 | 19 | Vendor telemetry | Cortex, "Engineering in the Age of AI: 2026 Benchmark Report", page 5 | 2026-11-13 | Verified |
| DORA 2024: each 25% increase in AI adoption goes with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability | 19 | Self-reported | DORA, "Accelerate State of DevOps Report 2024", pages 39 and 40 | 2026-11-13 | Verified |
| Faros Q3: mean time to remediate up 25.8% | 19 | Vendor telemetry | Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| LinearB: acceptance rate, "the percentage of PRs that get merged within X=30 days of being created (or moved out of draft)"; manual PRs 84.4%, AI PRs 32.7% | 19 | Vendor telemetry | LinearB, "2026 Software Engineering Benchmarks Report", pages 11 and 12 | 2026-11-13 | Verified |
| 52 mostly junior developers; AI group 50% on the quiz, hand-coding group 67%; "scored 17% lower"; time saving not statistically significant | 20 | Controlled study | Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 | 2026-11-13 | Verified |
| METR: 18 issues; agent passed the maintainers' tests on 38% (plus or minus 19%, 95% interval); of 15 read by hand "none of them are mergeable as-is"; 42 minutes to fix on average, 26 minutes for those passing the tests | 20 | Controlled study | METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 | 2026-11-13 | Verified |
| METR failure table, runs failing tests (n=11) and passing (n=4): core functionality 100% and 25%; testing coverage 91% and 100%; documentation 89% and 75%; linting 73% and 75%; other code quality 64% and 50% | 20 | Controlled study | METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 | 2026-11-13 | Verified |
| Ambig-SWE: models "struggle to distinguish between well-specified and underspecified instructions"; interaction gives "significant improvements in performance, up to 74% over the non-interactive settings" | 20 | Controlled study | Vijayvargiya et al., "Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering" | 2026-11-13 | Verified |
| DORA AI Capabilities Model: seven capabilities that "substantially either amplify or unlock the benefits of AI"; 78 in-depth interviews; a survey of almost 5,000 respondents | 20 | Self-reported | Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog, 23 September 2025 | 2026-11-13 | Verified |
| Veracode 2026: "The average security pass rate across models is 56%", "barely changed from 55% in the first report"; "roughly 44%" of tasks introduced a vulnerability | 21 | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| Veracode 2026: syntax correct "nearly 100% of the time" | 21 | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| Perry et al.: SQL injection, 36% of the assistant group against 7% of the control | 21 | Controlled study | Perry et al., ACM CCS 2023 | 2026-11-13 | Verified |
| GitGuardian: Claude Code-assisted commits 3.2% secret-leak rate against a 1.5% baseline | 21 | Vendor telemetry | GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 | 2026-11-13 | Verified |
| Agent PRs: each failed CI check cuts the odds of a merge by about 15%, across 33k agent PRs | 23 | Self-reported | Ehsani et al., "Where Do AI Coding Agents Fail?" | 2026-11-13, re-read 2026-11-14 | Verified |
| AGENTS.md: median runtime 28.64% lower, output tokens 16.58% lower, 10 repositories, 124 PRs | 23 | Controlled study | Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" | 2026-11-13 | Verified |
| Context files: no general gain in task success, inference cost up "over 20% on average"; developer-provided files up 2.4% on average, not significant | 23 | Controlled study | Gloaguen et al., "Evaluating AGENTS.md" | 2026-11-13 | Verified |
| 52 organizations interviewed, 153 leaders surveyed, preliminary | 24 | Self-reported | MIT NANDA, "The GenAI Divide: State of AI in Business 2025", July 2025; this copy returned an error on 2026-11-14 | 2026-10-24 | Verified |
| 95% saw no return, by the report's own definition of success | 24 | Self-reported | MIT NANDA, "The GenAI Divide: State of AI in Business 2025", July 2025 | 2026-10-24 | Verified |
| About 800 senior decision-makers (n=801) | 24 | Self-reported | Wharton and GBK Collective, "Accountable Acceleration", 2025 AI Adoption Report, full report | 2026-10-05, re-read 2026-11-14 | Verified |
| VPs and above 81%, managers 69% | 24 | Self-reported | Wharton and GBK Collective, "Accountable Acceleration", 2025 AI Adoption Report, full report | 2026-10-05, re-read 2026-11-14 | Verified |
The talk teaches what fits in 25 minutes and sends you here for the rest. These are the figures behind the second half, grouped by topic. A figure that appears under two topics has a row under each.
The readiness self-check on this site asks ten questions about your own environment. Its evidence is the rows below and on the slides; it adds one source with no figure, Ona's demonstration of an agent working around its own deny rule, so it has no row of its own here.
Studies of code and pull requests across many repositories. Where a study finds an agent's pull requests by the markings the pull requests carry, its figures are labeled self-reported.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| Cursor adopters: 806 repositories against a matched control group | Controlled study | He et al., "Speed at the Cost of Quality" | 2026-11-13 | Verified |
| Cursor: commits up 55.4% in month one and 14.5% in month two; lines added up 281.3% in month one and 48.4% in month two | Controlled study | He et al., "Speed at the Cost of Quality" | 2026-11-13 | Verified |
| Cursor: static analysis warnings up 30.3%, code complexity up 41.6%, persistent | Controlled study | He et al., "Speed at the Cost of Quality" | 2026-11-13 | Verified |
| Faros Q3 2026: 22,000 developers, 4,000 teams, 12 months | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3 2026: average PR size up 71.8% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3 2026: unreviewed merges up 76.3% (from 31.3% in April), time in QA up 300.6%, monthly incidents up 125.4%, restarts up 66.7% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3 2026: agents open "13–14% of PRs at the leading edge" | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Heavy agentic review: time to first review down 10.7%, change failure rate down 24.4% | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| AIDev: over 456,000 agent PRs, 61,000 repositories, 47,000 developers | Self-reported | Li et al., "The Rise of AI Teammates in Software Engineering (SE) 3.0" | 2026-11-13, re-read 2026-11-14 | Verified |
| AIDev-pop acceptance: human 76.8%; OpenAI Codex 65.3%, Claude Code 52.5%, Cursor 51.4%, Devin 48.9%, GitHub Copilot 38.2% | Self-reported | Li et al., "The Rise of AI Teammates in Software Engineering (SE) 3.0" | 2026-11-13, re-read 2026-11-14 | Verified |
| Agent PRs: each failed CI check cuts the odds of a merge by about 15%, across 33k agent PRs | Self-reported | Ehsani et al., "Where Do AI Coding Agents Fail?" | 2026-11-13, re-read 2026-11-14 | Verified |
| Rejected agent PRs: 228 of 600 annotated (38%) had no meaningful human review | Self-reported | Ehsani et al., "Where Do AI Coding Agents Fail?" | 2026-11-13, re-read 2026-11-14 | Verified |
| AGENTS.md: median runtime 28.64% lower, output tokens 16.58% lower, 10 repositories, 124 PRs | Controlled study | Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" | 2026-11-13 | Verified |
| Context files: no general gain in task success, inference cost up "over 20% on average"; developer-provided files up 2.4% on average, not significant | Controlled study | Gloaguen et al., "Evaluating AGENTS.md" | 2026-11-13 | Verified |
Benchmarks and user trials of the security of code that models and assistants write, plus vendor telemetry. The two studies that find AI-written code by its own markings are labeled self-reported.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| Copilot: 89 scenarios, 1,689 programs, "approximately 40% to be vulnerable" | Controlled study | Pearce et al., "Asleep at the Keyboard?", IEEE S&P 2022 | 2026-11-13 | Verified |
| Replication: vulnerable Python suggestions "reduced from 36.54% to 27.25%" | Controlled study | Majdinasab et al., 18 November 2023 | 2026-11-13 | Verified |
| Veracode 2025: 80 tasks, more than 100 LLMs, vulnerabilities in 45% of cases ("in 45 percent of cases") | Controlled study | Veracode press release, 30 July 2025 | 2026-11-13 | Verified |
| Veracode 2025: Java over 70%; Python, C# and JavaScript 38% to 45% ("over 70 percent", "between 38 percent and 45 percent") | Controlled study | Veracode press release, 30 July 2025 | 2026-11-13 | Verified |
| Veracode 2025: cross-site scripting and log injection failed in 86% and 88% of cases ("in 86 percent and 88 percent of cases") | Controlled study | Veracode press release, 30 July 2025 | 2026-11-13 | Verified |
| Veracode 2026: "The average security pass rate across models is 56%", "barely changed from 55% in the first report"; "roughly 44%" of tasks introduced a vulnerability | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| Veracode 2026: syntax correct "nearly 100% of the time" | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| Veracode 2026: best model (GPT-5.5) at 68% | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| Veracode 2026: SQL injection 83%, cryptographic algorithms 87%, cross-site scripting 15%, log injection 12% (pass rates) | Controlled study | Veracode blog, Natalie Tischler, 28 July 2026 | 2026-11-13 | Verified |
| BaxBench: 392 tasks; best model (o1) 62% correct; exploits on "around half of the correct programs" | Controlled study | Vero et al., BaxBench | 2026-11-13 | Verified |
| SusVibes: 186 tasks, 12 agent settings; SWE-Agent with Claude 4 Sonnet 57% correct, 11.8% secure | Controlled study | Zhao et al., "Is Vibe Coding Safe?", ICML 2026 | 2026-11-13 | Verified |
| Perry et al.: 47 participants, 33 experiment and 14 control | Controlled study | Perry et al., ACM CCS 2023 | 2026-11-13 | Verified |
| Perry et al.: SQL injection, 36% of the assistant group against 7% of the control | Controlled study | Perry et al., ACM CCS 2023 | 2026-11-13 | Verified |
| Perry et al.: sandboxed directory, 12% secure with the assistant against 29% without | Controlled study | Perry et al., ACM CCS 2023 | 2026-11-13 | Verified |
| Sandoval et al.: N=58; critical bugs "at a rate no greater than 10% more than the control" | Controlled study | Sandoval et al., "Lost at C", USENIX Security 2023 | 2026-11-13 | Verified |
| Asare et al.: Copilot reproduced the vulnerable code "about 33% of the time", the fix "at a 25% rate" | Controlled study | Asare et al., Empirical Software Engineering | 2026-11-13 | Verified |
| Fu et al.: 733 snippets in GitHub projects; 29.5% of Python and 24.2% of JavaScript affected; 43 CWE categories; up to 55.5% fixed by Copilot Chat | Self-reported | Fu et al., ACM TOSEM 2025 | 2026-11-13, re-read 2026-11-14 | Verified |
| Georgia Tech radar: over 43,000 advisories; 74 confirmed cases, 14 critical, 25 high; about 18 in the second half of 2025, 56 in the first three months of 2026, 35 in March 2026 | Self-reported | Georgia Tech School of Cybersecurity and Privacy, "Bad Vibes: AI-Generated Code is Vulnerable, Researchers Warn", 13 April 2026 | 2026-11-13, re-read 2026-11-14 | Verified |
| GitGuardian: Claude Code-assisted commits 3.2% secret-leak rate against a 1.5% baseline | Vendor telemetry | GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 | 2026-11-13 | Verified |
| GitGuardian: 24,008 unique secrets in MCP-related configuration files, 2,117 valid | Vendor telemetry | GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 | 2026-11-13 | Verified |
| CodeRabbit: 470 PRs, 320 AI-co-authored and 150 human-only; 10.83 against 6.45 issues per PR; security issues "up to 2.74× higher" | Vendor telemetry | CodeRabbit blog, 17 December 2025 | 2026-11-13 | Verified |
Logs, reports and commits from two sessions I ran myself, read in my public repositories at the commits named in each link. One session each, not a study, so every figure is labeled self-reported. Three figures come from my published incident record, which quotes a session transcript that is not public; they are marked unverified. The second teardown, the workshop fleet, is told in the talk on slide 12, and both are on appendix slide 22.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| 9 nodes, 3 control plane and 6 workers | Self-reported | the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md | 2026-11-13 | Verified |
| 187 pods (July baseline, not August) | Self-reported | the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md | 2026-11-13 | Verified |
| 18 persistent volumes (13 bound, 5 available), 17 claims all bound | Self-reported | the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md | 2026-11-13 | Verified |
| Prompt of 481 lines and 8 rules; rule 6 quoted | Self-reported | the-day-claude-code-deleted-my-cluster, my-home-cluster-rebuild-2025-August.md | 2026-11-13 | Verified |
| 4 running containers found, all on k8s03; 0 on seven nodes; -1 for k8s07 | Self-reported | DevOpsDaysAtlanta_2026_Cluster_Destruction_Ignite, discovery-20250823_215422.log | 2026-11-13 | Verified |
| "k8s03 has 4 running containers (damaged cluster)" | Self-reported | the-day-claude-code-deleted-my-cluster, issues-01.md | 2026-11-13 | Verified |
| About six minutes from discovery to reset | Self-reported | DevOpsDaysAtlanta_2026_Cluster_Destruction_Ignite, the discovery and reset logs | 2026-11-13 | Verified |
| Reset script targets 9 nodes | Self-reported | the-day-claude-code-deleted-my-cluster, 02-reset-nodes.sh | 2026-11-13 | Verified |
| 293 pending CSRs deleted | Self-reported | the-day-claude-code-deleted-my-cluster, CLUSTER-FIXED-REPORT.md | 2026-11-13 | Verified |
| 261 clusters planned: 250 attendee, 9 instructor, 2 personal | Self-reported | Unleash_an_Agent_Watch_It_Burn, commit ec9804b | 2026-11-13 | Verified |
| "150 clusters are provisioning concurrently" (19:46:56 UTC) | Self-reported | the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md | 2026-11-13 | Unverified: public record only, not checked at the primary |
| "0 attendee clusters exist right now." (19:58:43 UTC); about 12 minutes after the 150 claim | Self-reported | the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md | 2026-11-13 | Unverified: public record only, not checked at the primary |
| 0 attendee clusters at workshop time; "Yes. At workshop time there were zero student clusters. That is the hard fact and it's on me." | Self-reported | the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md | 2026-11-13 | Unverified: public record only, not checked at the primary |
| 11 keep-set clusters | Self-reported | Unleash_an_Agent_Watch_It_Burn, DECISION-LOG.md | 2026-11-13 | Verified |
| Zero feedback responses | Self-reported | Unleash_an_Agent_Watch_It_Burn, commit a70e164 | 2026-11-13 | Verified |
How review and testing changed when agents wrote more of the code, from a large quasi-experimental study, randomized trials, vendors' telemetry and companies' own field reports. Two figures are a model's forecast and are labeled as one.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| Harvard sample: 720,542 workers at 655 firms, January 2021 to August 2026 | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 1 and 9 | 2026-11-13 | Verified |
| After AI agents, submission to merge up 1.99 days, or 56.1% of a 3.56-day baseline | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 | 2026-11-13 | Verified |
| Backlog time up 59.9%; active review time up 58.5% | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 | 2026-11-13 | Verified |
| Review comments per PR up 38.6% | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 | 2026-11-13 | Verified |
| Share of PRs with a comment up 13.1%; share blocked up 38.5% of a 2.77% baseline | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 | 2026-11-13 | Verified |
| Reviewers review familiar code: 82.8% of reviews, against 17.2% at random | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 5 and 36 | 2026-11-13 | Verified |
| Time to merge up 69.7% in the top repository tercile, 25.4% in the bottom | Controlled study | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 40 | 2026-11-13 | Verified |
| 79.1% of firms had used AI in code review by July 2026; 23.3% of review comments by AI; 10.8% of PRs with an AI comment | Vendor telemetry | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 38 and 39 | 2026-11-13 | Verified |
| Coding productivity up 73.7%, review productivity down 7.0%; coding labor down 22.7%, review labor up 43.4% | Forecast | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 5 and section 6, page 42 | 2026-11-13 | Verified |
| Long run at mean demand elasticity: output up 16.1%, coding employment down 33.2%, review labor up 24.9% | Forecast | Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 44 | 2026-11-13 | Verified |
| CodeRabbit: 470 open-source PRs, 320 AI co-authored and 150 human-only; 10.83 issues per PR against 6.45, about 1.7x; "~1.4–1.7× more critical and major findings"; logic and correctness issues 75% more common in AI PRs | Vendor telemetry | David Loker, CodeRabbit, "Our new report: AI code creates 1.7x more problems", 17 December 2025 | 2026-11-14 | Verified |
| Faros: high agentic-review adoption "associated with a 10.7% reduction in time to first review and a 24.4% reduction in change failure rate" | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| Faros: agentic review covers 26% of PRs | Vendor telemetry | Faros AI, "The Speed Trap" report, behind a sign-up form | 2026-11-13 | Unverified: not read at the primary |
| Faros: heavy agentic-review teams see gate erosion at roughly half the general rate | Vendor telemetry | Faros AI, "The Speed Trap" report, behind a sign-up form | 2026-11-13 | Unverified: not read at the primary |
| Faros: agents open 13 to 14% of PRs at the leading edge | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros: median time to first PR review up 114.5% | Vendor telemetry | Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| AutoCommenter: useful ratio plateaued at about 54%; comment-resolution rate about 40% | Self-reported | Vijayvergiya et al., "AI-Assisted Assessment of Coding Practices in Modern Code Review", AIware 2024 | 2026-11-13 | Verified |
| Google: "In deployment, code-change authors at Google address 7.5% of all reviewer comments by applying an ML-suggested edit" | Self-reported | Frömmgen et al., "Resolving Code Review Comments with Machine Learning", ICSE-SEIP 2024 | 2026-11-14 | Verified |
| Meta MetaMateCR safety trial: reviews over 5% longer when reviewers saw AI patches; no regression when only authors saw them | Controlled study | Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" | 2026-11-13 | Verified |
| MetaMateCR in production: ActionableToApplied rate 19.7% | Self-reported | Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" | 2026-11-13 | Verified |
| Atlassian RovoDev: median PR cycle time down 30.8% (20.73 to 14.35 hours); human-written comments down 35.6%; 38.70% of comments led to a code change | Self-reported | Tantithamthavorn et al., "RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian", ICSE 2026 SEIP | 2026-11-13 | Verified |
| Meta TestGen-LLM: 75% of tests built, 57% passed reliably, 25% increased coverage; 11.5% of classes improved; 73% of recommendations accepted | Self-reported | Alshahwan et al., "Automated Unit Test Improvement using Large Language Models at Meta" | 2026-11-13 | Verified |
| Meta ACH: 9,095 mutants, 571 tests, 73% of tests accepted, 36% judged privacy relevant | Self-reported | Foster et al., "Mutation-Guided LLM-based Test Generation at Meta" | 2026-11-13 | Verified |
| Carnegie Mellon: 806 Cursor-adopting repositories against 1,380 matched; lines added up 281.3% in the first month; static analysis warnings up 30.3%; code complexity up 41.6% | Controlled study | He et al., "Speed at the Cost of Quality", MSR 2026 | 2026-11-13 | Verified |
The evidence behind the split between controls the agent can reason its way around and controls it cannot reach.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| Adaptive attacks bypass 12 recent defenses, attack success above 90% for most; most of those defenses had reported near-zero attack success | Controlled study | Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" | 2026-11-13 | Verified |
| Human red-teaming: over 500 participants, $20,000 in prizes, every challenge defeated | Controlled study | Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" | 2026-11-13 | Verified |
| CaMeL solves 77% of AgentDojo tasks with provable security, against 84% with an undefended system | Controlled study | Debenedetti et al., "Defeating Prompt Injections by Design" | 2026-11-13 | Verified |
| Sandboxing cut permission prompts by 84% in Anthropic's internal usage | Vendor telemetry | Anthropic, "Making Claude Code more secure and autonomous with sandboxing", 20 October 2025 | 2026-11-13 | Verified |
The measures that teams and researchers used in place of counting merged pull requests, with what each showed.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| DORA 2024 sample: "nearly 3,000 working professionals" this year; "roughly 39,000 professionals" across the decade | Self-reported | DORA, "Accelerate State of DevOps Report 2024", page 91, "Who took the survey" | 2026-11-13 | Verified |
| DORA 2024: each 25% increase in AI adoption goes with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability | Self-reported | DORA, "Accelerate State of DevOps Report 2024", pages 39 and 40 | 2026-11-13 | Verified |
| DORA 2024: each 25% increase in AI adoption goes with documentation quality up 7.5%, code quality up 3.4%, code review speed up 3.1%, approval speed up 1.3%, code complexity down 1.8% | Self-reported | DORA, "Accelerate State of DevOps Report 2024", page 37, Figure 9 | 2026-11-13 | Verified |
| DORA 2024 elite cluster: change fail rate 5%, recovery in less than one hour, 19% of respondents; low cluster: change fail rate 40%, 25% of respondents | Self-reported | DORA, "Accelerate State of DevOps Report 2024", page 13, the performance-levels table | 2026-11-13 | Verified |
| DORA 2025: nearly 5,000 technology professionals; "a positive relationship between AI adoption on both software delivery throughput and product performance"; "AI adoption does continue to have a negative relationship with software delivery stability" | Self-reported | Google Cloud blog, "Announcing the 2025 DORA Report: State of AI-assisted Software Development", 23 September 2025 | 2026-11-13 | Verified |
| Faros Q3 2026: 22,000 developers across 4,000 teams, 12 months of telemetry | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros: incidents per PR up 242.7% in April, up 14.5% in Q3; monthly incidents up 125.4% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3: PRs merged without review, a 31.3% rise in April, now 76.3% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3: average PR size up 71.8%; restarts up 66.7% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3: code churn still up 71.6% | Vendor telemetry | Faros AI, "The Speed Trap" report, behind a sign-up form | 2026-11-13 | Unverified: not read at the primary |
| Faros Q3: mean time to remediate up 25.8% | Vendor telemetry | Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| Faros Q3: deployment frequency up 13.8% among the roughly 10% of teams with deployment data | Vendor telemetry | Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| Faros: heavy agentic-review adoption "associated with a 10.7% reduction in time to first review and a 24.4% reduction in change failure rate"; "The findings are correlational" | Vendor telemetry | Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 | 2026-11-13 | Verified |
| Faros April 2026: PR size up 51%, bugs per PR up 28%, incidents per PR 3X, code churn 10X | Vendor telemetry | Faros AI, "The AI Acceleration Whiplash" research page, April 2026 | 2026-11-13 | Verified |
| Cortex: PRs per author up 20%, cycle time up 9%, change failure rate up 30%, incidents per PR up 23.5%, PR success rate up 2%, year over year Q3 2024 to Q4 2025 | Vendor telemetry | Cortex, "Engineering in the Age of AI: 2026 Benchmark Report", page 5 | 2026-11-13 | Verified |
| LinearB 2026: 8,109,244 pull requests, 4,813 teams, 163,820 active contributors, 42 countries | Vendor telemetry | LinearB, "2026 Software Engineering Benchmarks Report", pages 4 onward | 2026-11-13 | Verified |
| LinearB: acceptance rate, "the percentage of PRs that get merged within X=30 days of being created (or moved out of draft)"; manual PRs 84.4%, AI PRs 32.7% | Vendor telemetry | LinearB, "2026 Software Engineering Benchmarks Report", pages 11 and 12 | 2026-11-13 | Verified |
| LinearB: AI-assisted PRs 2.6x larger than unassisted ones | Vendor telemetry | LinearB, "2026 Software Engineering Benchmarks Report", page 17 | 2026-11-13 | Verified |
| GitClear: 211 million changed lines, January 2020 to December 2024; changed lines tied to refactoring ("moved") fell from 25% in 2021 to less than 10% in 2024; copy/pasted lines rose from 8.3% to 12.3% | Vendor telemetry | GitClear, "AI Copilot Code Quality: 2025 Look Back at 12 Months of Data" | 2026-11-13 | Verified |
| Uplevel: nearly 800 developers; Copilot access gave "no significant change in efficiency metrics" (cycle time, PR throughput); 41% more bugs; "Sustained Always On" reduced 28% without Copilot, 17% with | Vendor telemetry | Uplevel, "Can GenAI Actually Improve Developer Productivity?", published 18 October 2024, updated 21 July 2026 | 2026-11-13 | Verified |
| GitHub: 243 developers recruited, 202 valid submissions (104 with Copilot, 98 without), randomly assigned; 53.2% greater likelihood of passing all 10 unit tests; 13.6% more lines per readability error; 5% more likely to be approved; 1,293 blind reviews | Controlled study | GitHub blog, "Does GitHub Copilot improve code quality? Here's what the data says", 18 November 2024 | 2026-11-13 | Verified |
What the evidence says about review, testing, specification and systems thinking once agents write the code.
| Figure | Label | Source | Read | Status |
|---|---|---|---|---|
| 52 mostly junior developers; AI group 50% on the quiz, hand-coding group 67%; "scored 17% lower"; time saving not statistically significant | Controlled study | Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 | 2026-11-13 | Verified |
| Low-scoring (delegation) patterns averaged less than 40%; high-scoring patterns 65% or higher; "does not draw a causal link" | Controlled study | Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 | 2026-11-13 | Verified |
| METR: 18 issues; agent passed the maintainers' tests on 38% (plus or minus 19%, 95% interval); of 15 read by hand "none of them are mergeable as-is"; 42 minutes to fix on average, 26 minutes for those passing the tests | Controlled study | METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 | 2026-11-13 | Verified |
| METR failure table, runs failing tests (n=11) and passing (n=4): core functionality 100% and 25%; testing coverage 91% and 100%; documentation 89% and 75%; linting 73% and 75%; other code quality 64% and 50% | Controlled study | METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 | 2026-11-13 | Verified |
| METR: all 15 had at least three issues, 60% at least four, 20% all five | Controlled study | METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 | 2026-11-13 | Verified |
| Stack Overflow 2025: over 49,000 responses from 177 countries | Self-reported | Stack Overflow 2025 Developer Survey | 2026-11-13 | Verified |
| Stack Overflow 2025: distrust 46%, trust 33%, "highly trusting" 3%; experienced developers "highly trust" 2.6%, "highly distrust" 20% | Self-reported | Stack Overflow 2025 Developer Survey | 2026-11-13 | Verified |
| Stack Overflow 2025: 66% "AI solutions that are almost right, but not quite"; 45% "Debugging AI-generated code is more time-consuming"; 84% using or planning to use AI tools | Self-reported | Stack Overflow 2025 Developer Survey | 2026-11-13 | Verified |
| Microsoft Research: 319 knowledge workers, 936 examples; "higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking"; shift "toward information verification, response integration, and task stewardship" | Self-reported | Lee et al., "The Impact of Generative AI on Critical Thinking", CHI 2025, Microsoft Research publication page | 2026-11-13 | Verified |
| Borg et al.: 151 participants, 95% professional developers; randomized Phase 2 found "no significant differences in subsequent evolution with respect to completion time or code quality" | Controlled study | Borg et al., "Echoes of AI" | 2026-11-13 | Verified |
| Mathews and Nagappan: GPT-4 solved 80.5% of MBPP and 82.3% of HumanEval without tests; including tests solved an additional 12.0% and 8.5% | Controlled study | Mathews and Nagappan, "Test-Driven Development for Code Generation" | 2026-11-13 | Verified |
| He et al.: 806 Cursor-adopting repositories against a matched control; static analysis warnings up 30.3%, code complexity up 41.6%; "quality assurance as a major bottleneck for early Cursor adopters" | Controlled study | He et al., "Speed at the Cost of Quality" | 2026-11-13 | Verified |
| Faros Q3 2026: 22,000 developers across 4,000 teams; time in QA up 300.6%, "the largest deterioration among the major efficiency and flow metrics in this dataset"; PRs merged without review, a 31.3% rise in April, now 76.3% | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Faros Q3 2026: restarts up 66.7%, "a useful signal to indicate where AI agents lacked the context to get work right the first time" | Vendor telemetry | Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 | 2026-11-13 | Verified |
| Ambig-SWE: models "struggle to distinguish between well-specified and underspecified instructions"; interaction gives "significant improvements in performance, up to 74% over the non-interactive settings" | Controlled study | Vijayvargiya et al., "Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering" | 2026-11-13 | Verified |
| ClarifyGPT: GPT-4 Pass@1 on MBPP-sanitized from 70.96% to 80.80%; ten participants | Controlled study | Mu et al., "ClarifyGPT" | 2026-11-13 | Verified |
| Lulla et al.: 10 repositories, 124 pull requests; with AGENTS.md, median runtime 28.64% lower, output tokens 16.58% lower, "comparable task completion behavior" | Controlled study | Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" | 2026-11-13 | Verified |
| Gloaguen et al.: context files do "not generally improve task success rates, while increasing inference cost by over 20% on average"; useful "for specifying non-standard coding practices" | Controlled study | Gloaguen et al., "Evaluating AGENTS.md" | 2026-11-13 | Verified |
| DORA AI Capabilities Model: seven capabilities that "substantially either amplify or unlock the benefits of AI"; 78 in-depth interviews; a survey of almost 5,000 respondents | Self-reported | Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog, 23 September 2025 | 2026-11-13 | Verified |
| DORA 2025: "AI doesn't fix a team; it amplifies what's already there"; 90% use AI at work; more than 80% believe it increased their productivity; 30% report little or no trust in AI-generated code | Self-reported | Google Cloud blog, "Announcing the 2025 DORA Report: State of AI-assisted Software Development", 23 September 2025 | 2026-11-13 | Verified |
As of 14 November 2026, every second-half research piece has landed and its figures are in the tables above. No figure is pending. A figure added to the talk after that date is listed here, marked pending, until it has been read in its source and dated.
Still open among the figures above:
The fact check behind this page also records figures that were read and left out of the talk, quotes and recommendations that carry no figure, and searches that found nothing. They are not listed here. Two figures that a second-half page names only to say why it does not rely on them are left out of the tables for the same reason: Faros's claim of "many companies running AI review on 50–80% of PRs", which its own report puts at 26%, and Apiiro's "4x velocity, 10x vulnerabilities", whose page did not load.