Every figure in the talk, with its label, its source and the date it was read

This page goes with Deploying Agents at Scale: What Enterprise Adoption Actually Looks Like, given by Michael Rishi Forrester, AI Startup Advisor, Neo4j Startup Program, and author of Agentic DevOps with Claude Code (Packt, 2026), at Agentic Engineering Days Zurich on 20 November 2026. The session promised to say which numbers are controlled studies, which are vendor telemetry, and which are forecasts, "so you can check my work". This is that list: every figure on the slides, and every figure in the second half that the talk sends you here for.

Every row has its own link. Add the row's name after a # at the end of this page's address to point someone at a single figure, for example #metr-19-percent-longer.

How to read a row

Label says what kind of evidence the figure is. There are four.

Label What it means
Controlled study Researchers set the conditions and measured the result, with a comparison group. Benchmarks and agent evaluations count here when they fix the tasks and measure the output.
Vendor telemetry A company measuring its own customers' systems or its own usage, and publishing the result. It usually sells the fix.
Self-reported What people or organizations say about themselves: surveys, interviews and company statements. It also covers studies that identify AI-written code by the markings the code carries, and my own incident records.
Forecast A prediction or a model's estimate of what will happen. Never evidence of what did.

When a figure could take two labels, it takes the weaker one: a forecast is weaker than self-reported, which is weaker than vendor telemetry, which is weaker than a controlled study.

Source links to where the figure was read. Where a source is a PDF or a long paper, the row names the page.

Read gives the date the figure was read in that source, and any later date it was read again.

Status is one of three:

On the slides

Slide numbers are those of the 25-slide deck: 1 to 16 are taught, and 18 to 24 are the appendix after the close. "Spoken" means the figure is in the speaker notes and said aloud, not shown.

Figure Slides Label Source Read Status
Tasks took 19% longer with AI allowed 2, spoken on 15 Controlled study METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 2026-10-05, re-read 2026-11-14 Verified
Confidence interval +2% to +39% 2 Controlled study METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 2026-10-24, re-read 2026-11-14 Verified
16 developers, 246 tasks 2 Controlled study METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 2026-10-05, re-read 2026-11-14 Verified
Expected a 24% speedup before; believed 20% after 2, spoken on 15 Controlled study METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025 2026-10-05, re-read 2026-11-14 Verified
Nearly 5,000 technology professionals surveyed 4 Self-reported DORA, "DORA 2025 State of AI-assisted Software Development report", research.google 2026-10-18, re-read 2026-10-29, 2026-11-14 Verified
22,000 developers, 4,000+ teams, two years of telemetry 4, 6 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14 Verified
"30% to 50% of developers ... did not want to do them without AI" 5 Self-reported METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 2026-10-24, re-read 2026-11-14 Verified
Raw 2026 estimates: 10 returning developers -18% (interval -38% to +9%); 47 new developers -4% (interval -15% to +9%) Spoken on 5 Controlled study METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026 2026-10-24 Verified
Epics completed per developer up 66% 6 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14, 2026-11-17 Verified
Task throughput per developer up 33.7% Spoken on 6 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-24, re-read 2026-11-14 Verified
Incidents-to-PR ratio up 242.7% (rounded on screen to 243%) 6, 7, 9 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14, 2026-11-17 Verified
Code churn up 861% 6, 7 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14, 2026-11-17 Verified
Median time in review up 441.5% (rounded on screen to 442%) 6 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14, 2026-11-17 Verified
PRs merged with no review up 31.3% (rounded on screen to 31%). A rise over the low-adoption baseline, not a share of PRs 6, 8 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-10-05, re-read 2026-11-14, 2026-11-17 Verified
April: monthly incidents up 57.9% (rounded on screen to 58%) 8 Vendor telemetry Faros AI, "Ten takeaways from the AI Engineering Report 2026: The Acceleration Whiplash", 12 April 2026 2026-11-14 Verified
Q3: incidents per PR up 14.5% (rounded on screen to 15%, exact in the appendix) 7, 9, 19 Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-10-05, re-read 2026-11-14 Verified
Q3: code churn still up 71.6% (rounded on screen to 72%) 7 Vendor telemetry Faros AI, "The Speed Trap" report, behind a sign-up form 2026-10-05 Unverified: not read at the primary
Q3: deployment frequency up 13.8% (rounded on screen to 14%), among about 10% of teams with deployment data 7 Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-10-24, re-read 2026-10-29, 2026-11-14 Verified
Roughly 10% of Faros's teams have deployment data, the scope of the 13.8% 7 Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-10-25, re-read 2026-10-29, 2026-11-14 Verified
Q3: unreviewed merges up 76.3% (rounded on screen to 76%) 8 Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-10-05, re-read 2026-11-14 Verified
Q3: time in QA up 300.6% (rounded on screen to 301%) 8 Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-10-05, re-read 2026-11-14 Verified
Q3: monthly incidents up 125.4% (rounded on screen to 125%, exact in the appendix) 8, 9, 19 Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-10-05, re-read 2026-11-14 Verified
200,000 onboarded LLM Suite users within eight months 10 Self-reported JPMorgan Chase, "LLM Suite named 2025 Innovation of the Year by American Banker", 3 June 2025 2026-10-05, re-read 2026-11-14 Verified
Over 90% of engineers use AI code assistants 10 Self-reported JPMorgan Chase, line-of-business CEO letters to shareholders, 2025 annual report, page 13 of the PDF 2026-11-14 Verified
"10% to 20%" for tens of thousands of engineers, the bank's own statement Not on a slide Self-reported Reuters, "JPMorgan engineers' efficiency jumps as much as 20% from using coding assistant", 13 March 2025, as Yahoo Finance carries it 2026-11-14 Verified as Reuters reports it
2025 report: 10,000+ developers; team output up, organization metrics flat 11 Vendor telemetry Faros AI, "The AI Productivity Paradox Research Report", 2025 2026-10-05, re-read 2026-11-14 Verified
80% report better individual productivity; 37% report EBIT impact 11 Self-reported McKinsey, "The state of AI in 2026: On the road to ROI", as reported in press coverage 2026-10-05 Verified in secondary coverage
Nearly three-quarters of high performers redesigned workflows, against about a quarter of others 13 Self-reported McKinsey, "The state of AI in 2026: On the road to ROI", as reported in press coverage 2026-10-05 Verified in secondary coverage
21% report a mature governance model for agents 13 Self-reported GRC Report, coverage of Deloitte's State of AI in the Enterprise 2026 2026-10-05, re-read 2026-11-14 Verified in secondary coverage
After AI agents, a pull request took 56.1% longer from submission to merge 14 Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 2026-11-13 Verified; a quasi-experiment with staggered adoption dates, not random assignment
Heavy agentic-review adoption: 10.7% less time to first review, 24.4% lower change failure rate 14, 19, 23 Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified as Faros's published statement; the findings are correlational
Reviews over 5% longer when reviewers saw AI patches (a randomized safety trial at Meta) 14 Controlled study Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" 2026-11-13 Verified
Engineers accepted 73% of the tests that survived Meta's filters 14 Self-reported Foster et al., "Mutation-Guided LLM-based Test Generation at Meta" 2026-11-13 Verified. The slide's source line names Alshahwan et al., whose 73% counts recommendations; the wording here matches Foster et al., linked
Adaptive attacks bypass 12 recent defenses, attack success above 90% for most; most of those defenses had reported near-zero attack success 18 Controlled study Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" 2026-11-13 Verified
CaMeL solves 77% of AgentDojo tasks with provable security, against 84% with an undefended system 18 Controlled study Debenedetti et al., "Defeating Prompt Injections by Design" 2026-11-13 Verified
Cortex: PRs per author up 20%, cycle time up 9%, change failure rate up 30%, incidents per PR up 23.5%, PR success rate up 2%, year over year Q3 2024 to Q4 2025 19 Vendor telemetry Cortex, "Engineering in the Age of AI: 2026 Benchmark Report", page 5 2026-11-13 Verified
DORA 2024: each 25% increase in AI adoption goes with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability 19 Self-reported DORA, "Accelerate State of DevOps Report 2024", pages 39 and 40 2026-11-13 Verified
Faros Q3: mean time to remediate up 25.8% 19 Vendor telemetry Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
LinearB: acceptance rate, "the percentage of PRs that get merged within X=30 days of being created (or moved out of draft)"; manual PRs 84.4%, AI PRs 32.7% 19 Vendor telemetry LinearB, "2026 Software Engineering Benchmarks Report", pages 11 and 12 2026-11-13 Verified
52 mostly junior developers; AI group 50% on the quiz, hand-coding group 67%; "scored 17% lower"; time saving not statistically significant 20 Controlled study Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 2026-11-13 Verified
METR: 18 issues; agent passed the maintainers' tests on 38% (plus or minus 19%, 95% interval); of 15 read by hand "none of them are mergeable as-is"; 42 minutes to fix on average, 26 minutes for those passing the tests 20 Controlled study METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 2026-11-13 Verified
METR failure table, runs failing tests (n=11) and passing (n=4): core functionality 100% and 25%; testing coverage 91% and 100%; documentation 89% and 75%; linting 73% and 75%; other code quality 64% and 50% 20 Controlled study METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 2026-11-13 Verified
Ambig-SWE: models "struggle to distinguish between well-specified and underspecified instructions"; interaction gives "significant improvements in performance, up to 74% over the non-interactive settings" 20 Controlled study Vijayvargiya et al., "Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering" 2026-11-13 Verified
DORA AI Capabilities Model: seven capabilities that "substantially either amplify or unlock the benefits of AI"; 78 in-depth interviews; a survey of almost 5,000 respondents 20 Self-reported Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog, 23 September 2025 2026-11-13 Verified
Veracode 2026: "The average security pass rate across models is 56%", "barely changed from 55% in the first report"; "roughly 44%" of tasks introduced a vulnerability 21 Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
Veracode 2026: syntax correct "nearly 100% of the time" 21 Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
Perry et al.: SQL injection, 36% of the assistant group against 7% of the control 21 Controlled study Perry et al., ACM CCS 2023 2026-11-13 Verified
GitGuardian: Claude Code-assisted commits 3.2% secret-leak rate against a 1.5% baseline 21 Vendor telemetry GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 2026-11-13 Verified
Agent PRs: each failed CI check cuts the odds of a merge by about 15%, across 33k agent PRs 23 Self-reported Ehsani et al., "Where Do AI Coding Agents Fail?" 2026-11-13, re-read 2026-11-14 Verified
AGENTS.md: median runtime 28.64% lower, output tokens 16.58% lower, 10 repositories, 124 PRs 23 Controlled study Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" 2026-11-13 Verified
Context files: no general gain in task success, inference cost up "over 20% on average"; developer-provided files up 2.4% on average, not significant 23 Controlled study Gloaguen et al., "Evaluating AGENTS.md" 2026-11-13 Verified
52 organizations interviewed, 153 leaders surveyed, preliminary 24 Self-reported MIT NANDA, "The GenAI Divide: State of AI in Business 2025", July 2025; this copy returned an error on 2026-11-14 2026-10-24 Verified
95% saw no return, by the report's own definition of success 24 Self-reported MIT NANDA, "The GenAI Divide: State of AI in Business 2025", July 2025 2026-10-24 Verified
About 800 senior decision-makers (n=801) 24 Self-reported Wharton and GBK Collective, "Accountable Acceleration", 2025 AI Adoption Report, full report 2026-10-05, re-read 2026-11-14 Verified
VPs and above 81%, managers 69% 24 Self-reported Wharton and GBK Collective, "Accountable Acceleration", 2025 AI Adoption Report, full report 2026-10-05, re-read 2026-11-14 Verified

The second half, on this site

The talk teaches what fits in 25 minutes and sends you here for the rest. These are the figures behind the second half, grouped by topic. A figure that appears under two topics has a row under each.

The readiness self-check on this site asks ten questions about your own environment. Its evidence is the rows below and on the slides; it adds one source with no figure, Ona's demonstration of an agent working around its own deny rule, so it has no row of its own here.

Where agent rollouts stall, and what the teams that kept the gains changed

Studies of code and pull requests across many repositories. Where a study finds an agent's pull requests by the markings the pull requests carry, its figures are labeled self-reported.

Figure Label Source Read Status
Cursor adopters: 806 repositories against a matched control group Controlled study He et al., "Speed at the Cost of Quality" 2026-11-13 Verified
Cursor: commits up 55.4% in month one and 14.5% in month two; lines added up 281.3% in month one and 48.4% in month two Controlled study He et al., "Speed at the Cost of Quality" 2026-11-13 Verified
Cursor: static analysis warnings up 30.3%, code complexity up 41.6%, persistent Controlled study He et al., "Speed at the Cost of Quality" 2026-11-13 Verified
Faros Q3 2026: 22,000 developers, 4,000 teams, 12 months Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3 2026: average PR size up 71.8% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3 2026: unreviewed merges up 76.3% (from 31.3% in April), time in QA up 300.6%, monthly incidents up 125.4%, restarts up 66.7% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3 2026: agents open "13–14% of PRs at the leading edge" Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Heavy agentic review: time to first review down 10.7%, change failure rate down 24.4% Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
AIDev: over 456,000 agent PRs, 61,000 repositories, 47,000 developers Self-reported Li et al., "The Rise of AI Teammates in Software Engineering (SE) 3.0" 2026-11-13, re-read 2026-11-14 Verified
AIDev-pop acceptance: human 76.8%; OpenAI Codex 65.3%, Claude Code 52.5%, Cursor 51.4%, Devin 48.9%, GitHub Copilot 38.2% Self-reported Li et al., "The Rise of AI Teammates in Software Engineering (SE) 3.0" 2026-11-13, re-read 2026-11-14 Verified
Agent PRs: each failed CI check cuts the odds of a merge by about 15%, across 33k agent PRs Self-reported Ehsani et al., "Where Do AI Coding Agents Fail?" 2026-11-13, re-read 2026-11-14 Verified
Rejected agent PRs: 228 of 600 annotated (38%) had no meaningful human review Self-reported Ehsani et al., "Where Do AI Coding Agents Fail?" 2026-11-13, re-read 2026-11-14 Verified
AGENTS.md: median runtime 28.64% lower, output tokens 16.58% lower, 10 repositories, 124 PRs Controlled study Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" 2026-11-13 Verified
Context files: no general gain in task success, inference cost up "over 20% on average"; developer-provided files up 2.4% on average, not significant Controlled study Gloaguen et al., "Evaluating AGENTS.md" 2026-11-13 Verified

The security of AI-generated code

Benchmarks and user trials of the security of code that models and assistants write, plus vendor telemetry. The two studies that find AI-written code by its own markings are labeled self-reported.

Figure Label Source Read Status
Copilot: 89 scenarios, 1,689 programs, "approximately 40% to be vulnerable" Controlled study Pearce et al., "Asleep at the Keyboard?", IEEE S&P 2022 2026-11-13 Verified
Replication: vulnerable Python suggestions "reduced from 36.54% to 27.25%" Controlled study Majdinasab et al., 18 November 2023 2026-11-13 Verified
Veracode 2025: 80 tasks, more than 100 LLMs, vulnerabilities in 45% of cases ("in 45 percent of cases") Controlled study Veracode press release, 30 July 2025 2026-11-13 Verified
Veracode 2025: Java over 70%; Python, C# and JavaScript 38% to 45% ("over 70 percent", "between 38 percent and 45 percent") Controlled study Veracode press release, 30 July 2025 2026-11-13 Verified
Veracode 2025: cross-site scripting and log injection failed in 86% and 88% of cases ("in 86 percent and 88 percent of cases") Controlled study Veracode press release, 30 July 2025 2026-11-13 Verified
Veracode 2026: "The average security pass rate across models is 56%", "barely changed from 55% in the first report"; "roughly 44%" of tasks introduced a vulnerability Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
Veracode 2026: syntax correct "nearly 100% of the time" Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
Veracode 2026: best model (GPT-5.5) at 68% Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
Veracode 2026: SQL injection 83%, cryptographic algorithms 87%, cross-site scripting 15%, log injection 12% (pass rates) Controlled study Veracode blog, Natalie Tischler, 28 July 2026 2026-11-13 Verified
BaxBench: 392 tasks; best model (o1) 62% correct; exploits on "around half of the correct programs" Controlled study Vero et al., BaxBench 2026-11-13 Verified
SusVibes: 186 tasks, 12 agent settings; SWE-Agent with Claude 4 Sonnet 57% correct, 11.8% secure Controlled study Zhao et al., "Is Vibe Coding Safe?", ICML 2026 2026-11-13 Verified
Perry et al.: 47 participants, 33 experiment and 14 control Controlled study Perry et al., ACM CCS 2023 2026-11-13 Verified
Perry et al.: SQL injection, 36% of the assistant group against 7% of the control Controlled study Perry et al., ACM CCS 2023 2026-11-13 Verified
Perry et al.: sandboxed directory, 12% secure with the assistant against 29% without Controlled study Perry et al., ACM CCS 2023 2026-11-13 Verified
Sandoval et al.: N=58; critical bugs "at a rate no greater than 10% more than the control" Controlled study Sandoval et al., "Lost at C", USENIX Security 2023 2026-11-13 Verified
Asare et al.: Copilot reproduced the vulnerable code "about 33% of the time", the fix "at a 25% rate" Controlled study Asare et al., Empirical Software Engineering 2026-11-13 Verified
Fu et al.: 733 snippets in GitHub projects; 29.5% of Python and 24.2% of JavaScript affected; 43 CWE categories; up to 55.5% fixed by Copilot Chat Self-reported Fu et al., ACM TOSEM 2025 2026-11-13, re-read 2026-11-14 Verified
Georgia Tech radar: over 43,000 advisories; 74 confirmed cases, 14 critical, 25 high; about 18 in the second half of 2025, 56 in the first three months of 2026, 35 in March 2026 Self-reported Georgia Tech School of Cybersecurity and Privacy, "Bad Vibes: AI-Generated Code is Vulnerable, Researchers Warn", 13 April 2026 2026-11-13, re-read 2026-11-14 Verified
GitGuardian: Claude Code-assisted commits 3.2% secret-leak rate against a 1.5% baseline Vendor telemetry GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 2026-11-13 Verified
GitGuardian: 24,008 unique secrets in MCP-related configuration files, 2,117 valid Vendor telemetry GitGuardian, "The State of Secrets Sprawl 2026", blog, 17 March 2026 2026-11-13 Verified
CodeRabbit: 470 PRs, 320 AI-co-authored and 150 human-only; 10.83 against 6.45 issues per PR; security issues "up to 2.74× higher" Vendor telemetry CodeRabbit blog, 17 December 2025 2026-11-13 Verified

Two teardowns of my own agent incidents

Logs, reports and commits from two sessions I ran myself, read in my public repositories at the commits named in each link. One session each, not a study, so every figure is labeled self-reported. Three figures come from my published incident record, which quotes a session transcript that is not public; they are marked unverified. The second teardown, the workshop fleet, is told in the talk on slide 12, and both are on appendix slide 22.

Figure Label Source Read Status
9 nodes, 3 control plane and 6 workers Self-reported the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md 2026-11-13 Verified
187 pods (July baseline, not August) Self-reported the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md 2026-11-13 Verified
18 persistent volumes (13 bound, 5 available), 17 claims all bound Self-reported the-day-claude-code-deleted-my-cluster, cluster-health-report-2025-07-16.md 2026-11-13 Verified
Prompt of 481 lines and 8 rules; rule 6 quoted Self-reported the-day-claude-code-deleted-my-cluster, my-home-cluster-rebuild-2025-August.md 2026-11-13 Verified
4 running containers found, all on k8s03; 0 on seven nodes; -1 for k8s07 Self-reported DevOpsDaysAtlanta_2026_Cluster_Destruction_Ignite, discovery-20250823_215422.log 2026-11-13 Verified
"k8s03 has 4 running containers (damaged cluster)" Self-reported the-day-claude-code-deleted-my-cluster, issues-01.md 2026-11-13 Verified
About six minutes from discovery to reset Self-reported DevOpsDaysAtlanta_2026_Cluster_Destruction_Ignite, the discovery and reset logs 2026-11-13 Verified
Reset script targets 9 nodes Self-reported the-day-claude-code-deleted-my-cluster, 02-reset-nodes.sh 2026-11-13 Verified
293 pending CSRs deleted Self-reported the-day-claude-code-deleted-my-cluster, CLUSTER-FIXED-REPORT.md 2026-11-13 Verified
261 clusters planned: 250 attendee, 9 instructor, 2 personal Self-reported Unleash_an_Agent_Watch_It_Burn, commit ec9804b 2026-11-13 Verified
"150 clusters are provisioning concurrently" (19:46:56 UTC) Self-reported the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md 2026-11-13 Unverified: public record only, not checked at the primary
"0 attendee clusters exist right now." (19:58:43 UTC); about 12 minutes after the 150 claim Self-reported the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md 2026-11-13 Unverified: public record only, not checked at the primary
0 attendee clusters at workshop time; "Yes. At workshop time there were zero student clusters. That is the hard fact and it's on me." Self-reported the-day-claude-code-deleted-my-cluster, 2026-FLEET-INCIDENT.md 2026-11-13 Unverified: public record only, not checked at the primary
11 keep-set clusters Self-reported Unleash_an_Agent_Watch_It_Burn, DECISION-LOG.md 2026-11-13 Verified
Zero feedback responses Self-reported Unleash_an_Agent_Watch_It_Burn, commit a70e164 2026-11-13 Verified

Review and testing at machine speed

How review and testing changed when agents wrote more of the code, from a large quasi-experimental study, randomized trials, vendors' telemetry and companies' own field reports. Two figures are a model's forecast and are labeled as one.

Figure Label Source Read Status
Harvard sample: 720,542 workers at 655 firms, January 2021 to August 2026 Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 1 and 9 2026-11-13 Verified
After AI agents, submission to merge up 1.99 days, or 56.1% of a 3.56-day baseline Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 2026-11-13 Verified
Backlog time up 59.9%; active review time up 58.5% Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 2026-11-13 Verified
Review comments per PR up 38.6% Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 2026-11-13 Verified
Share of PRs with a comment up 13.1%; share blocked up 38.5% of a 2.77% baseline Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 37 2026-11-13 Verified
Reviewers review familiar code: 82.8% of reviews, against 17.2% at random Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 5 and 36 2026-11-13 Verified
Time to merge up 69.7% in the top repository tercile, 25.4% in the bottom Controlled study Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 40 2026-11-13 Verified
79.1% of firms had used AI in code review by July 2026; 23.3% of review comments by AI; 10.8% of PRs with an AI comment Vendor telemetry Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, pages 38 and 39 2026-11-13 Verified
Coding productivity up 73.7%, review productivity down 7.0%; coding labor down 22.7%, review labor up 43.4% Forecast Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 5 and section 6, page 42 2026-11-13 Verified
Long run at mean demand elasticity: output up 16.1%, coding employment down 33.2%, review labor up 24.9% Forecast Chen and Stratton, "Artificial Intelligence in the Firm: Bottlenecks in Software Production", 9 November 2026 version, page 44 2026-11-13 Verified
CodeRabbit: 470 open-source PRs, 320 AI co-authored and 150 human-only; 10.83 issues per PR against 6.45, about 1.7x; "~1.4–1.7× more critical and major findings"; logic and correctness issues 75% more common in AI PRs Vendor telemetry David Loker, CodeRabbit, "Our new report: AI code creates 1.7x more problems", 17 December 2025 2026-11-14 Verified
Faros: high agentic-review adoption "associated with a 10.7% reduction in time to first review and a 24.4% reduction in change failure rate" Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
Faros: agentic review covers 26% of PRs Vendor telemetry Faros AI, "The Speed Trap" report, behind a sign-up form 2026-11-13 Unverified: not read at the primary
Faros: heavy agentic-review teams see gate erosion at roughly half the general rate Vendor telemetry Faros AI, "The Speed Trap" report, behind a sign-up form 2026-11-13 Unverified: not read at the primary
Faros: agents open 13 to 14% of PRs at the leading edge Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros: median time to first PR review up 114.5% Vendor telemetry Faros AI, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
AutoCommenter: useful ratio plateaued at about 54%; comment-resolution rate about 40% Self-reported Vijayvergiya et al., "AI-Assisted Assessment of Coding Practices in Modern Code Review", AIware 2024 2026-11-13 Verified
Google: "In deployment, code-change authors at Google address 7.5% of all reviewer comments by applying an ML-suggested edit" Self-reported Frömmgen et al., "Resolving Code Review Comments with Machine Learning", ICSE-SEIP 2024 2026-11-14 Verified
Meta MetaMateCR safety trial: reviews over 5% longer when reviewers saw AI patches; no regression when only authors saw them Controlled study Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" 2026-11-13 Verified
MetaMateCR in production: ActionableToApplied rate 19.7% Self-reported Maddila et al., "AI-Assisted Fixes to Code Review Comments at Scale" 2026-11-13 Verified
Atlassian RovoDev: median PR cycle time down 30.8% (20.73 to 14.35 hours); human-written comments down 35.6%; 38.70% of comments led to a code change Self-reported Tantithamthavorn et al., "RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian", ICSE 2026 SEIP 2026-11-13 Verified
Meta TestGen-LLM: 75% of tests built, 57% passed reliably, 25% increased coverage; 11.5% of classes improved; 73% of recommendations accepted Self-reported Alshahwan et al., "Automated Unit Test Improvement using Large Language Models at Meta" 2026-11-13 Verified
Meta ACH: 9,095 mutants, 571 tests, 73% of tests accepted, 36% judged privacy relevant Self-reported Foster et al., "Mutation-Guided LLM-based Test Generation at Meta" 2026-11-13 Verified
Carnegie Mellon: 806 Cursor-adopting repositories against 1,380 matched; lines added up 281.3% in the first month; static analysis warnings up 30.3%; code complexity up 41.6% Controlled study He et al., "Speed at the Cost of Quality", MSR 2026 2026-11-13 Verified

Guardrails in two layers

The evidence behind the split between controls the agent can reason its way around and controls it cannot reach.

Figure Label Source Read Status
Adaptive attacks bypass 12 recent defenses, attack success above 90% for most; most of those defenses had reported near-zero attack success Controlled study Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" 2026-11-13 Verified
Human red-teaming: over 500 participants, $20,000 in prizes, every challenge defeated Controlled study Nasr et al., "The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections" 2026-11-13 Verified
CaMeL solves 77% of AgentDojo tasks with provable security, against 84% with an undefended system Controlled study Debenedetti et al., "Defeating Prompt Injections by Design" 2026-11-13 Verified
Sandboxing cut permission prompts by 84% in Anthropic's internal usage Vendor telemetry Anthropic, "Making Claude Code more secure and autonomous with sandboxing", 20 October 2025 2026-11-13 Verified

What to measure in place of PRs merged

The measures that teams and researchers used in place of counting merged pull requests, with what each showed.

Figure Label Source Read Status
DORA 2024 sample: "nearly 3,000 working professionals" this year; "roughly 39,000 professionals" across the decade Self-reported DORA, "Accelerate State of DevOps Report 2024", page 91, "Who took the survey" 2026-11-13 Verified
DORA 2024: each 25% increase in AI adoption goes with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability Self-reported DORA, "Accelerate State of DevOps Report 2024", pages 39 and 40 2026-11-13 Verified
DORA 2024: each 25% increase in AI adoption goes with documentation quality up 7.5%, code quality up 3.4%, code review speed up 3.1%, approval speed up 1.3%, code complexity down 1.8% Self-reported DORA, "Accelerate State of DevOps Report 2024", page 37, Figure 9 2026-11-13 Verified
DORA 2024 elite cluster: change fail rate 5%, recovery in less than one hour, 19% of respondents; low cluster: change fail rate 40%, 25% of respondents Self-reported DORA, "Accelerate State of DevOps Report 2024", page 13, the performance-levels table 2026-11-13 Verified
DORA 2025: nearly 5,000 technology professionals; "a positive relationship between AI adoption on both software delivery throughput and product performance"; "AI adoption does continue to have a negative relationship with software delivery stability" Self-reported Google Cloud blog, "Announcing the 2025 DORA Report: State of AI-assisted Software Development", 23 September 2025 2026-11-13 Verified
Faros Q3 2026: 22,000 developers across 4,000 teams, 12 months of telemetry Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros: incidents per PR up 242.7% in April, up 14.5% in Q3; monthly incidents up 125.4% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3: PRs merged without review, a 31.3% rise in April, now 76.3% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3: average PR size up 71.8%; restarts up 66.7% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3: code churn still up 71.6% Vendor telemetry Faros AI, "The Speed Trap" report, behind a sign-up form 2026-11-13 Unverified: not read at the primary
Faros Q3: mean time to remediate up 25.8% Vendor telemetry Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
Faros Q3: deployment frequency up 13.8% among the roughly 10% of teams with deployment data Vendor telemetry Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
Faros: heavy agentic-review adoption "associated with a 10.7% reduction in time to first review and a 24.4% reduction in change failure rate"; "The findings are correlational" Vendor telemetry Faros blog, Neely Dunlap, "How AI affects developer productivity and engineering workflows", 13 October 2026 2026-11-13 Verified
Faros April 2026: PR size up 51%, bugs per PR up 28%, incidents per PR 3X, code churn 10X Vendor telemetry Faros AI, "The AI Acceleration Whiplash" research page, April 2026 2026-11-13 Verified
Cortex: PRs per author up 20%, cycle time up 9%, change failure rate up 30%, incidents per PR up 23.5%, PR success rate up 2%, year over year Q3 2024 to Q4 2025 Vendor telemetry Cortex, "Engineering in the Age of AI: 2026 Benchmark Report", page 5 2026-11-13 Verified
LinearB 2026: 8,109,244 pull requests, 4,813 teams, 163,820 active contributors, 42 countries Vendor telemetry LinearB, "2026 Software Engineering Benchmarks Report", pages 4 onward 2026-11-13 Verified
LinearB: acceptance rate, "the percentage of PRs that get merged within X=30 days of being created (or moved out of draft)"; manual PRs 84.4%, AI PRs 32.7% Vendor telemetry LinearB, "2026 Software Engineering Benchmarks Report", pages 11 and 12 2026-11-13 Verified
LinearB: AI-assisted PRs 2.6x larger than unassisted ones Vendor telemetry LinearB, "2026 Software Engineering Benchmarks Report", page 17 2026-11-13 Verified
GitClear: 211 million changed lines, January 2020 to December 2024; changed lines tied to refactoring ("moved") fell from 25% in 2021 to less than 10% in 2024; copy/pasted lines rose from 8.3% to 12.3% Vendor telemetry GitClear, "AI Copilot Code Quality: 2025 Look Back at 12 Months of Data" 2026-11-13 Verified
Uplevel: nearly 800 developers; Copilot access gave "no significant change in efficiency metrics" (cycle time, PR throughput); 41% more bugs; "Sustained Always On" reduced 28% without Copilot, 17% with Vendor telemetry Uplevel, "Can GenAI Actually Improve Developer Productivity?", published 18 October 2024, updated 21 July 2026 2026-11-13 Verified
GitHub: 243 developers recruited, 202 valid submissions (104 with Copilot, 98 without), randomly assigned; 53.2% greater likelihood of passing all 10 unit tests; 13.6% more lines per readability error; 5% more likely to be approved; 1,293 blind reviews Controlled study GitHub blog, "Does GitHub Copilot improve code quality? Here's what the data says", 18 November 2024 2026-11-13 Verified

The skills engineers needed

What the evidence says about review, testing, specification and systems thinking once agents write the code.

Figure Label Source Read Status
52 mostly junior developers; AI group 50% on the quiz, hand-coding group 67%; "scored 17% lower"; time saving not statistically significant Controlled study Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 2026-11-13 Verified
Low-scoring (delegation) patterns averaged less than 40%; high-scoring patterns 65% or higher; "does not draw a causal link" Controlled study Anthropic, "How AI assistance impacts the formation of coding skills", 29 January 2026 2026-11-13 Verified
METR: 18 issues; agent passed the maintainers' tests on 38% (plus or minus 19%, 95% interval); of 15 read by hand "none of them are mergeable as-is"; 42 minutes to fix on average, 26 minutes for those passing the tests Controlled study METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 2026-11-13 Verified
METR failure table, runs failing tests (n=11) and passing (n=4): core functionality 100% and 25%; testing coverage 91% and 100%; documentation 89% and 75%; linting 73% and 75%; other code quality 64% and 50% Controlled study METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 2026-11-13 Verified
METR: all 15 had at least three issues, 60% at least four, 20% all five Controlled study METR, David Rein, "Research Update: Algorithmic vs. Holistic Evaluation", 13 August 2025 2026-11-13 Verified
Stack Overflow 2025: over 49,000 responses from 177 countries Self-reported Stack Overflow 2025 Developer Survey 2026-11-13 Verified
Stack Overflow 2025: distrust 46%, trust 33%, "highly trusting" 3%; experienced developers "highly trust" 2.6%, "highly distrust" 20% Self-reported Stack Overflow 2025 Developer Survey 2026-11-13 Verified
Stack Overflow 2025: 66% "AI solutions that are almost right, but not quite"; 45% "Debugging AI-generated code is more time-consuming"; 84% using or planning to use AI tools Self-reported Stack Overflow 2025 Developer Survey 2026-11-13 Verified
Microsoft Research: 319 knowledge workers, 936 examples; "higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking"; shift "toward information verification, response integration, and task stewardship" Self-reported Lee et al., "The Impact of Generative AI on Critical Thinking", CHI 2025, Microsoft Research publication page 2026-11-13 Verified
Borg et al.: 151 participants, 95% professional developers; randomized Phase 2 found "no significant differences in subsequent evolution with respect to completion time or code quality" Controlled study Borg et al., "Echoes of AI" 2026-11-13 Verified
Mathews and Nagappan: GPT-4 solved 80.5% of MBPP and 82.3% of HumanEval without tests; including tests solved an additional 12.0% and 8.5% Controlled study Mathews and Nagappan, "Test-Driven Development for Code Generation" 2026-11-13 Verified
He et al.: 806 Cursor-adopting repositories against a matched control; static analysis warnings up 30.3%, code complexity up 41.6%; "quality assurance as a major bottleneck for early Cursor adopters" Controlled study He et al., "Speed at the Cost of Quality" 2026-11-13 Verified
Faros Q3 2026: 22,000 developers across 4,000 teams; time in QA up 300.6%, "the largest deterioration among the major efficiency and flow metrics in this dataset"; PRs merged without review, a 31.3% rise in April, now 76.3% Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Faros Q3 2026: restarts up 66.7%, "a useful signal to indicate where AI agents lacked the context to get work right the first time" Vendor telemetry Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research", 18 September 2026 2026-11-13 Verified
Ambig-SWE: models "struggle to distinguish between well-specified and underspecified instructions"; interaction gives "significant improvements in performance, up to 74% over the non-interactive settings" Controlled study Vijayvargiya et al., "Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering" 2026-11-13 Verified
ClarifyGPT: GPT-4 Pass@1 on MBPP-sanitized from 70.96% to 80.80%; ten participants Controlled study Mu et al., "ClarifyGPT" 2026-11-13 Verified
Lulla et al.: 10 repositories, 124 pull requests; with AGENTS.md, median runtime 28.64% lower, output tokens 16.58% lower, "comparable task completion behavior" Controlled study Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" 2026-11-13 Verified
Gloaguen et al.: context files do "not generally improve task success rates, while increasing inference cost by over 20% on average"; useful "for specifying non-standard coding practices" Controlled study Gloaguen et al., "Evaluating AGENTS.md" 2026-11-13 Verified
DORA AI Capabilities Model: seven capabilities that "substantially either amplify or unlock the benefits of AI"; 78 in-depth interviews; a survey of almost 5,000 respondents Self-reported Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog, 23 September 2025 2026-11-13 Verified
DORA 2025: "AI doesn't fix a team; it amplifies what's already there"; 90% use AI at work; more than 80% believe it increased their productivity; 30% report little or no trust in AI-generated code Self-reported Google Cloud blog, "Announcing the 2025 DORA Report: State of AI-assisted Software Development", 23 September 2025 2026-11-13 Verified

Pending

As of 14 November 2026, every second-half research piece has landed and its figures are in the tables above. No figure is pending. A figure added to the talk after that date is listed here, marked pending, until it has been read in its source and dated.

Still open among the figures above:

The fact check behind this page also records figures that were read and left out of the talk, quotes and recommendations that carry no figure, and searches that found nothing. They are not listed here. Two figures that a second-half page names only to say why it does not rely on them are left out of the tables for the same reason: Faros's claim of "many companies running AI review on 50–80% of PRs", which its own report puts at 26%, and Apiiro's "4x velocity, 10x vulnerabilities", whose page did not load.