Which skills the engineers needed: read, test, specify, see the system

This page is part of the second half of Deploying Agents at Scale: What Enterprise Adoption Actually Looks Like, Agentic Engineering Days Zurich, 20 November 2026. The talk promised "which skills their engineers needed".

Two findings first. No study read for this page compares teams that succeeded with teams that did not by the skills of their engineers. What the studies show is what went wrong when a skill was missing, and which practices went with better results. And the four strands are not equally strong: review has a randomized trial and an independent evaluation of agent pull requests; testing and specification rest mostly on benchmark experiments, which measure agents rather than engineers; systems thinking rests on DORA's survey, so it is the thinnest.

Every figure carries a label, so you can judge it yourself. A controlled study means researchers compared a treated group with a comparison group: randomized when people were assigned at random, quasi-experimental when the comparison came from adoption dates, agent evaluation when an agent was run on real tasks and scored, and benchmark experiment when models or agents were run with and without a treatment on fixed tasks, with no engineers as subjects. Vendor-run means the company that ran it sells an AI tool. Vendor telemetry is a vendor's measurements of its customers' systems. Self-reported, survey means the figure rests on what respondents said. No figure here is a forecast. A quoted piece of advice is marked recommendation, not a figure. Every source was read live in its primary on 2026-11-13; the table at the end gives each one with its date and link.

Review: reading, debugging and judging code

Controlled study, randomized, vendor-run. Anthropic's trial of 52 mostly junior developers learning a new Python library: the AI group averaged 50% on a quiz taken minutes later and the hand-coding group 67%. "The largest gap in scores between the two groups was on debugging questions." The AI group's time saving was not statistically significant. Patterns of heavy delegation averaged less than 40% on the quiz, and patterns that used the assistant to build understanding averaged 65% or higher; the authors say this analysis "does not draw a causal link".

Controlled study, agent evaluation. METR ran an agent on 18 real issues from large open-source repositories. Its pull requests passed the maintainers' own tests on 38% (plus or minus 19%, 95% interval), and of the 15 read by hand, "none of them are mergeable as-is". Each needed an estimated 42 minutes of fixing on average, 26 minutes for those that passed the tests.

Self-reported, survey. Stack Overflow 2025, over 49,000 responses from 177 countries: 46% distrust the accuracy of AI tools and 33% trust it; experienced developers "highly trust" least (2.6%) and "highly distrust" most (20%). The top frustration, for 66%, is "AI solutions that are almost right, but not quite", and 45% say "Debugging AI-generated code is more time-consuming".

Self-reported, survey. Microsoft Research, 319 knowledge workers and 936 examples: "higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking". Knowledge workers in general, not engineers only.

Controlled study, randomized. Borg and colleagues, 151 participants, 95% of them professional developers: when developers evolved code written with or without AI, the randomized phase found "no significant differences in subsequent evolution with respect to completion time or code quality". Code written with an assistant was not harder for the next engineer to change.

Testing: checking the work against something other than the author

Controlled study, agent evaluation. Of the 15 agent pull requests METR read, test coverage was inadequate in 91% of those that failed the maintainers' tests (11) and in 100% of those that passed (4). Every one had at least three of five problems; 60% had four and 20% all five.

Controlled study, benchmark experiment. Giving GPT-4 the tests along with the problem solved an additional 12.0% of MBPP problems and 8.5% of HumanEval problems, against 80.5% and 82.3% solved without them (Mathews and Nagappan).

Controlled study, quasi-experimental. 806 Cursor-adopting repositories against a matched control: static analysis warnings up 30.3% and code complexity up 41.6%, which the authors find drive the later slowdown. They identify "quality assurance as a major bottleneck for early Cursor adopters" (He et al.).

Vendor telemetry. Faros, 22,000 developers across 4,000 teams: time in QA up 300.6%, "the largest deterioration among the major efficiency and flow metrics in this dataset", while the rise in pull requests merged without review grew from 31.3% in April 2026 to 76.3%.

Specification: saying what done means before the agent starts

Controlled study, benchmark experiment. On Ambig-SWE, an underspecified version of a standard benchmark, models "struggle to distinguish between well-specified and underspecified instructions", but when they ask, the answers bring "significant improvements in performance, up to 74% over the non-interactive settings".

Controlled study, benchmark experiment. Clarifying ambiguous requirements before generating raised GPT-4's Pass@1 on MBPP-sanitized from 70.96% to 80.80%, with ten participants answering the questions (ClarifyGPT).

Controlled study, benchmark experiment. With an AGENTS.md file, agents on 124 pull requests in 10 repositories had a median runtime 28.64% lower and output tokens 16.58% lower, with comparable task completion (Lulla et al.).

Controlled study, benchmark experiment. Context files did "not generally improve task success rates, while increasing inference cost by over 20% on average"; they help "for specifying non-standard coding practices" (Gloaguen et al.).

Vendor telemetry. Faros: restarts up 66.7%, which Faros reads as "a useful signal to indicate where AI agents lacked the context to get work right the first time".

A precise, written statement of the task helps, and a generic one costs tokens without helping. These are experiments on agents, so they show what a good specification does, not who on a team could write one.

Systems thinking: the team, the platform and the user

Self-reported, survey. DORA's AI Capabilities Model, from 78 interviews and a survey of almost 5,000 respondents, names seven capabilities that "substantially either amplify or unlock the benefits of AI": a clear AI stance, healthy data ecosystems, AI-accessible internal data, strong version control practices, small batches, a user-centric focus and quality internal platforms. "in the absence of a user-centric focus, AI adoption can have a negative impact on team performance."

Self-reported, survey. DORA: "frequent commits amplify AI's positive influence on individual effectiveness, while the frequent use of rollback features boosts the performance of AI-assisted teams."

Self-reported, survey. DORA 2025: "AI doesn't fix a team; it amplifies what's already there." 90% of respondents use AI at work, more than 80% believe it raised their productivity, and 30% report little or no trust in AI-generated code.

Recommendation, not a figure. DORA: "Encourage teams to become highly proficient in using rollback and revert features."

DORA's capabilities are organizational conditions, not individual skills. They count as systems thinking on this page because each asks an engineer to judge a change against the whole delivery system (its batch size, its rollback, its platform and its user) rather than the diff alone. That reading is this page's, not DORA's.

What this evidence does not show

Checked and not used

Claim Status
Gartner: generative AI will require most of the engineering workforce to upskill through 2027 (a forecast) Unverified: the press release returned HTTP 403, so no figure from it is used
OpenAI, SWE-bench Verified: how many benchmark tasks had underspecified problem statements or unfair tests Unverified: the page returned HTTP 403
A study that compares successful and unsuccessful teams by their engineers' skills Unverified: none found

Sources

Every source was read live on 2026-11-13 for the research behind this page.

Source Published Label Read URL
Anthropic, "How AI assistance impacts the formation of coding skills"; paper: Shen and Tamkin, "How AI Impacts Skill Formation", arXiv 2601.20245 29 January 2026 Controlled study, randomized, vendor-run 2026-11-13 https://www.anthropic.com/research/AI-assistance-coding-skills
David Rein, METR, "Research Update: Algorithmic vs. Holistic Evaluation" 13 August 2025 Controlled study, agent evaluation 2026-11-13 https://metr.org/blog/2025-08-12-research-update-towards-reconciling-slowdown-with-time-horizons/
Stack Overflow 2025 Developer Survey, AI section 2025 Self-reported, survey 2026-11-13 https://survey.stackoverflow.co/2025/ai
Lee et al., "The Impact of Generative AI on Critical Thinking", CHI 2025, Microsoft Research 2025 Self-reported, survey 2026-11-13 https://www.microsoft.com/en-us/research/publication/the-impact-of-generative-ai-on-critical-thinking-self-reported-reductions-in-cognitive-effort-and-confidence-effects-from-a-survey-of-knowledge-workers/
Borg et al., "Echoes of AI" arXiv v3, 26 February 2026 Controlled study, randomized 2026-11-13 https://arxiv.org/abs/2507.00788
Mathews and Nagappan, "Test-Driven Development for Code Generation" arXiv v2, 11 June 2024 Controlled study, benchmark experiment 2026-11-13 https://arxiv.org/html/2402.13521v2
He, Miller, Agarwal, Kästner, Vasilescu, "Speed at the Cost of Quality", MSR 2026 arXiv v3, 26 January 2026 Controlled study, quasi-experimental 2026-11-13 https://arxiv.org/abs/2511.04427
Faros AI, "The Speed Trap: 8 takeaways from our latest AI engineering research" 18 September 2026 Vendor telemetry 2026-11-13 https://www.faros.ai/blog/ai-speed-trap-takeaways
Vijayvargiya, Zhou, Yerukola, Sap, Neubig, "Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering" arXiv v3, 21 February 2026 Controlled study, benchmark experiment 2026-11-13 https://arxiv.org/abs/2502.13069
Mu et al., "ClarifyGPT" arXiv v1, 17 October 2023 Controlled study, benchmark experiment 2026-11-13 https://arxiv.org/abs/2310.10996
Lulla et al., "On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents" arXiv v2, 30 March 2026 Controlled study, benchmark experiment 2026-11-13 https://arxiv.org/abs/2601.20404
Gloaguen et al., "Evaluating AGENTS.md" arXiv v3, 29 September 2026 Controlled study, benchmark experiment 2026-11-13 https://arxiv.org/abs/2602.11988
Storer and DeBellis, "Introducing the DORA AI Capabilities Model", Google Cloud blog 23 September 2025 Self-reported, survey 2026-11-13 https://cloud.google.com/blog/products/ai-machine-learning/introducing-doras-inaugural-ai-capabilities-model
Google Cloud blog, "Announcing the 2025 DORA Report: State of AI-assisted Software Development" 23 September 2025 Self-reported, survey 2026-11-13 https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report

Every figure above also has its own row in the evidence index, under "The skills engineers needed".