We Studied 150 Developers Using AI (Here's What's Actually Changed...)

Dave Farley (Modern Software Engineering) · video · 2026-01-27 Reader 🖥️

AI as Amplifier Behavioral Code Analysis Changeability & Optionality Cognitive Debt DORA & Engineering Metrics Software Design Fundamentals Technical Debt

A pre-registered, two-phase controlled experiment with 151 mostly-professional developers found that AI-assisted code is no harder or easier for a different developer to maintain later than human-written code, and slightly easier when written by experienced habitual AI users — because AI acts as an amplifier of a developer's existing skill and engineering practice, not a substitute for it, so maintainability outcomes are determined by practice and discipline, not by whether AI was used.

theoretical · mixed — Primarily expository (reporting a pre-registered controlled experiment's findings) in service of a theoretical claim about AI's actual effect on maintainability, with a practical coda (which engineering practices determine the outcome).

Structure

Part	Role
Why maintainability (not initial development speed) is the metric that matters — maintenance dominates lifetime software cost.	motivation / framing
Description of the pre-registered, two-phase experimental design (151 participants; phase one builds with/without AI, phase two evolves the code blind to AI origin, without AI access).	methodology
Headline finding: no significant maintainability difference between AI-assisted and non-AI-assisted code.	core result
Secondary findings: AI speeds initial development substantially; experienced habitual users show a small maintainability edge; developer skill matters more than AI usage (amplifier thesis, consistent with DORA).	secondary results
Named long-term risks (code bloat, cognitive debt) and the specific practices (per Jason Gorman) that determine whether AI amplifies good or bad outcomes.	practical application / risk framing
Self-critique: participants recruited from the author's own channel, plausibly pre-selected for better practice.	limitation / caveat

Key terms

Maintainability
Downstream ease of understanding and evolving code, operationalized in the study as: time for an independent, AI-origin-blind second developer to extend the code; objective code-health/quality metrics (CodeScene); test coverage; and perceived productivity (the SPACE framework) — explicitly multi-dimensional, not one number.
AI as amplifier AI as Amplifier
AI magnifies whatever engineering practice and skill a developer already has — good practice compounds into (slightly) better downstream outcomes, and by analogy to DORA's findings, poor practice would compound into worse outcomes — rather than AI having an independent, uniform effect on code quality in either direction.
Cognitive debt Cognitive Debt
The erosion of a developer's own understanding and skill over time from not thinking hard about code an AI generates — a slow, compounding risk distinct from, but related to, code bloat.
Code bloat
Excess code volume generated because producing code is now nearly free — a driver of complexity in its own right, independent of any single defect or quality problem in the generated code.

Propositions

explicitMaintenance, not initial development, is the dominant cost of software over a system's lifetime — estimated at 3-4x the initial development cost and 50-80% of total cost of ownership — which is why AI productivity studies that measure only initial development speed are measuring the wrong thing.conf: high
explicitA pre-registered, two-phase controlled experiment with 151 participants (95% professional developers) measured maintainability by having phase-one developers build a feature with or without AI assistance on a realistic Java web app, then having a different, AI-origin-blind set of phase-two developers evolve that code without AI access — isolating downstream maintainability from the original author's own familiarity with their own code.conf: high
explicitcontestedThe study found no significant difference in downstream maintenance cost, objective code quality, or test coverage between AI-assisted and non-AI-assisted code — AI-assisted code was neither harder nor easier for a different developer to pick up and evolve later.conf: high
contradictsCode maintainability plummets in the AI coding era · p6 — Farley's controlled, pre-registered experiment found no significant maintainability difference from AI assistance, directly in tension with this large-scale observational study's finding of rising technical debt across eight maintainability metrics in the AI-commit era — a methodology (controlled small-N vs. observational large-N) tension worth surfacing, not a simple error on either side.
contradicts ←Study finds AI assistants help developers produce code that's more likely to be buggy · p1 — This 2022 study found AI assistance produced measurably less secure/buggier code; the later, larger, pre-registered Farley-reported study found no significant difference in objective code quality or maintenance cost between AI-assisted and non-AI-assisted code — a direct empirical tension worth reading together rather than assuming either superseded the other.
explicitAI assistance sped up initial development by roughly 30% overall and roughly 55% for habitual AI users, with no evidence in this study that the speed gain came at a hidden downstream maintainability cost.conf: high
explicitWhen experienced developers habitually used AI, their code showed a small but consistent improvement in later maintainability, plausibly because AI tends to produce 'boring,' idiomatic, unsurprising code, and unsurprising code is generally easier to maintain.conf: medium
explicitDeveloper skill and sound engineering practice matter more than whether AI was used at all; AI functions as an amplifier of whatever practice already exists rather than a substitute for it, a reading consistent with recent DORA research on AI-assisted development.conf: high
supports12) Greger Teigre Wedel: Når sjefen har svaret, tier resten · p14 — Both frame AI as amplifying existing engineering practice, making guardrails/discipline more important, not less, with AI assistance.
supports11) AI-spesial med Vilde Opsal og Anders Sveen · p5 — Both apply the DORA-style amplifier/force-multiplier framing: AI magnifies existing team strength or weakness.
explicitThe study's participant pool was recruited largely from viewers of the author's own 'Modern Software Engineering' channel, which plausibly pre-selects for people already disposed toward good engineering practice — a factor the author himself flags as a possible confound on how well the findings generalize.conf: high
explicitCode bloat (excess code volume, now nearly free to generate) and cognitive debt (eroding developer understanding and skill from not thinking hard about AI-generated code) are two specific long-term risks the study's authors highlight that would not show up in a short-horizon sprint metric.conf: high
explicitcorroboratedGood decomposition — breaking problems into small pieces that both guide AI assistants effectively and localize the 'blast radius' of future changes — is the central durable technical skill that differentiates good software development from AI-generated 'slop,' more than typing speed or AI usage itself.conf: high
supportsAgentic Code Review · p7 — Both treat blast radius as a central variable: Farley argues decomposition should localize it at design time; the other piece argues it should determine how much review a change needs.
supports ←Gene Kim on Vibe Coding - Part III | Episode 412 | Troubleshooting Agile · p4 — Kim's 'specification is the job now' claim converges with Farley's empirically-grounded finding that decomposition is the central durable skill distinguishing good AI-assisted development from slop.
explicitWorking in small batches, iterating with continuous testing/review/refactoring, architecting modular designs that localize blast radius, organizing around end-to-end outcomes, and enabling high-autonomy, timely decisions on the ground are the specific practices (cited from Jason Gorman) that determine whether AI amplifies good or bad engineering outcomes.conf: high

Arguments

AI coding assistants do not, by themselves, damage (and may marginally help) code maintainability, so the real risk in AI-assisted development is not the AI itself but whatever engineering discipline — or its absence — the AI amplifies.

Explicit premises:

A controlled, blind, two-phase experiment found no significant maintainability difference between AI-assisted and non-AI-assisted code.
Experienced habitual AI users showed a small, consistent downstream maintainability improvement.
AI speeds initial development substantially (30% overall, 55% for habitual users).
Developer skill mattered more than AI usage in the study's own data, consistent with DORA research.

Implicit premises:

A controlled experiment isolating downstream maintainability from the original author's own familiarity is a more valid test of 'does AI hurt maintainability' than studies that only measure typing speed or self-reported productivity.
The amplifier framing (AI magnifies existing practice) is the correct causal model for interpreting the result, rather than AI having an independent, uniform effect on code quality.

Inferences:

Teams should expect that improving engineering practice, not whether they adopt AI, will determine whether AI accelerates them toward better or worse systems.
Organizations worried about AI's effect on code quality should audit and invest in underlying engineering discipline rather than restricting AI tool usage.

Narratives

The two-phase, blind experimental design — phase-one developers build with or without AI; phase-two developers, unaware of AI involvement and without AI access, evolve the code — is itself engineered specifically to separate 'did AI help write this' from 'is this actually easier for someone else to maintain.'
→ Supports p2/p3 by establishing the methodological validity behind the headline null result.
Dave Farley discloses that study participants were recruited largely from his own YouTube channel's audience, and volunteers this himself as a likely source of pre-selection bias toward already-good engineering practice.
→ Supports p7 and motivates the gap about how much this confound could be driving the headline findings.

Questions addressed

✅ Does AI-assisted code impose a hidden maintainability cost in exchange for development speed?
No measurable cost was found in this controlled study; AI sped up initial development substantially with no significant downstream maintainability penalty, and a small benefit among habitual expert users.
⭕ Under what conditions does AI amplify good versus bad engineering outcomes?
Developer skill and specific practices (small batches, modular design, blast-radius localization, high autonomy) are asserted as the determining factors, citing DORA and Jason Gorman, but the study itself doesn't appear to test a cohort with poor baseline practice using AI — the 'amplifies bad practice into worse outcomes' half of the thesis is inferred by analogy, not directly measured here.
⭕ How much does the self-selected, channel-recruited participant pool specifically bias the maintainability finding (as opposed to the speed finding)?

Gaps

incompleteFarley flags that participants were recruited from his own channel's audience and were 'presumably... predisposed to doing a better job,' but doesn't follow through on what this means for the study's two distinct findings: self-selection for discipline could fully explain the small maintainability improvement among habitual AI users, independent of any amplifying effect from AI itself, while leaving the null maintainability-difference finding unaffected — he raises, then drops, a confound that could reinterpret his own most interesting result.
Read next: Look for a replication attempt with a less self-selected, randomly recruited professional sample to see whether the habitual-user maintainability edge survives.
incompleteThe 'amplifier' thesis (AI amplifies whatever practice already exists, for better or worse) is only tested on its positive half in this study (skilled/disciplined developers using AI) — there's no reported cohort with poor baseline engineering practice using AI, so the 'amplifies bad practice into worse outcomes' half of the claim is asserted by analogy to DORA's correlational survey findings rather than demonstrated by this study's own controlled design.
Read next: Look for a controlled (not just correlational) study that specifically measures downstream maintainability for developers assessed in advance as having weak baseline engineering practice.
incompleteLarge-scale observational evidence elsewhere (623 million real-world commits) finds AI-assisted commits correlate with rising technical-debt signals across eight maintainability metrics — directly in tension with this study's controlled null/slightly-positive result. Farley doesn't address this kind of large-N observational counter-evidence, and neither study fully reconciles the gap between controlled small-N causal evidence and large-N real-world correlational evidence.
Read next: Compare study designs directly: what differs in codebase type, task duration, and AI-usage pattern between a two-phase controlled Java-webapp experiment and a 623-million-commit real-world corpus, to see whether they're actually measuring the same underlying phenomenon.