The productivity paradox of AI coding assistants

Lisa Dziuba · article · 2025-09-12 source ↗ Reader 🖥️

Agent Security Context Rot & Context Limits Measuring AI Impact Software Supply-Chain Security Technical Debt

AI coding assistants create a productivity placebo — they feel fast because of instant dopamine-reward feedback — but measured across independent studies they deliver marginal or negative net productivity for experienced developers on real codebases, degrade code quality and security, and the widely repeated '10x productivity' claim does not survive controlled measurement.

practical · argumentative — Builds a case against the prevailing '10x productivity' narrative by marshaling a sequence of external studies, each addressing one facet (perceived vs. measured speed, quality, security) of the same thesis.

Structure

Part	Role
Team split anecdote + MVP/side-project caveat	frames where the author concedes real gains exist, narrowing the scope of the critique that follows
'Dopamine vs. reality' (METR RCT, Hutchins quote, Stack Overflow survey)	core evidence for the perception/measurement gap
'The quality problem' (context rot, Stack Overflow frustration data)	evidence for degraded output quality, not just speed
'Where is the magical 10x boost?' (GitHub/Microsoft 2023, MIT/Harvard/Microsoft 2024, Faros AI 2025)	directly tests and rejects the 10x claim using controlled/field data
'Security is where the gap shows most clearly' (Stanford 2023, Apiiro 2024)	extends the critique from productivity to security/compliance risk
'How AI assistants create new attack surfaces' (Gemini CLI RCE, Amazon Q poisoned update)	concrete incident evidence for the structural supply-chain risk claim
'The 70% problem' (Osmani reference, demo-vs-production framing)	explains the mechanism behind why experienced developers specifically lose time
'Business opinion vs. developer opinion'	conclusion diagnosing why the 10x narrative persists despite the evidence

Key terms

productivity paradox / productivity placebo
The gap between how fast AI coding assistance *feels* (dopamine-driven instant feedback in the editor) and how fast it actually *is* when measured against working, production-ready code.
context rot Context Rot & Context Limits
The degradation of AI output quality as more context accumulates within a session, because the model starts pulling in irrelevant details from earlier prompts.
the 70% problem / demo gap
AI assistants reliably get a task about 70% of the way to a working solution, but the remaining 30% (edge cases, architecture fixes, tests, cleanup) is the hard, slow part; a demo only has to run once, production code has to run a million times without breaking, and that gap is why experienced developers in particular don't see net time savings.

Propositions

explicitAcross multiple independent studies (METR's RCT, the 2025 Stack Overflow Developer Survey, Faros AI's telemetry), AI coding assistants' measured productivity effect is marginal or negative for experienced developers working in codebases they already know, even though those developers subjectively feel faster.conf: high
supportsThe Acceleration Whiplash · p5 — Both independently make the same core claim: telemetry/measured data shows developers feel more productive with AI even as review/coordination costs rise, which self-report alone would miss.
explicitIn METR's July 2025 RCT, experienced open-source developers using AI tools (mainly Cursor Pro with Claude 3.5/3.7 Sonnet) were on average 19% slower, despite predicting beforehand they would be 24% faster and still believing afterward they had been sped up by roughly 20%.conf: high
supportsPresentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p14 — Both cite the identical METR 2025 RCT finding (developers felt faster, measured 19%/20% slower) as evidence that self-assessed AI productivity cannot be trusted.
explicitOutput quality degrades within a session as context accumulates ('context rot'), because the model pulls in irrelevant details from earlier prompts, producing bloated or off-target code that looks correct but doesn't solve the actual problem.conf: medium
explicitThe claim that AI makes developers '10x' more productive does not hold up against controlled research: GitHub/Microsoft's 2023 vendor-run benchmark found a 55.8% speedup concentrated among less-experienced developers on scaffolding work, and the MIT/Harvard/Microsoft 2024 field study of 4,867 professional developers found a 26.08% task-completion increase, again concentrated among junior/newer hires, with senior developers already familiar with the codebase seeing little or no measurable speed-up.conf: high
explicitAI-enabled parallelism (Faros AI's July 2025 telemetry across 10,000+ developers in 1,255 teams: +9% tasks, +47% pull requests per developer per day) trades typing time for more context-switching, orchestration, and validation overhead across more parallel workstreams, canceling out much of the apparent speed-up.conf: medium
explicitAI-generated code introduces measurably more security risk than human-written code: Apiiro's 2024 research found 322% more privilege-escalation paths, 153% more design flaws, a 40% increase in secrets exposure, a 2.5x higher rate of critical (CVSS 7.0+) vulnerabilities, and 60% more reviewer security comments — while AI-assisted commits were merged into production 4x faster, bypassing normal review depth.conf: medium
supportsThe Acceleration Whiplash · p3 — Both document rising downstream quality/incident costs (bugs, vulnerabilities, review burden) accompanying AI-adoption-driven throughput, from different vendor telemetry sources (Apiiro's security metrics here, Faros's delivery metrics there).
explicitAI coding assistants expand the software supply-chain attack surface itself (new runtimes, plugins, MCP connections) as shown concretely by the 2025 Google Gemini CLI remote-code-execution bug and the 2024 poisoned Amazon Q VS Code extension update that instructed the assistant to delete local files and shut down AWS EC2 instances.conf: high
explicitAI closes the 'demo gap' quickly (getting to a working-looking first version, roughly '70% of the way') but the remaining ~30% — edge cases, architecture fixes, tests, production cleanup — is where experienced developers who already know the right solution lose the most time, because the assistant adds friction rather than removing it for them.conf: medium
supportsHow AI will change software engineering · p3 — Both describe the same underlying mechanism — fast AI-generated output that looks done but isn't — as the hidden cost that offsets apparent speed gains, framed by McMahon's piece as 'flab' that increases the rest of the team's downstream cost.
implicitThe '10x engineer'/10x-AI-productivity narrative persists disproportionately in business leadership relative to developers because it offers a politically convenient justification ('do more with less') that doesn't require engaging with where the actual delivery bottlenecks (design review, PR queues, test failures, context switching, deployment waits) actually lie.conf: medium

Arguments

The common claim that AI coding assistants deliver dramatic (even '10x') productivity gains for professional software development does not hold up to controlled measurement — especially for experienced developers working on real, already-familiar codebases. strength: medium

Explicit premises:

METR's RCT found experienced OSS developers 19% slower with AI despite believing they were faster.
The 2025 Stack Overflow survey found only 16.3% of developers report AI improved productivity 'to a great extent,' with 41.4% reporting little or no effect.
The 2023 GitHub/Microsoft and 2024 MIT/Harvard/Microsoft studies found real but modest gains (55.8% and 26.08%) concentrated among less-experienced developers doing scaffolding-type work, not seniors.
Faros AI's 2025 telemetry shows AI-enabled parallelism is partly consumed by increased context-switching and review/orchestration overhead.

Implicit premises:

The specific study populations (OSS maintainers for METR; Stack Overflow's self-selected respondents; enterprise developers at Microsoft/Accenture/a Fortune 100 firm for the MIT study) generalize to professional software development broadly despite differing tasks, tools, and time windows.
Telemetry and RCT data are more trustworthy than self-reported developer sentiment or vendor-run benchmarks, and the specific studies foregrounded are representative of the evidence base rather than selected because they support the thesis.

Inferences:

Headcount or roadmap decisions premised on an assumed 10x AI productivity multiplier are likely miscalibrated, and the business pressure to believe the narrative may itself be sustaining it independent of the evidence.

Narratives

The author's own Cerbos dev team is split between an AI-enthusiast camp (Cursor/Claude Code for shipping speed) and a skeptical camp; a quoted teammate calls agent code output untrustworthy and 'an instant tech-debt factory' while conceding it's useful as a codebase-grokking tool, and separately flags that patterns learned while using these tools break with every model update.
→ Grounds the abstract productivity-paradox argument in the author's own team's lived, ongoing disagreement, and supplies two observations (demo-vs-production distrust; model-update instability) not covered by the cited external studies.
Marcus Hutchins's quote describing LLMs as hijacking 'the human brain's reward system,' giving the feeling of achievement without the heavy lifting.
→ Supplies the psychological mechanism behind the METR illusion (why developers felt faster while measurably being slower).

Questions addressed

✅ Do AI coding assistants actually make professional developers more productive?
Modestly and unevenly — real gains concentrated in MVP/scaffolding/junior-developer contexts; marginal-to-negative for experienced developers working in codebases they already know well.
✅ Where does the oft-repeated '10x productivity' claim come from, and does it hold up?
It does not hold up against controlled studies; the author attributes its persistence partly to a business incentive to believe a convenient 'do more with less' narrative.
⭕ Are the studies juxtaposed here (different years, populations, tools, and methodologies) actually measuring a comparable quantity, such that their specific percentages can be meaningfully compared or summed into one 'productivity paradox' picture?

Gaps

incompleteThe article juxtaposes studies with materially different methodologies, populations, task types, and time windows (2023 vendor-run benchmark on a toy HTTP-server task; 2024 enterprise field study; 2025 METR RCT on real OSS repos; 2025 Faros telemetry) as though they measure the same underlying quantity ('AI coding productivity'), without discussing whether the specific percentages are actually comparable across such different conditions.
Read next: Any methodological meta-analysis of AI-coding-productivity studies, if one exists in the corpus or elsewhere, would test whether this juxtaposition actually holds up.
incompleteThe security findings (Apiiro's 322% / 153% / 40% / 2.5x / 60% figures) come from a single 2024 vendor-published study, cited without independent replication, methodology detail, or any discussion of Apiiro's own commercial interest (a code-security vendor) in publicizing high AI-code-risk numbers — a pattern this corpus elsewhere explicitly flags when a vendor's own research underwrites an adoption or risk claim.
Read next: extract-ai-daily-brief-chatgpt-work-agent#p9 and similar corpus entries on vendor-sourced claims treated as adequate evidence.
incompleteThe article does not explore what, specifically, would move developers from the '19% slower, more technical debt' regime it documents toward the real gains it concedes exist for junior/newer-hire populations — e.g. richer context engineering, codebase-specific guardrails, or test-based quality gates before merge — leaving its implicit practical conclusion ('don't expect big productivity gains') less actionable than it could be.
Read next: extract-faros-acceleration-whiplash#p9 (raise code quality at the point of AI authoring: richer context, codebase standards, guardrails, test-based quality gates).