Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools
Mert Demirer, Leon Musolff, Liyuan Yang · pdf · 2026-05-27 source ↗ Reader 🖥️
Code Was Never the BottleneckDORA & Engineering MetricsMeasuring AI ImpactOrchestration Tax / Human Review BottleneckPull Requests & Async Code Review
Across three successive generations of AI coding tools, large task-level productivity gains (up to 180% more commits) attenuate sharply as they propagate up the software production hierarchy to final shipped output (only 20-30% more releases), because AI and downstream human effort are strong complements rather than substitutes (estimated elasticity of substitution of about 0.25) — so increasing automation of code-writing alone yields only bounded gains in what actually reaches users, a pattern that also shows up in aggregate GitHub activity and across four app marketplaces, where new app releases have risen but end-user usage has not.
theoretical · mixed — An NBER working paper: large-N empirical measurement (matched event studies over 100,000+ GitHub developers plus four app-marketplace panels) combined with a formal hierarchical production model used to interpret the empirical attenuation pattern.
Structure
| Part | Role |
|---|---|
| Background on developer workflow and the three tool generations (autocomplete, sync agents, async agents) | context / setup |
| The hierarchical CES production model (augmentation vs. automation; elasticity of substitution) | theoretical framework |
| Matched event-study estimates of task-level productivity across tool generations (Section 6) | evidence |
| Attenuation across the production hierarchy and the fitted elasticity of substitution (Section 7) | evidence / model-fitting |
| Aggregate GitHub activity and four app-marketplace panels (Section 8) | evidence / robustness at a different level of aggregation |
Key terms
- weak-link hypothesis / bottleneck hypothesis
When stages of a production process are complements rather than substitutes, automating one stage has a bounded effect on total output because the stage with the least capacity caps the whole chain — applied here vertically, within a single software production process, to the layers code/commits/PRs/releases. - elasticity of substitution (between upstream AI output and downstream human effort)
The key structural parameter of the model: how substitutable AI-generated output at one layer is for human effort at the next layer. Estimated at 0.25 — close to Leontief (no substitutability) — indicating strong complementarity. - augmentation vs. automation
Two distinct, mutually exclusive channels through which AI enters a layer in the model: augmentation multiplies the value of a human's own effort (human still does the work, faster); automation produces output directly, but that output is still Leontief-complementary with human review effort — unreviewed AI output is modeled as unproductive. - production hierarchy
The ordered sequence of software-development outputs, each aggregating the layer below it: lines of code → files → commits → pull requests → projects/repositories → releases. - task-level productivity vs. final output
Task-level productivity = how much more of a narrow unit of work (e.g. commits) a developer produces after adopting a tool. Final output = what is actually shipped (releases) and, beyond that, what reaches and is used by end users (app usage).
Propositions
- explicitAutocomplete, sync agents, and async agents each significantly increase coding activity (weekly commits), with respective cumulative effects of 40%, 140%, and 180%, growing monotonically across tool generations.conf: highextendsAI isn't making developers more productive – it's making them busier · p3 — Primary-source relationship for the tool-generation productivity figures (40%/140%/180%) that the secondary article reports as background.
- explicitThese task-level gains attenuate sharply moving up the production hierarchy: the 180% cumulative commit effect falls to roughly 50% for the number of projects and to about 30% for actual releases; for sync agents specifically, a 741% increase in lines of code translates into only a 65% increase in pull requests and just a 20% increase in releases.conf: highextendsAI isn't making developers more productive – it's making them busier · p2 — This paper is the primary NBER source behind that secondary article's identical headline statistic (741% lines of code / 65% PRs / 20% releases); this extraction supplies the full causal design, robustness checks, and model behind the number that article reports directly.supportsThe Acceleration Whiplash · p3 — Faros's cross-company telemetry finds AI adoption correlating with rising downstream costs (bugs, incidents, unreviewed PRs) even as task-level throughput rises; this paper's causally-identified attenuation-to-output finding is an independent, methodologically stronger confirmation of the same basic shape — large upstream gains, much smaller (or costlier) downstream translation.
- explicitThe attenuation pattern is consistent with an estimated elasticity of substitution of about 0.25 between AI-generated upstream output and downstream human effort at each layer, indicating strong complementarity rather than substitutability between AI and human labor along the production chain.conf: highextendsAI isn't making developers more productive – it's making them busier · p4 — Same primary-source relationship for the 'weak-link hypothesis' explanation: this paper supplies the formal model and the fitted elasticity of substitution (0.25) that the secondary article summarizes in prose.supportsState Of AI-Assisted Software Development · p11 — DORA's survey-based claim that downstream capacity (testing, review, integration, deployment) is a fixed constraint limiting local gains is independently confirmed here by a causally-identified, large-N estimate of exactly that constraint (elasticity of substitution of 0.25 between AI output and human downstream effort).
- explicitProductivity effects are larger for less active/lower-baseline developers than for the most active ones, though gains remain substantial throughout the activity distribution.conf: high
- explicitAcross four major app marketplaces (Apple App Store, Google Play, Chrome Web Store, SourceForge), new application releases have accelerated since mid-2025 (sharply on iOS and Chrome, more modestly on Android, not at all on SourceForge), but total user-side usage (ratings/downloads) accumulated within the first three months of release has not increased in any of them.conf: highextendsAI isn't making developers more productive – it's making them busier · p6 — Primary-source relationship for the app-marketplace 'releases up, usage flat' claim; this extraction adds the four-marketplace detail, the small-audience-share trend, and the paper's own acknowledged inability to distinguish quality from discovery-bottleneck explanations.
- explicitThe share of newly released apps that fail to reach even a modest audience within three months has risen across marketplaces since 2025 (e.g. from about 79% to 86% on iOS, 18% to 31% on Chrome), which rules out both a 'new blockbuster apps' explanation and a 'long-tail/better-matching' explanation for where the additional supply is going.conf: high
- explicitDifferent tools within the same sync-agent category show substantially different effect magnitudes — Claude Code's long-run commit effect (199%) is far larger than GitHub Sync Agent's (43%) or OpenAI Codex's (94%) — a gap the paper cannot cleanly attribute to tool quality versus self-selection of more sophisticated adopters into a given tool.conf: high
- explicitThe locus of the human bottleneck is itself moving upward as AI capability advances: autocomplete intervenes only at the code-writing layer, while sync and async agents intervene at progressively higher layers (files, commits, and — for async agents specifically — pull-request creation itself), which the authors argue could narrow the task-level-to-output gap over time rather than leave it fixed.conf: highsupportsHuman judgment doesn't leave the software factory. It relocates. · p2 — Osmani's claim that human judgment relocates upstream (into intent, system design, the quality bar) as agent autonomy increases has an econometric analogue here: as AI capability advances and automates progressively higher layers, the empirical bottleneck is observed moving up the production hierarchy too, rather than disappearing.
- explicitUnreviewed AI-generated output is treated as structurally unproductive in the formal model (a Leontief complementarity between AI output and human review effort at each layer): the model is built so that even unbounded automation of an upstream layer yields only finite downstream output unless the human review stage is itself automated.conf: high
- explicitThe paper cannot distinguish, for the marketplace-usage result, between the marginal AI-enabled apps being genuinely lower quality versus a separate consumer-side bottleneck (discovery, attention, adoption lag) capping usage regardless of quality.conf: high
- explicitThe cross-layer attenuation is not an artifact of AI-generated verbosity or changed commit/PR granularity: placebo tools (GitHub Pro, Docker) show much smaller attenuation at every layer, average lines-per-commit does not fall after adoption, and the pattern holds even for autocomplete, where each suggestion is individually accepted or rejected by the developer.conf: high
- implicitBecause the model treats augmentation and automation as mutually exclusive at a given layer for a given tool (a corner-solution result, not a direct empirical observation), any apparent blend of the two effects for a tool at a layer actually reflects different users or different sessions using the tool in different modes, not a stable intermediate mode.conf: medium
- implicitBy making 'shipped and used' software the terminal measure of value, the paper's own hierarchy implicitly defines productivity around throughput-to-release rather than around other candidate outcomes (defect rates, maintainability, user-reported value) that this corpus's other sources treat as co-equal or more important consequences of the same AI-driven volume increase.conf: medium
Arguments
Large task-level AI productivity gains in software have so far translated only partially into shipped, used software. strength: high
Explicit premises:
- Commit-level gains of up to 180% (cumulative across tool generations).
- Release-level gains of only 20-30% for the same adopters.
- App-marketplace release counts are up since mid-2025, but usage within the first three months is flat across all four marketplaces studied.
Implicit premises:
- Shipped and used output, not lines-of-code, commits, or even pull requests, is the economically relevant measure of software productivity.
- The attenuation pattern is not simply a measurement artifact of AI-generated verbosity or changed work granularity (addressed via placebo and robustness checks).
Inferences:
- Teams and tool vendors should not treat upstream activity metrics (lines of code, commits, even PRs) as a reliable proxy for value delivered, since the paper's central finding is exactly that this proxy increasingly diverges from shipped, used output as AI capability rises.
The attenuation is best explained by strong complementarity (not substitutability) between AI and human effort at each stage of a hierarchical production process, rather than by AI being unable to perform upstream work well. strength: medium
Explicit premises:
- A formal nested-CES hierarchical model makes the elasticity of substitution between AI-generated upstream output and downstream human review/integration effort the parameter governing attenuation.
- Fitting the model to the observed attenuation yields an estimated elasticity of 0.25 — close to a fixed-proportions (Leontief) relationship.
Implicit premises:
- Each production layer requires the output of the layer below as an input, and human review/integration capacity does not itself scale with AI output.
- The model's stylized structure (corner-solution augmentation vs. automation, Leontief review requirement) is an adequate simplification of actual software production.
Inferences:
- If the elasticity of substitution between AI output and human downstream effort were to rise — e.g. future models produce code requiring less review, or review/integration is itself automated — the same upstream gains would propagate further into shipped output; the current attenuation is therefore a function of where AI currently sits in the complementarity structure, not a fixed ceiling.
Narratives
- Robert Solow's 1987 observation that 'you can see the computer age everywhere but in the productivity statistics,' invoked to frame the paper's app-marketplace findings.
→ Positions the flat-usage, rising-small-audience-share finding as a recurrence of a decades-old, unresolved debate about whether micro-level technology gains ever show up in macro/aggregate output — lending the paper's result historical weight and signaling the authors see this as a structural pattern, not an AI-specific quirk. - The short history of AI coding tools (Section 3.2): GitHub Copilot's 2022 autocomplete, the early-2025 emergence of sync agents (Claude Code, Cursor, GitHub Copilot Agent Mode), and the May 2025 launch of fully autonomous async agents (GitHub's async agent, Codex, Claude Code in async mode).
→ More than background — this timeline is the implicit justification for the paper's event-study design: because the three tool categories arrived in genuinely distinct calendar waves, matching each adoption against a one-year-earlier control cohort and comparing effects 'across generations' is a meaningful empirical exercise rather than an arbitrary categorization.
Questions addressed
- ✅ How do productivity effects of AI coding tools evolve across successive tool generations?
Effects grow monotonically across generations: roughly 40% (autocomplete) → 140% (plus sync agents) → 180% (plus async agents) cumulative effect on commits. - ✅ Do task-level AI productivity gains translate into proportional increases in final output?
No — they attenuate sharply moving up the production hierarchy, consistent with the weak-link/O-ring hypothesis, with an estimated elasticity of substitution of about 0.25 between AI output and human downstream effort. - ⭕ Does flat marketplace usage despite rising app-release counts reflect lower-quality marginal apps or a separate consumer-side discovery/adoption bottleneck?
The authors explicitly state their data cannot distinguish between the two explanations. - ⭕ Is the attenuation pattern the paper documents (AI's gains absorbed as work-in-progress at the review/integration layer) sustainable for the humans actually sitting at that bottleneck — their workload, review backlog, or wellbeing?
Gaps
- incompleteThe paper models the review/integration layer's human capacity as fixed and treats attenuation purely as an output/throughput phenomenon, with no direct measure of review time, backlog growth, or reviewer burnout — even though its own mechanism (surplus upstream output passing through unchanged human capacity) is exactly the setup other sources in this corpus (Faros, DORA) use to describe rising reviewer burden and burnout risk from the same AI-driven volume increase.
Read next: Cross-reference against telemetry-based studies (Faros, DORA) that directly measure review time, PR backlog, and incident/defect rates under rising AI-authored code volume. - incompleteThe paper explicitly cannot distinguish whether flat marketplace usage reflects lower-quality marginal apps or a consumer-side discovery bottleneck, but it doesn't engage with a third possibility its own data seems to hint at: that lowering the fixed cost of producing a 'minimally publishable' app may simply attract projects (personal tools, internal experiments, portfolio pieces) that were never intended to find a broad audience, and that counting these as market 'supply' in the first place may be the wrong frame.
Read next: Look for research that segments new app releases by evident intent (commercial vs. personal/experimental) rather than treating all new listings as attempts to reach a broad audience. - uninformedThe paper explicitly excludes enterprise and internal-only software — likely where most professional, team-based software engineering work actually happens — from its analysis. Its production-hierarchy model and 0.25 elasticity estimate, derived from public-repository and consumer-marketplace data, may not transfer to internal systems with different review norms, continuous-deployment cadences without a discrete 'release' artifact, and different definitions of 'shipped.'
Read next: Look for internal, enterprise-scale telemetry studies (e.g. DORA's survey-based team archetypes) to see whether the same attenuation/elasticity pattern holds inside organizations rather than on public GitHub and consumer app stores.
Closes gaps elsewhere
- Is AI coding on average 17.3x faster or 1.3x faster? Both. — LinkedIn post by Fabrice Bernhard · g5 — Gives a direct empirical/causal answer to this exact Lean question for software specifically: AI's upstream code-writing gains are real and large (up to 741% more lines), but they are largely absorbed as additional work passing through the fixed-capacity review/integration bottleneck rather than converted into proportional shipped output (only 20% more releases) — closer to classical Lean's 'inflates WIP upstream' prediction than to 'frees capacity to attack the bottleneck,' with an estimated elasticity of substitution of just 0.25 indicating strong complementarity and weak pass-through.
Related documents
- Human judgment doesn't leave the software factory. It relocates.
- State Of AI-Assisted Software Development
- AI isn't making developers more productive – it's making them busier
- Agentic Code Review
- The 8 software engineering metrics AI broke
- AI coding creates two kinds of debt. You're only measuring one
- CannotMeasureProductivity
- AI-Assisted Software Development: Team Profiles and Capabilities for Putting Research into Action
- The Acceleration Whiplash
- Five studies that are changing how I think about AI in software engineering