Measuring AI's Unintended Consequences

James Shore · rss · 2026-09-22 Reader 🖥️

AI Unit Economics AI Vendor/Cost Lock-In Burnout & Recovery DORA & Engineering Metrics Goodhart's Law Measuring AI Impact Proxy Validity (Construct Validity of Measurement Proxies)Technical Debt

Because AI's unintended consequences — declining maintainability, burnout, and vendor/token lock-in — carry real economic costs but, unlike delivery speed, cannot be measured by a controlled trial and instead surface only gradually and noisily, organizations should track a cluster of imperfect lagging indicators (chiefly value-add percentage, backed by DORA metrics and token usage) and build explicit long-horizon scenario models (a maintenance-cost 'tax,' an excess-code-churn model, and a lock-in variant of both) to inform — never replace — judgment, even though the most trusted of these indicators is itself unusually easy to game.

practical · mixed — Practical: prescribes how to measure and model AI's unintended consequences (not merely to understand them). Form is mixed — expository framing (why unintended consequences matter economically), constructive/argumentative content (proposing and defending 'cost of change,' value-add percentage, and three explicit scenario models), and practitioner narrative (a personal VP-of-Engineering anecdote, a named third-party's real tmux commit-history validation). This is part 3 of a 4-part-plus-epilogue series quantifying AI's impact on software development: part 1 argued why measurement matters and what not to measure (extract-shore-ai-money-worth-part1); part 2 (delivery speed) is being extracted in parallel to extract-shore-ai-delivery-speed-part2.json and does not yet exist in this corpus as of this extraction; this piece covers unintended consequences (maintainability, burnout, lock-in); part 4 (business outcomes, 29 Sep 2026) and a CFO-facing epilogue are still to come. Several propositions here explicitly depend on part 2's speed-trial output as an input, so this extraction should be read as incomplete without it.

Structure

Part	Role
Series recap and roadmap: part 1 (why measure / what not to measure), part 2 (delivery speed), this piece = part 3 (unintended consequences), part 4 (business outcomes, 29 Sep) plus a CFO epilogue.	structural framing
'Does Maintainability Even Matter?' — pre-empts the 'it's just a job' objection by grounding maintainability, burnout, and lock-in in concrete downstream costs (slower future work, attrition/hiring costs, a forced big-bill-or-cutback choice); notes the AI-code-comprehension dispute is unresolved by evidence; states the core epistemic stance that unintended consequences are gradual/fuzzy/noisy/lagging, unlike part 2's rigorous speed trial, so models and judgment must supplement measurement.	motivating problem statement + epistemic framing
'One Ring to Rule Them All' — introduces cost of change as the unifying construct, then three indicators: value-add percentage (with muda/discretionary-non-discretionary distinction and a personal VP anecdote), DORA metrics, and token usage (confounded, advisory-only, watch individual-level outliers).	indicator layer — imperfect, easy-to-collect lagging signals
'One Ring to Find Them' — argues value-add percentage is the most trustworthy indicator, describes a lightweight tracking method, and admits its central weakness: 3-6 month lag to show trends, a year to confirm causation.	operational recommendation + honest limitation
'One Ring to Bring Them All' — four explicit models: maintainability-as-tax (with a worked numeric example and two graphs), excess lines of code (validated against real tmux commit history), lock-in (a variant of the maintainability model), and burnout (explicitly not modelable, only trackable via a mental-exhaustion scale, surveys, turnover, and exit interviews).	constructive methodology — the essay's central positive contribution
'And in the Darkness Bind Them' — critiques value-add percentage's gameability, including a non-fraudulent scenario where a leader defers maintenance to look good before changing jobs; the only offered remedy is part 1's Goodhart's Law discipline (measure approaches, not people), which Shore admits is not a complete answer.	self-critical counter-argument
Closing: development speed and unintended consequences are 'two sides of the same coin' describing how much software gets delivered; whether it's the right software is deferred to part 4.	transition to next essay in the series

Key terms

cost of change
A single unifying construct for AI's unintended consequences: a 'tax' on making changes to software. When low, even complicated changes are achievable; when high, even simple changes can take weeks. Not itself directly measurable — the rest of the essay is indicators and models that approximate it.
value-add percentage
The percentage of development effort spent on work that brings direct business value (new features) versus 'muda' (non-value-add work such as bug fixes, incidents, and manual release cycles). Shore's most-trusted cost-of-change indicator, tracked via lightweight categorization rather than detailed time sheets.
muda (discretionary vs. non-discretionary)
Muda (Japanese, via the Toyota Production System/Lean Software Development) is non-value-add work. Shore splits it further: discretionary muda is chosen (e.g. investing in cost-reducing code changes); non-discretionary muda is imposed (bug fixes, incidents, dependency upgrades). A rising non-discretionary-muda baseline signals declining cost of change.
Spending Efficiency / Cost per Value-Add Day
Now defined (previewed but left undefined in part 1): the output metric of the maintainability model — dollars spent per day of value-adding work. Shore's worked example shows it initially dropping under AI, then crossing back above the pre-AI baseline and ending roughly double the baseline by month 120.
lock-in AI Vendor/Cost Lock-In
The risk that rising AI token costs force a forced choice between a much larger bill, cutting AI use, reducing headcount, or switching to a less capable model. Modeled as a variant of the maintainability model, including a separate estimate for humans maintaining AI-written code once AI use is reduced.
mental exhaustion (as a burnout proxy)
A 1-10 self-report scale (Energized to Nothing Left) Shore proposes folding into the development-speed study and periodic one-question surveys, since he holds that burnout itself 'can't be modeled with any accuracy' but can still be tracked.

Propositions

explicitAI's unintended consequences matter because of real economic impact: unmaintainable code makes features and bug fixes slower (taking time from valuable work), burnout reduces effort and drives attrition (with institutional-knowledge loss and hiring costs), and lock-in can force a choice between a big bill and cutting back development.conf: high
supportsAI productivity is burning out your best engineers · p4 — Both argue that costs invisible to throughput/velocity metrics (validation labor, burnout, attrition, hiring cycles) are still real economic costs that need to be surfaced, not just felt.
explicitThere is widespread agreement that heavy AI use produces code humans have trouble understanding, but disagreement about whether that matters if AI itself is doing the maintenance, and Shore states there isn't much actual evidence on the question.conf: high
explicitUnlike delivery speed (which admits a rigorous controlled trial per part 2), unintended consequences are gradual and fuzzy; their measurements will be noisy and lagging, requiring models and scenario exploration, and decisions should ultimately be informed by measurement and models but driven by judgment.conf: high
explicitCost of change — a tax on making software changes — is offered as the single idea that best captures the majority of AI's unintended consequences, though it is not itself easy to measure directly.conf: high
explicitValue-add percentage is the cost-of-change indicator Shore trusts most because it operates at the level that matters (where time goes) and doesn't require heavyweight time tracking; as VP of Engineering he doubled it over three years, freeing capacity for new initiatives without increasing headcount.conf: high
supportsTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p8 — Both endorse flow/lean-style metrics for continuous improvement and internal trend-watching, explicitly not as a precise external ROI proxy — the same discipline Cutler prescribes generally, applied by Shore specifically to value-add percentage.
explicitValue-add percentage is noisy and slow: trends took 3-6 months to show up, and it took a year of tracking before Shore was confident his cost-of-change reduction efforts had produced lasting results.conf: high
explicitBecause improving cost of change is much harder than making it worse, if AI is causing serious problems they should show up in value-add percentage more quickly than genuine improvements would.conf: high
explicitDORA metrics and token usage are worth tracking as secondary cost-of-change indicators, but token usage specifically is advisory-only — it fluctuates for many reasons besides cost of change (models, harnesses, prompts, and especially individual adoption level), so individual-level usage changes (particularly among already-high-usage people) are more informative than aggregate trends.conf: high
supportsTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p10 — Both treat token-usage metrics as far noisier and less validly precise than their apparent measurability suggests, warning against using tokens as a standalone ROI or cost-of-change signal.
supportsAI-Assisted Software Development: Team Profiles and Capabilities for Putting Research into Action · p5 — Shore's recommendation to watch DORA metrics as an AI unintended-consequence indicator matches DORA's own 2025 finding that delivery instability rises with AI adoption even as perceived quality also rises.
explicitA maintainability model — treating maintenance as a mandatory tax on future engineering capacity, fed by measured AI speed improvements (from part 2) and assumed maintenance-cost/headcount multipliers — can show that a large raw speed improvement nets out to merely breaking even once offsetting costs are modeled: e.g., +50% speed, -25% headcount from token costs, and doubled maintenance costs yields only breakeven.conf: high
explicitIn Shore's worked example, AI-driven spending efficiency (cost per value-add day) initially undercuts the pre-AI baseline by roughly a third but crosses back above it within about a year and ends up roughly double the baseline by month 120 (about $11,992 vs. $6,183); total production shows a temporary ~11,000-value-add-day lead around months 48-60 but ends about 12,000 value-add days behind baseline by month 120.conf: high
extendsAre You Getting Your Money's Worth from AI? · p16 — Part 1 previewed a 'Spending Efficiency' graph and the claim that AI-driven productivity can spike then fall below baseline, without defining the underlying model; part 3 supplies and fully works the maintainability-tax model that produces exactly that spike-then-cross-under shape, with concrete numbers.
explicitLines of code is a poor productivity measure but a reasonable maintenance-cost proxy, because more code generally produces more maintenance burden; this can be modeled via gross code churn categorized as value-add versus maintenance.conf: high
extendsCode maintainability plummets in the AI coding era · p6 — GitClear's telemetry measures rising code churn/duplication under AI empirically across 623M commits; Shore's excess-lines-of-code model turns that same signal (gross code churn) into a forward-looking maintenance-cost projection tool.
explicitA maintenance-churn model built from real 19-year commit history of the open-source tmux project (independently provided and LLM-classified by Christophe Stjernlöf) validated Shore's a priori guessed ratios (roughly 1:1 maintenance-to-value-add churn in the first year, 0.1:1 in subsequent years) as a close match to the actual data; by year 19, maintenance churn was about double value-add churn.conf: high
explicitLock-in should be modeled as a variant of the maintainability model: if token costs rise, an organization may need to cut headcount, reduce AI use, or switch to a less capable model, with a separate estimate for the cost of humans maintaining AI-written code under those scenarios.conf: high
extendsAre You Getting Your Money's Worth from AI? · p9 — Part 1 argued future AI pricing direction is genuinely uncertain and ROI models should test optimistic/pessimistic scenarios; part 3's lock-in model operationalizes that recommendation specifically for token-price increases forcing headcount cuts or reduced AI use.
extendsYou Outsourced the AI—but You Still Own the Risk · p5 — Operationalizes that essay's Exposure 3 (Jenga-like AI vendor lock-in, diagnosed qualitatively) into a concrete economic model projecting the headcount/capability tradeoff a token-price increase would force.
explicitBurnout cannot be modeled with real accuracy, but can be tracked: fold a mental-exhaustion self-report scale into the same speed-comparison study used for part 2, run periodic one-question mental-energy surveys, track turnover against historical baselines, and watch for AI complaints in exit interviews.conf: high
extendsAre You Getting Your Money's Worth from AI? · p17 — Part 1 flagged burnout risk via a single anonymized anecdote; part 3 supplies a concrete (if explicitly unmodelable) measurement protocol — a mental-exhaustion scale folded into the speed study, periodic surveys, turnover tracking, and exit-interview attention.
supportsAI productivity is burning out your best engineers · p7 — Both insist speed and sustainability are distinct, and that oversight/burnout effects must be measured with the same rigor applied to velocity targets, not left invisible.
explicitValue-add percentage is highly gameable: once people believe their work categorization is being judged, they face strong incentive to reclassify muda as value-add or defer maintenance — even without fraud, a leader with prioritization authority can defer maintenance for a couple of years to look good before moving to a bigger role elsewhere, leaving a successor to inherit the deficit.conf: high
explicitShore has no complete fix for value-add percentage's gameability beyond the general Goodhart's Law discipline (from part 1) of using metrics to evaluate an approach, never a person — an explicit admission that the essay's own most-trusted metric is 'more gameable than most.'conf: high
extendsAre You Getting Your Money's Worth from AI? · p14 — Reapplies part 1's Goodhart's Law discipline (measure approaches, not people) to a new metric, value-add percentage, while explicitly admitting its limits in this new case.
explicitDelivering more software isn't sufficient on its own; development speed and unintended consequences together describe how much software gets delivered, but whether it's the right software is a separate question deferred to part 4 (business outcomes).conf: high
implicitBuilding explicit scenario models is worth the effort even though every key input (maintenance-cost multipliers, headcount reductions, token-price direction) must be estimated or guessed, because a structured what-if model is treated as more decision-useful than either no model or the noisy raw indicators alone.conf: medium
implicitA single practitioner's personal, single-organization experience (Shore doubling his own value-add percentage as VP of Engineering over three years) is treated as adequate support for recommending value-add percentage as the most trustworthy cost-of-change indicator, without independent replication or cross-organizational benchmarking — a smaller-scale instance of the anecdote reliance part 1 explicitly warned readers against.conf: medium
implicitThe tmux validation is treated as evidence the excess-lines-of-code model generalizes, but it only validates the shape/ratio of maintenance-to-value-add churn in one long-lived, pre-AI-era, human-authored open-source project classified after the fact by a third party's LLM — not that the same ratios, or the model's translation of churn into dollar maintenance costs, hold for AI-accelerated, AI-authored enterprise codebases, which is the model's actual intended use case.conf: medium

Arguments

Organizations should track cost of change primarily via value-add percentage, supplemented by DORA metrics and token usage, rather than seeking precise measurement. strength: medium

Explicit premises:

Cost of change is a useful unifying construct for AI's unintended consequences.
Value-add percentage operates at the level that matters (where time goes) and doesn't require heavyweight tracking infrastructure.
DORA metrics are a standard framework most organizations likely already track.
Token usage is a literal cost signal, though confounded and noisy.

Implicit premises:

A leader can reliably and consistently categorize work as muda vs. value-add across many teams over time without much oversight infrastructure — an assumption the essay later undercuts in its own gameability discussion.
The value-add/muda categorization judgment is stable enough over time to support trend comparison, despite fuzzy edge cases (e.g., whether preventing a demanding customer's complaint counts as value-add or muda depends on framing not specified in the essay).

Inferences:

If value-add percentage is as gameable as the essay itself later admits, any organization lacking Shore's personal ability to police classification integrity should expect the indicator's reliability to degrade precisely under the professional-incentive conditions where measuring AI's impact matters most (i.e., when leaders are under pressure to show AI ROI) — a tension introduced in one section and never resolved in the section that recommends the metric.

Because unintended consequences are gradual, noisy, and can't be measured directly (unlike delivery speed), scenario-based economic models are necessary to see the full long-run shape of AI's cost-of-change impact. strength: medium

Explicit premises:

Value-add percentage takes 3-6 months to show trends and a year to confirm causation.
Token usage and DORA metrics are confounded by many factors beyond cost of change.
The maintainability model, worked through, shows a 50%-speed-increase scenario netting to breakeven once headcount cuts and doubled maintenance costs are modeled.
The excess-lines-of-code model, validated against tmux's real 19-year commit history, shows maintenance costs can end up double value-add costs over time.

Implicit premises:

A model built on assumed multipliers (e.g., 'estimate how AI will change maintenance costs') is more decision-useful than not modeling at all, even when its key inputs are estimated guesses rather than measurements.
The tmux validation — one 19-year-old, human-authored, pre-AI-era open-source terminal-multiplexer project, classified after the fact by a third party's LLM — is representative enough of typical enterprise codebases (including AI-accelerated ones) to justify treating its guessed ratios as a credible starting point elsewhere.

Inferences:

Because the model's output is highly sensitive to assumed inputs (doubling the maintenance-cost multiplier flips the worked example from a clear win to a breakeven), the real analytical burden shifts from choosing a metric to defending the model's input assumptions — a burden the essay never gives readers a rigorous method for discharging, beyond 'wisdom of the crowd' and a single historical validation case.

Value-add percentage should still be used — 'with caution' — despite being unusually easy to game. strength: low

Explicit premises:

Gaming value-add percentage is real, and can be entirely non-fraudulent (a leader deferring maintenance to look good ahead of a career move).
There is no metric immune to gaming.
Part 1's general remedy — measure approaches, not people — is offered as the mitigation.

Implicit premises:

The 'measure approaches, not people' discipline, designed in part 1 primarily as a remedy for individual/team activity metrics like PR count (where a worker games a target imposed on them), transfers cleanly to a leadership-level resource-allocation metric like value-add percentage, even though the incentive to distort it here comes from an individual leader's own career interest, not from a worker responding to an externally imposed target.

Inferences:

If value-add percentage's gaming vector is the leader's own career incentive rather than a worker's response to a target set by someone else, then 'measure the approach, not the person' does not neutralize the risk the way it does for PR count — because the person choosing what to measure and report, and the person with incentive to distort it, are the same individual. The essay does not address this structural difference, leaving its headline remedy weaker here than in the case it was originally designed for.

Narratives

Shore's personal VP-of-Engineering anecdote: he made decreasing muda a priority and doubled value-add percentage over three years, freeing people for substantial new initiatives without increasing headcount.
→ Supports the claim that value-add percentage is a genuinely useful, trustworthy lagging indicator despite its noise — the essay's sole direct evidence for recommending it.
The open-source tmux project's real 19-year commit history, classified as value-add or maintenance by an LLM at the direction of external contributor Christophe Stjernlöf, closely matched Shore's a priori guessed churn ratios.
→ Validates (partially) the excess-lines-of-code maintenance model — supplies a named, checkable, non-anecdotal data source, unlike most of the essay's other evidence.
A worked numeric scenario: AI increases speed by 50%, but token costs force a 25% headcount cut and maintenance costs are estimated to double, resulting in only breaking even despite the large raw performance improvement.
→ Illustrates the maintainability model's core warning that unintended consequences can fully erase an apparently large productivity win.
The hypothetical 'demanding and crotchety customer' who asks 'why didn't you just do it right the first time?' and 'why don't you just ship it already?'
→ Defines and makes concrete what counts as muda (bug fixes, releases) versus value-add work, even when that work is necessary or valuable.
The unnamed hypothetical leader who defers maintenance for a couple of years to inflate their value-add percentage, lines up a bigger title at a different company, and leaves a successor holding the bag.
→ Illustrates value-add percentage's gaming risk and its delayed-consequence structure — the damage surfaces only after the person responsible has already left.
The 1-10 mental-exhaustion scale (Energized to Nothing Left), illustrated with an AI-generated image (explicitly disclaimed as made with GPT-5.6 Sol).
→ Operationalizes an otherwise 'can't be modeled with any accuracy' construct (burnout) into something concretely trackable via self-report.

Questions addressed

✅ Does maintainability (and other unintended consequences) actually matter, or is it a 'namby-pamby' concern?
Yes — because of real, concrete downstream economic costs: slower future work, attrition/hiring costs, and forced big-bill-or-cutback lock-in tradeoffs.
⭕ If AI is doing the maintenance, does human comprehension of AI-generated code still matter?
Explicitly left open — Shore reports disagreement among practitioners and states there isn't much actual evidence either way.
✅ How do you measure something as gradual and fuzzy as unintended consequences, when a controlled trial (like part 2's speed study) isn't available?
Via a cluster of noisy lagging indicators (chiefly value-add percentage, plus DORA metrics and token usage) that inform, not drive, judgment, supplemented by explicit scenario models.
✅ Which cost-of-change indicator is most trustworthy?
Value-add percentage, despite its noise, its 3-6 month lag, and its high gameability.
✅ How do you model AI's effect on long-run maintenance cost, lock-in exposure, and burnout when none can be precisely measured?
Three explicit models — maintainability-as-tax, excess-lines-of-code (validated against tmux data), and a lock-in variant of the maintainability model. Burnout is explicitly stated as not modelable with accuracy, only trackable.
⭕ Does the essay's own recommended fix for metric-gaming ('measure approaches, not people,' from part 1) actually neutralize value-add percentage's gameability, given that here the gaming vector is a leader's own career incentive rather than a worker responding to an externally imposed target?

Gaps

incompleteThe value-add-percentage recommendation rests on a single practitioner's (Shore's own) three-year, single-organization experience, with no independent replication, cross-organizational benchmark, or comparison against alternative cost-of-change indicators — the same reliance on unverified personal anecdote that part 1 explicitly warned readers against trusting elsewhere in this very series.
Read next: Watch whether part 4 or the CFO epilogue supplies cross-organizational data on value-add percentage or a comparable indicator; compare against GitClear's larger-sample maintainability telemetry in extract-code-maintainability-plummets-ai-coding-era.
illogicalThe essay reapplies part 1's Goodhart's Law remedy ('measure approaches, not people') to value-add percentage's gameability, but the gaming vector here is structurally different from the PR-count case it was designed for: a leader with prioritization authority games their own reported categorization for personal career benefit, not a worker responding to an externally imposed target. The essay never examines whether the remedy actually neutralizes this self-referential case — 'I don't have an easy answer' is an honest admission, not a defended one.
Read next: Compare against extract-8-engineering-metrics-ai-broke#g3, which raises the structurally similar problem that the piece's own recommended 'safe' outcome metrics are themselves targetable.
incompleteThe maintainability and excess-lines-of-code models' conclusions are highly sensitive to a handful of assumed multipliers (maintenance-cost doubling, 25% headcount reduction, token-price direction) that the essay says to estimate via 'wisdom of the crowd' or historical analogy, but gives no method for calibrating or stress-testing these assumptions beyond the single tmux validation case — leaving the models' epistemic status closer to structured what-if scenarios than validated forecasts.
Read next: Read the linked spreadsheet and the full 'You Need AI That Reduces Your Maintenance Costs' essay for a calibration method; compare against extract-tbm-437-tokens-hours-points-proxies' warning that a metric's apparent precision can be illusory.
incompleteThe tmux validation supports only the shape of one long-lived, human-authored, pre-AI-era open-source project's maintenance-to-value-add churn ratio, classified after the fact by a third party's LLM — not that the same ratios, or the model's translation of churn into dollar maintenance costs, hold for AI-accelerated, AI-authored enterprise codebases, which is the actual case the model is meant to project.
Read next: Read extract-code-maintainability-plummets-ai-coding-era's GitClear telemetry (which directly compares AI-era vs. pre-AI-era commits) alongside this model to test whether AI-authored code's churn ratios diverge from tmux's human-authored baseline.
uninformedThe essay states there is 'widespread agreement' that AI-heavy code is harder for humans to understand but 'disagreement about how much that matters' and 'not a lot of actual evidence' either way — an open, acknowledged evidentiary gap that the essay doesn't attempt to resolve or even point toward evidence-gathering for, despite the series' stated commitment to rigor over anecdote.
Read next: Look for empirical studies connecting human comprehension of AI-generated code to actual defect or incident rates, e.g. work on cognitive/comprehension debt.

Closes gaps elsewhere

Are You Getting Your Money's Worth from AI? · g1 — The excess-lines-of-code model is partially validated against real, named, checkable commit data from the open-source tmux project (provided by Christophe Stjernlöf, LLM-classified as value-add/maintenance across a 19-year history) — giving one component of the series' promised methodology a checkable data source rather than an unsourced anonymized anecdote. This is a partial closure: the model's other key inputs (maintenance-cost multipliers, headcount-reduction assumptions) remain estimated via 'wisdom of the crowd,' and the value-add-percentage recommendation still rests on Shore's own unreplicated anecdote (see this extraction's own gap 1).
Are You Getting Your Money's Worth from AI? · g4 — Part 3 supplies constructive methodology for the unintended-consequences dimension specifically named in g4's reading direction: the cost-of-change construct, value-add percentage/DORA/token-usage as indicators, and three explicit scenario models (maintainability-as-tax, excess-lines-of-code, lock-in). This is a partial, not full, closure of g4 — the models explicitly consume part 2's speed-trial output as an input (not yet extracted in this corpus), and part 4 (business outcomes) and the CFO epilogue are still needed to complete the series' promised picture.

Measuring AI's Unintended Consequences

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents