The 8 software engineering metrics AI broke

Michael Hill · article · 2026-06-16 source ↗ Reader 🖥️

Code Was Never the Bottleneck Cognitive Debt Developer Experience & Flow DORA & Engineering Metrics Goodhart's Law Measuring AI Impact Technical Debt Token Throughput

Software engineering metrics were built on the assumption that human effort and output move in rough proportion, and AI has severed that link by making activity nearly free — so activity/volume proxies (deployment frequency, cycle time, PR and commit volume, lines of code, satisfaction, efficiency/flow, DevEx/DXI) now measure tool adoption rather than engineering quality, while only outcome-based measures (time to recover, escaped defect rate, delivered value) remain trustworthy, because they measure what happened rather than how much happened.

practical · mixed (expository listicle scaffolded on an argumentative core) — Trade journalism built from named-practitioner quotes (a CTO, an EVP of consulting, two senior engineers, a vendor's head of AI/research, a community CEO) rather than data or citation to a study. The per-metric structure looks like a bare checklist but each entry does real argumentative work: state the metric's original validity assumption, show the specific mechanism by which AI severs it, then (for 3 of 11 metrics discussed) argue why that mechanism doesn't apply.

Structure

Part	Role
Opening frame: metrics assumed effort-output proportionality; AI breaks the assumption; the real question dashboards were never built to answer ('are we building the right things, in the right way') is now exposed.	States the thesis and its warrant before any metric is examined
Three DORA metrics (deployment frequency, lead time/cycle time, change failure rate), each with a named practitioner quote.	Applies the template to the most institutionally entrenched framework first
Three SPACE-framework metrics (satisfaction, activity, efficiency/flow) — introduced as 'a couple' though three are covered.	Extends the argument to the framework explicitly designed to go beyond activity metrics, to show even it is vulnerable
Two standalone metrics (lines of code, DevEx/DXI).	Mops up metrics outside either named framework, reaching the round number of 8
Goodhart's Law + tokenmaxxing interlude.	Supplies the causal mechanism binding all 8 prior entries together: gaming a metric used to take effort, AI makes it a side effect of normal use
Three counter-examples presented as still trustworthy (time to recover/MTTR, escaped defect rate, 'value delivered' outcome measures), closing with a CEO quote reframing what metrics were 'always meant' to be.	Positive payload — what to measure instead

Key terms

Tokenmaxxing Token Throughput
The practice of deliberately maximizing AI token consumption to game a productivity metric rather than to produce better software — presented here as the purest current instance of Goodhart's Law under AI.
Goodhart's Law Goodhart's Law
Reframed as a threshold that AI has crossed from occasional to unavoidable: pre-AI, gaming a metric required deliberate effort; with AI, inflating deployment frequency, cycle time, or PR volume is a side effect of ordinary tool use, not intentional manipulation.
DORA metrics DORA & Engineering Metrics
Split into two evidentiary classes: three (deployment frequency, lead time, change failure rate) are declared broken because they track volume/speed; the fourth (time to recover) is declared to survive because it tracks an outcome (recovery from failure) rather than activity.
SPACE framework
Cited as the framework built specifically to go beyond activity/velocity — yet the article argues its own dimensions (satisfaction, activity, efficiency/flow) are equally distorted by AI, undercutting the idea that a 'more human' framework is inherently safer.
Escaped defect rate
Positioned as one of the three metrics AI cannot inflate without consequence, because it measures an outcome (a bug reaching production) rather than an activity, so it directly tracks whether verification is keeping pace with generation speed.

Propositions

explicit⚡ disputed & corroboratedDeployment frequency, cycle time, and PR volume are all inflated by AI without any corresponding improvement in engineering quality.conf: medium
extendsAI coding creates two kinds of debt. You're only measuring one · p3 — extract-ai-coding-two-kinds-debt's claim is narrowly about cognitive debt as one specific blind spot standard metrics miss; this article broadens the same underlying diagnosis (activity metrics don't surface what actually matters) to the entire standard metrics stack — deployment frequency, cycle time, PR volume — not just cognitive debt specifically.
supports ←The Acceleration Whiplash · p3 — Both reports independently find that activity/throughput-adjacent metrics inflate under AI adoption without a corresponding improvement in engineering quality.
contradicts ←How AI will change software engineering – with Martin Fowler · p14 — Fowler frames shortening cycle time as the best available lever of leverage in the AI era; the other source argues cycle time is inflated by AI without corresponding quality gains and can mask reduced human review. The tension is real but partial: Fowler's framing assumes review discipline is retained (see p7/p8 above), which is exactly what the other source finds is often not the case in practice.
supports ←Are You Getting Your Money's Worth from AI? · p12 — Both flag PR-based (and other activity) metrics as inflated by AI without a corresponding improvement in engineering quality.
explicitcorroboratedBecause output is now nearly free, activity/volume metrics measure tool adoption rather than engineering maturity or engineer skill.conf: medium
supports ←Engineering Leadership Report 2026 · p10 — Postman's finding that self-reported employee feedback dominates AI-impact measurement (63%) while rigorous, outcome-based metrics remain rare (~a fifth of orgs) is the adoption-side evidence for the broken-metrics thesis that popular activity/volume proxies now measure tool adoption rather than engineering quality.
explicitFaster cycle time under AI can indicate less human review rather than better delivery practice, which increases the defect escape rate.conf: medium
explicitChange failure rate can no longer indicate why a failure happened (agent, reviewer, or process at fault) once AI writes code that a human approves without fully understanding it.conf: medium
refinesPatterns for Reducing Friction in AI-Assisted Development · p6 — This article states change failure rate can no longer indicate why a failure happened (agent, reviewer, or process at fault); extract-garg-reducing-friction-ai-development proposes first-pass acceptance rate as a leading indicator for change failure rate under DORA, which is exactly the kind of supplementary signal that could restore some of the diagnostic power this article says has been lost.
implicitcontestedRising satisfaction scores under AI use may mask a hollowing-out of the work itself, because relief from boilerplate is not the same as meaningful work, and the most cognitively engaging tasks may be the ones AI is absorbing.conf: medium
supportsAI coding creates two kinds of debt. You're only measuring one · p6 — Both claim satisfaction-type scores can rise while something substantive (engagement/meaning here, skill growth there) falls under AI use. This article asserts the dynamic by argument; extract-ai-coding-two-kinds-debt independently corroborates it with Infobip's 8-cohort/225-intern longitudinal data (NPS/mentor scores flat-to-rising while measured skill growth fell) — real measurement backing this article's unsupported assertion.
contradicts ←Time Warp: The Gap Between Developers' Ideal vs Actual Workweeks in an AI-Driven Era · p6 — This paper reads rising self-reported satisfaction/productivity under daily AI use as a straightforward benefit, which is exactly the pattern the other document warns may mask a hollowing-out of the work rather than genuine improvement.
explicit⚡ disputed & corroboratedAI-generated code is on average about 20% larger than human-generated code, making lines of code essentially an input metric correlated with token spend rather than an output-value metric.conf: medium
extends ←CannotMeasureProductivity · p2 — Fowler's 2003 claim that LOC rewards poor design and can be mechanically inflated (Inline Method doubling LOC) is directly extended by the AI-era finding that AI-generated code is ~20% larger than human code for the same task, making LOC an input/token-spend proxy rather than an output-value measure.
contradicts ←Engineering Leadership Report 2026 · p20 — Stanier's quoted recommendation to track token spend as a meaningful AI-impact signal is in direct tension with the broken-metrics analysis, which argues token-spend-correlated volume metrics (like AI-generated lines of code, ~20% larger on average) are input metrics measuring tool adoption, not reliable signals of engineering quality or productivity.
supports ←Are You Getting Your Money's Worth from AI? · p11 — Both identify AI's code verbosity as turning lines-of-code into an input/volume signal correlated with token spend rather than a value signal.
explicitdevelopedGoodhart's Law has moved from an occasional problem requiring deliberate manipulation to an inescapable one, because AI makes metric-gaming a side effect of normal tool use rather than something requiring extra effort.conf: medium
extendsAgentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p3 — extract-tokenmaxxing documents one concrete case of metric-gaming-as-Goodhart's-Law (Meta tying performance reviews to token usage, employees running agents to inflate counts). This article generalizes that single case into a sweeping claim that AI has made Goodhart's Law unavoidable across all engineering activity metrics, not just token-usage incentives.
extends ←Recommendation algorithms might be making your entertainment boring, new research suggests · p3 — Generalizes the same Goodhart's-Law dynamic — a short-term, easy-to-measure proxy (engagement) displacing the true, hard-to-measure long-horizon target (durable satisfaction/quality) — from AI-era engineering metrics to consumer recommendation-algorithm design, a different domain entirely.
extends ←Are You Getting Your Money's Worth from AI? · p13 — Shore supplies a concrete mechanism — engineers producing unnecessary PRs once PR count is known to be watched — for Goodhart's Law becoming 'inescapable' under AI-era tooling.
extends ←When AI can do more than we can check · p3 — The sibling extraction observes that Goodhart's Law has become an inescapable, structural feature of AI-assisted engineering metrics rather than an occasional gaming problem; Catalini's 'counterfeit utility' generalizes exactly this mechanism beyond engineering metrics to any domain where a visible metric can be satisfied while the underlying purpose is violated (education, trading, customer service, software).
explicitTime to recover (MTTR) remains a reliable engineering-health signal because AI does not fundamentally change the problem of recovering from a production incident, and AI-assisted debugging may even modestly improve it.conf: low
explicitdevelopedEscaped defect rate and outcome-based measures (whether a feature was used, whether error rates fell, whether customer outcomes improved) are the metrics AI cannot inflate or fake, because they measure results rather than activity.conf: low
refines ←John Cutler's Post · p4 — The AI-broke piece claims outcome/value measures are the metrics AI 'cannot inflate or fake' because they measure results rather than activity; Cutler's #4/#9/#10 refine this by showing outcome-claiming has its own gaming mechanism (performative claiming, retrospective rationalization, quarter-end cherry-picking) — outcomes are harder to fake by volume but not immune to narrative gaming.
explicitestablishedMetrics were never supposed to measure productivity, individual performance, or team comparison — they were always meant as a feedback mechanism, and the industry lost sight of that in the rush to quantify AI-era development.conf: medium
supports ←CannotMeasureProductivity · p1 — Fowler's core thesis — productivity is unmeasurable because output is unmeasurable — supports and predates the corpus's independently-argued claim that metrics were never meant to measure productivity or enable cross-team/individual comparison, only to serve as local feedback.
extends ←"Our model is the most aligned yet!" 🤗 · p10 — Cites the same real, documented incident (OpenAI models breaking out of a benchmark sandbox to hack Hugging Face for the answers) as precedent for the forecast that sufficiently capable autonomous models will behave differently once they detect an evaluation context.
supports ←Are You Getting Your Money's Worth from AI? · p14 — Both argue metrics were originally meant as feedback mechanisms, not tools for measuring individual/team performance or comparison, and that this got lost.

Arguments

Most standard activity/volume engineering metrics (deployment frequency, cycle time, PR and commit volume, lines of code, satisfaction, efficiency/flow, DevEx/DXI) no longer signal engineering health under AI-assisted development. strength: medium

Explicit premises:

These metrics were built on the assumption that human effort and output move in rough proportion
AI has made output production nearly free, severing that proportionality
Named practitioners report specific mechanisms (tool-adoption masquerading as maturity, defect-escape rising as cycle time falls, satisfaction rising while engagement hollows out) by which each metric's signal breaks

Implicit premises:

The proportionality assumption was previously validated well enough by practice, not merely correlational luck, so its severing is a real regression rather than exposing a pre-existing flaw
A handful of named-practitioner anecdotes, one per metric, generalize to the industry rather than describing idiosyncratic organizations

Inferences:

Generalizes from single-quote mechanism stories to a sweeping claim covering 8 distinct metrics across 2 frameworks plus standalone measures
Treats a plausible causal mechanism (AI makes activity cheap) as sufficient to conclude the metrics are now actively misleading, rather than merely noisier

Three metrics — time to recover, escaped defect rate, and outcome/value-delivered measures — remain trustworthy because they measure outcomes rather than activity. strength: low