Goodhart's Law

The principle that once a measure becomes a target, it ceases to be a good measure — the mechanism by which AI-era activity metrics (deploy frequency, PR volume, velocity) get gamed or inflated without the underlying quality they were meant to proxy.

Also known as: Goodhart's Law, when a measure becomes a target, metric gaming

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (10)

• Measuring AI's Unintended Consequences James Shore · 2026-09-22 Reader 🖥️
  Because AI's unintended consequences — declining maintainability, burnout, and vendor/token lock-in — carry real economic costs but, unlike delivery speed, cannot be measured by a controlled trial and instead surface only gradually and noisily, organizations should track a cluster of imperfect lagging indicators (chiefly value-add percentage, backed by DORA metrics and token usage) and build explicit long-horizon scenario models (a maintenance-cost 'tax,' an excess-code-churn model, and a lock-in variant of both) to inform — never replace — judgment, even though the most trusted of these indicators is itself unusually easy to game.
• Measuring AI's Impact on Delivery Speed James Shore · 2026-09-15 Reader 🖥️
  Because no unit of software output is standardized and raw on-time-delivery rates are corrupted by chronic estimate over-optimism, an organization can only find out whether AI is really making delivery faster by running a randomized, within-subject, repeated-measures trial that compares each person's or team's own 'with AI' and 'without AI' actual-to-estimate ratios, collected before assignment is known, under an otherwise identical process.
• John Cutler's Post John Cutler · 2026-09-12 source ↗ Reader 🖥️
  Shifting from outputs to outcomes fails not because the idea is wrong but because teams adopt the language of outcomes while keeping the psychology of outputs — chasing certainty, easy measurement, and reputational credit instead of committing to the qualitative judgment, tolerance for ambiguity, and honest attribution that outcome-orientation actually demands.
• Are You Getting Your Money's Worth from AI? James Shore · 2026-09-08 Reader 🖥️
  Because AI's costs and benefits in software development are real, substantial, and easily misjudged by anecdote, self-assessment, or gut feel, organizations facing consequential tradeoffs between speed, cost, and people's jobs must rigorously measure both sides — using durable, outcome-anchored, approach-level metrics rather than gameable activity proxies like lines of code, pull-request count, or individual performance targets.
• When AI can do more than we can check Christian Catalini · 2026-09-08 Reader 🖥️
  As AI collapses the cost of producing plausible-looking output while the cost of verifying that output stays tied to human time, expertise, and slow real-world feedback, verification — not generation or intelligence — becomes the scarce, binding resource that reshapes careers, companies, investment, and policy, and whoever builds the durable capacity to verify, insure, and correct AI captures the value while everyone else risks sliding into a 'Hollow Economy.'
• 3 Warning Signs Your Supplier Is in Distress Mehrdokht Pournader, Andrew Kach, Stephanie Eckerd, Niharika Garud · 2026-07-30 Reader 🖥️
  Multinational buyers can detect supplier distress before formal failure by scoring six early-warning dimensions (innovation, investment, capability, service resilience, commercial participation, network position) that conventional lagging metrics (audits, orders filled, service levels) miss, and should respond with escalating, stage-matched interventions as the supplier moves from range thinning to descaling to withdrawal.
• From dashboards to decisions Brian Houck (Engineering Enablement / DX newsletter) · 2026-07-08 Reader 🖥️
  A developer-productivity measurement system creates value only when it changes what leaders decide to do next, not when it simply reports whether a number went up or down — which requires measuring outcomes (Speed, Ease, Quality) across multiple dimensions simultaneously with Thriving as a non-negotiable guardrail, because any single activity metric, examined alone, will tell a confident and sometimes completely misleading story.
• The 8 software engineering metrics AI broke Michael Hill · 2026-06-16 source ↗ Reader 🖥️
  Software engineering metrics were built on the assumption that human effort and output move in rough proportion, and AI has severed that link by making activity nearly free — so activity/volume proxies (deployment frequency, cycle time, PR and commit volume, lines of code, satisfaction, efficiency/flow, DevEx/DXI) now measure tool adoption rather than engineering quality, while only outcome-based measures (time to recover, escaped defect rate, delivered value) remain trustworthy, because they measure what happened rather than how much happened.
• Recommendation algorithms might be making your entertainment boring, new research suggests Eric W. Dolan · 2026-06-02 source ↗ Reader 🖥️
  Because algorithms that optimize for short-term engagement mistake familiarity-driven engagement for quality and evaluate over a horizon far shorter than the decade-plus timescale on which aesthetic taste actually evolves, they systematically under-explore and trap users in an eventually-boring set of favorites — and a mathematical model shows that deliberately injecting moderate noise (forcing exploration) into such a system improves users' long-term satisfaction.
• DORA Community Discussion - DX Core 4 DORA · 2025-03-13 source ↗ Reader 🖥️
  A software organization's performance multiplier lives at the organizational/cultural level, not in individual '10x engineers', and is best captured not by any single metric framework but by a blended quantitative-plus-qualitative system — DORA, then SPACE, now distilled into DX Core 4's four dimensions (speed, effectiveness, quality, impact) anchored by a composite Developer Experience Index — that treats flow state and cognitive load as the proximate mechanism linking developer experience to delivery outcomes.

Propositions touching this concept (7)

By publication month, oldest first (extraction month when no publication date is known).

2025-03

• explicitcorroboratedRelying on any single isolated metric as an organizational target invites Goodhart's Law — the metric gets gamed and stops being informative — so a spectrum of metrics across several dimensions is required instead.
  DORA Community Discussion - DX Core 4 · p6

2026-04

• implicitHeightened management pressure for throughput is itself a confounding variable in longitudinal analyses -- some throughput gains may be Goodhart's Law in action.
  Measuring AI impact, assessing readiness, and new data trends · p9
• explicitcorroboratedTop-down AI utilization mandates decrease psychological safety and lead to metric gaming rather than genuine productivity gains
  Presentation: Leadership in AI-Assisted Engineering · p3

2026-06

• explicitdevelopedGoodhart's Law has moved from an occasional problem requiring deliberate manipulation to an inescapable one, because AI makes metric-gaming a side effect of normal tool use rather than something requiring extra effort.
  The 8 software engineering metrics AI broke · p7
• implicitCost per change went down because of platform effect, not because the change-mix got easier — Goodhart's law is bracketed.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering · p14

2026-09

• explicitdevelopedMetrics of AI's impact should be used to evaluate a particular approach to using AI, then retired once learned from — never used as an ongoing measure of individual or team performance — per Goodhart's Law.
  Are You Getting Your Money's Worth from AI? · p14
• explicitShore has no complete fix for value-add percentage's gameability beyond the general Goodhart's Law discipline (from part 1) of using metrics to evaluate an approach, never a person — an explicit admission that the essay's own most-trusted metric is 'more gameable than most.'
  Measuring AI's Unintended Consequences · p16

Open gaps (20)

• illogicalThe article's own framing is internally inconsistent about scope: the intro promises 'ten metrics that are breaking down,' the SPACE section is introduced as covering 'a couple of metrics' but actually covers three, and the title promises '8' broken metrics — which does match the count of metrics explicitly presented as broken (3 DORA + 3 SPACE + 2 standalone), but only by excluding the 3 counter-example metrics from the 'ten' the intro promised. The total metrics actually discussed is 11, not 10, and 'a couple' undercounts by one. This suggests the taxonomy was fitted to a round headline number rather than derived from a principled classification.
  The 8 software engineering metrics AI broke — read next: Compare against the live leaddev.com page for possible post-publication editing (the mismatch is exactly the kind an editor introduces when trimming or adding an example late); no corpus cross-reference needed.
• incompleteThe proposed remedy assumes evals can be built comprehensively enough to substitute for everything natural language previously carried — intent, edge cases, tone, nuanced judgment calls — but the piece never addresses how to author a sufficiently complete eval suite, nor whether some behaviors resist reduction to scorable metrics at all, nor the risk that the eval itself becomes the thing being gamed once it becomes the target.
  The Problem Is Prompt Debt — read next: Read about eval-suite design methodology and Goodhart's-law-style failure modes for LLM evals — does 'specify with measurements' just relocate prompt debt into 'eval debt'?
• incompleteAll three case studies are drawn from the author's own EngThrive program (built largely at Microsoft); the newsletter doesn't report an independent replication at an organization with different scale, culture, or tooling maturity, so it is unclear how much of the framework's apparent success depends on resources (a dedicated measurement team, longitudinal survey infrastructure) that most engineering orgs lack.
  From dashboards to decisions — read next: Look for a non-Microsoft, non-DX-client case study applying the same Speed/Ease/Quality/Thriving framework, especially at smaller scale.
• illogicalThe Time-to-First-PR argument generalizes from one successful case ('a well-designed metric is one where gaming equals improving') to a design principle without acknowledging that this property is a post-hoc observation about one metric, not a general test an author could apply prospectively when designing a new metric — the piece offers no method for predicting in advance whether a candidate metric will have this property before trying it.
  From dashboards to decisions — read next: Compare against Goodhart's-Law literature in the corpus (DORA/DX community material) for any proposed ex-ante test of 'ungameable-by-construction' metrics, rather than only ex-post classification.
• incompleteNo discussion of Goodhart's law — that turning a measure into a target erodes its value as a measure. With 10,000+ vendor-friendly tasks and labs explicitly racing to improve scores, this is the central risk the piece sidesteps.
  Kaggle is making AI benchmark creation effortless — read next: Manheim & Garrabrant on Goodhart taxonomies; recent evaluation-contamination work (Sainz et al., Zhou et al.); Gary Marcus on benchmark-overfitting in LLM math performance.
• incompleteNo discussion of failure modes for the CResO function itself: accountability laundering (other executives off-load problems to CResO), resilience theater (dashboards proliferate, actual capability does not improve), single-point-of-failure (a strong CResO leaves and capability collapses), incentive misalignment (CResO scored on time-to-recover incentivizes minimizing reported incidents).
  The Case for Hiring a Chief Resilience Officer — read next: Goodhart's Law literature; Cutler on metrics that displace work; security industry's experience with CISO accountability since SOX/Sarbanes-Oxley.
• illogicalThe six sub-scores are explicitly qualitative managerial judgment calls ('no meaningful sign,' 'an early or isolated sign,' 'a recurring or material sign,' 'a severe sign' mapped to 0-3), yet they are summed into a single integer and compared against sharp numeric thresholds (4-7 vs. 8-13 vs. above 13) that determine categorically different managerial responses. The article never establishes that a 'severe' signal in one dimension is commensurable with a 'severe' signal in another, nor why simple addition — rather than a weighted sum, a maximum, or a pattern rule — is the right way to combine them. The authors' own caveat that 'a severe signal in commercial participation or network position should trigger immediate investigation even when the aggregate score remains lower' implicitly concedes that the additive total can itself mislead (a score of 4 built from one severe signal is treated as more urgent in practice than a score of 4 spread thinly across dimensions), yet no revised scoring rule is offered to resolve that inconsistency. The framework's apparent numerical precision (specific integer cutoffs) masks a fundamentally qualitative, unweighted, unvalidated input process.
  3 Warning Signs Your Supplier Is in Distress — read next: Read measurement-theory or operations-research literature on composite risk indices (e.g., credit-risk scorecards, safety-culture indices, early-warning systems in banking supervision) for how they handle ordinal-to-cardinal aggregation and dimension weighting, and look for any published validation of this specific instrument — e.g., whether a score of 9 actually predicts subsequent supplier failure or exit at a higher rate than a score of 6.
• incompleteThe article repeatedly cites 'our ongoing research' and 'recent studies suggest' but never reports sample size, response or attrition rates, effect sizes, or predictive validity for either the three-stage taxonomy or the specific score thresholds. Without this, it is impossible to tell whether the taxonomy tracks real, measurable outcome differences among suppliers or is a plausible narrative overlay imposed on qualitative interview data from Vietnam and Australia.
  3 Warning Signs Your Supplier Is in Distress — read next: Look for the underlying academic paper(s) by Pournader, Kach, Eckerd, and Garud on the Vietnam/Australia supplier study for methodology, sample size, and any reported validation statistics.
• incompleteThe framework positions the buyer as the neutral diagnostician of supplier distress, while the article's own evidence names buyer behavior (cost-shifting without burden-sharing, duplicate and overlapping audits, uncoordinated compliance demands from different corporate customers) as a direct driver of the very distress the buyer is being taught to detect. The piece never proposes that buyers systematically score or audit their own contribution to a given supplier's distress (e.g., 'are we crowding out this supplier's investment capacity relative to other customers?') the way it asks them to score the supplier — the diagnostic tool has no mirror pointed at the diagnostician.
  3 Warning Signs Your Supplier Is in Distress — read next: Look for principal-agent or platform-power literature on how dominant buyers/platforms externalize compliance and cost burdens onto weaker counterparties, and whether any published supplier-health framework includes a buyer self-assessment component alongside the supplier-facing one.
• uninformedIllustrative 'recent reporting' references — U.S. auto-parts and battery manufacturing pressures, an unnamed auto-parts supplier's bankruptcy and wind-down of key units, unnamed fashion companies leaving the market — are invoked as corroborating real-world endpoints but no company, date, or source is named or cited, making the claims unverifiable and potentially selected after the fact to fit the three-stage narrative rather than independent confirmation that the pattern generalizes.
  3 Warning Signs Your Supplier Is in Distress — read next: Identify the specific companies/events being referenced and check whether the range-thinning-to-descaling-to-withdrawal pattern actually preceded those specific failures as a predictive signal, or whether it is being read into them retrospectively.
• incompleteThe 'objective systems hide bias' claim (takeaway 9) is asserted from conviction with no supporting data, case study, or citation of a specific performance-review system shown to smuggle in bias this way — and never asks whether a transparent, auditable, semi-automated evaluation could reduce rather than hide bias, treating any claim of objectivity as inherently suspect.
  How Do You Know That? — read next: Algorithmic-fairness/bias literature, or the corpus's own more developed treatment of the same claim in extract-tbm-428-yes-robot-yes-boss (epistemic justice and AI 'objectivity').
• illogicalThe post implicitly treats 'outcome-based measures' as harder to game than output measures (the standard critique it's reacting against), yet its own #4/#9/#10 show outcomes are gameable too — just via narrative/timing manipulation (retrospective rationalization, quarter-end clustering, evangelism-over-efficacy) rather than volume inflation. The post doesn't reconcile this tension: it never explicitly revises the premise that switching to outcomes solves the gaming problem, even though its own examples show it merely relocates the gaming.
  John Cutler's Post — read next: Read alongside Goodhart's Law material in this corpus (extract-8-engineering-metrics-ai-broke.json, extract-shore-ai-money-worth-part1.json) which makes explicit what Cutler leaves implicit: any measure that becomes a target, outcome or output, is subject to gaming — the fix is retiring metrics after they've been learned from, not choosing 'harder to fake' metric categories.
• incompleteNo mention of psychological safety or incentive/reward-system design, even though several failure modes (#4 credit-seeking, #9 cherry-picking around promo time, #11 fabricating causal stories) are natural responses to a reward system that punishes honest reporting of ambiguity, setbacks, or unattributable work. The post diagnoses the symptom (behavior) without naming the likely structural cause (what gets rewarded).
  John Cutler's Post
• incompleteThe article never states the model's functional forms, parameter values, or sensitivity analysis (how the appreciation curve is shaped, what noise levels were tested, how many Monte Carlo trials produced the reported effect sizes). Without these, a reader can't judge how robust the 'moderate noise helps' conclusion is to the specific modeling choices — a purely theoretical model's conclusions can be highly sensitive to functional-form assumptions.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Read the underlying paper directly — Samsun Knight, "Engagement-based curation and the evolution of taste," Journal of Cultural Economics (2026), https://doi.org/10.1007/s10824-026-09591-3 — for the actual model specification and robustness checks.
• incompleteThe article treats platforms as having essentially one dial (prediction accuracy vs. noise), but real recommendation systems already blend multiple discovery mechanisms — curated new-release rows, human-programmed playlists, explicit 'diversity' injections, cold-start exploration bonuses. It never asks whether real-world systems already implement something functionally like the paper's proposed fix, which would make the normative recommendation partially moot rather than novel.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Platform-side accounts of recommender-system design (e.g., published descriptions of Spotify's Discover Weekly / exploration mechanisms, or general recommender-systems literature on the exploration-exploitation trade-off) to check how much of this gap is already closed operationally.
• uninformedThe study is a purely theoretical, simulated model with no human-subject or real-usage data — the article flags this directly, but doesn't examine whether the inverted-U 'consumption capital' curve (moderate exposure raises liking, excess exposure causes satiation) is itself empirically well-established for genre-level entertainment tastes specifically, as opposed to being carried over from general cultural-economics theory (Stigler & Becker) or adjacent phenomena like the mere-exposure effect and hedonic adaptation.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Empirical psychology/economics literature on hedonic adaptation and the mere-exposure effect (Zajonc) to check whether an inverted-U specifically holds at the genre/style level for music, film, and fiction — not just for individual songs or images in lab settings.
• illogicalThe paper's proposed fix is framed as making the algorithm 'less accurate' or 'noisier,' but the actual causal lever the article describes is forced exploration regardless of predicted engagement — a distinct design choice from genuinely degrading prediction quality. Conflating 'noisy predictions' with 'committed exploration' muddies the practical implication: a platform could keep maximally accurate predictions everywhere and still reserve a fixed fraction of recommendation slots for exploration, capturing the same long-term benefit without the framing cost of 'deliberately recommend worse content.' This is exactly the systems-thinking move the article skips: it treats accuracy and exploration as one dial when the model's own results imply they're two.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Multi-armed bandit / exploration-exploitation literature (e.g., Sutton & Barto, *Reinforcement Learning*) which cleanly separates prediction accuracy from exploration policy — useful for reframing the paper's own proposed fix as a policy-layer change rather than an accuracy trade-off.
• incompleteThis design is a per-person, per-team, repeated comparison of actual to estimated time — functionally much closer to 'measuring people' than part 1's own prescription ('Measure Approaches, Not People,' grounded in Goodhart's Law) admits. The essay never states the safeguard that would preserve part 1's discipline (e.g., that results must be reported and acted on only at the approach level, never surfaced per individual), leaving open exactly the gaming risk part 1 warned about.
  Measuring AI's Impact on Delivery Speed — read next: Watch whether part 3 or part 4 of this series addresses how the trial's per-person data should be governed, reported, or anonymized to avoid becoming an individual performance target.
• incompleteShore's remedy for Goodhart's Law ('measure the approach, learn from it, then stop measuring') is not reconciled with his own point that AI's pricing and capability landscape is actively shifting (subsidies ending, prices potentially dropping, models improving). If an approach's cost/benefit profile can change after the fact, 'stop measuring' risks leaving an organization blind to exactly the kind of change the essay warns about earlier in the same piece.
  Are You Getting Your Money's Worth from AI? — read next: Watch for whether later parts of the series (or Shore's broader writing) specify a cadence or trigger condition for re-measuring an approach, rather than a one-time 'learn and stop.'
• illogicalThe essay reapplies part 1's Goodhart's Law remedy ('measure approaches, not people') to value-add percentage's gameability, but the gaming vector here is structurally different from the PR-count case it was designed for: a leader with prioritization authority games their own reported categorization for personal career benefit, not a worker responding to an externally imposed target. The essay never examines whether the remedy actually neutralizes this self-referential case — 'I don't have an easy answer' is an honest admission, not a defended one.
  Measuring AI's Unintended Consequences — read next: Compare against extract-8-engineering-metrics-ai-broke#g3, which raises the structurally similar problem that the piece's own recommended 'safe' outcome metrics are themselves targetable.