Proxy Validity (Construct Validity of Measurement Proxies)

The recurring failure where organizations adopt a metric (hours, story points, cycle time, tokens) because it is easy to observe rather than because it validly represents the value it claims to measure. Distinct from goodharts-law, which is about gaming a target after the fact — this is about the initial selection bias toward measurability over validity. The corrective is an explicit causal model of the system plus validated leading indicators.

Also known as: construct validity, measurement proxy, easy-to-measure bias, theory of the system

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (10)

• Erik van der Pluijm's Post Erik van der Pluijm · 2026-10-08 source ↗ Reader 🖥️
  When Claude Code collapses the time to build software, the 2-week sprint (and the ceremonies and metrics built around it) stops fitting, but this doesn't mean Scrum has failed — because Scrum was always about empiricism (tight feedback loops of inspect-and-adapt), not about sprint length, so the right response is to compress the cycle to 2 days, drop effort-based metrics (story points, velocity), and replace them with a recurring value judgment ('should this ship?').
• Closing the Gap Between AI Investment and Financial Return David L. Rogers, Krishnan Sankaranarayanan · 2026-09-23 Reader 🖥️
  Because enterprise AI value is the product of three multiplicative factors—technology, adoption, and strategic alignment—rather than their sum, companies fail to convert heavy AI investment into measurable financial return primarily because they neglect the most overlooked and highest-leverage factor, strategic alignment, and must instead anchor every AI investment to their own value drivers, industry-value-chain position, and income-statement economics rather than pursuing generic, vendor-shared use cases.
• Measuring AI's Unintended Consequences James Shore · 2026-09-22 Reader 🖥️
  Because AI's unintended consequences — declining maintainability, burnout, and vendor/token lock-in — carry real economic costs but, unlike delivery speed, cannot be measured by a controlled trial and instead surface only gradually and noisily, organizations should track a cluster of imperfect lagging indicators (chiefly value-add percentage, backed by DORA metrics and token usage) and build explicit long-horizon scenario models (a maintenance-cost 'tax,' an excess-code-churn model, and a lock-in variant of both) to inform — never replace — judgment, even though the most trusted of these indicators is itself unusually easy to game.
• Measuring AI's Impact on Delivery Speed James Shore · 2026-09-15 Reader 🖥️
  Because no unit of software output is standardized and raw on-time-delivery rates are corrupted by chronic estimate over-optimism, an organization can only find out whether AI is really making delivery faster by running a randomized, within-subject, repeated-measures trial that compares each person's or team's own 'with AI' and 'without AI' actual-to-estimate ratios, collected before assignment is known, under an otherwise identical process.
• John Cutler's Post John Cutler · 2026-09-12 source ↗ Reader 🖥️
  Shifting from outputs to outcomes fails not because the idea is wrong but because teams adopt the language of outcomes while keeping the psychology of outputs — chasing certainty, easy measurement, and reputational credit instead of committing to the qualitative judgment, tolerance for ambiguity, and honest attribution that outcome-orientation actually demands.
• Are You Getting Your Money's Worth from AI? James Shore · 2026-09-08 Reader 🖥️
  Because AI's costs and benefits in software development are real, substantial, and easily misjudged by anecdote, self-assessment, or gut feel, organizations facing consequential tradeoffs between speed, cost, and people's jobs must rigorously measure both sides — using durable, outcome-anchored, approach-level metrics rather than gameable activity proxies like lines of code, pull-request count, or individual performance targets.
• TBM 437: AI and the Recontextualization Tax John Cutler · 2026-09-03 Reader 🖥️
  Organizations pay a real, often-hidden cost -- the 'recontextualization tax' -- every time people flatten a working reality into a simplified version to suit whoever has the most power, the least appetite for detail, or the least time, and AI is newly good at absorbing that translation labor because it can render one underlying, accurate context into many audience-tailored formats without forcing teams to first standardize how they actually work.
• TBM 437: Tokens, Hours, Points, and Other Curious Proxies John Cutler · 2026-08-17 Reader 🖥️
  Tokens are the latest in a lineage of seductive measurement proxies -- hours, story points, flow metrics -- that organizations mistake for a theory of value because they happen to be unusually easy to measure, and the fix is not a better proxy but making the underlying causal model of value explicit (treating the team as the cost unit, funding durable lanes, stating falsifiable hypotheses) so that ROI becomes the *output* of that model rather than a number computed directly from an easy-to-meter input.
• From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI Margaret-Anne Storey · 2026-07-28 Reader 🖥️
  Software debt should be reasoned about across three interacting layers of a software system rather than one -- technical debt (in code), cognitive debt (in the team's shared understanding), and intent debt (in externalized rationale and goals) -- because generative AI is simultaneously shrinking technical debt through automation while accelerating the other two, previously under-recognized and increasingly consequential, forms of debt; managing software health in the AI era therefore requires treating shared understanding and externalized intent as first-class deliverables with the same discipline historically reserved for code quality.
• Recommendation algorithms might be making your entertainment boring, new research suggests Eric W. Dolan · 2026-06-02 source ↗ Reader 🖥️
  Because algorithms that optimize for short-term engagement mistake familiarity-driven engagement for quality and evaluate over a horizon far shorter than the decade-plus timescale on which aesthetic taste actually evolves, they systematically under-explore and trap users in an eventually-boring set of favorites — and a mathematical model shows that deliberately injecting moderate noise (forcing exploration) into such a system improves users' long-term satisfaction.

Propositions touching this concept (3)

By publication month, oldest first (extraction month when no publication date is known).

2026-02

• explicitCognitive debt — the fragmentation or loss of a team's shared theory of the system — can paralyze a team faster than technical debt.
  Fragments: February 13 · p6

2026-08

• explicitcorroboratedThe workable alternative to precise-but-invalid proxies is to treat the team as the indivisible cost unit, connect it to an explicit causal model of value, fund durable lanes of focus, maintain a documented business case (causal assumptions, proxy validity, expected returns, uncertainties), and judge return through movement in validated proxies rather than by attributing outcomes to individual units of work.
  TBM 437: Tokens, Hours, Points, and Other Curious Proxies · p9
• explicitdevelopedA defensible AI business case is not a single point-ROI estimate ('$10M spent -> $30M value') but a set of explicit, falsifiable 'theory of the system' hypotheses, from which ROI is properly the observed output rather than the initial pitch.
  TBM 437: Tokens, Hours, Points, and Other Curious Proxies · p13

Open gaps (12)

• incompleteThe implicit definition of 'benchmark' collapses to 'any scorable single-shot prompt.' This may be category-confusion with what the eval community means by 'benchmark' (a curated, statistically powered suite).
  Kaggle is making AI benchmark creation effortless — read next: Liang et al. on HELM construct validity; recent agentic-eval work (SWE-bench Verified, GAIA) on what makes a task suite vs. a task-bag.
• illogicalThe equation Value = Technology x Adoption x Strategic Alignment is presented as a genuine multiplicative model, with the associated 'any near-zero factor collapses value' inference, but is never operationalized with measurable units for any factor; it functions as a metaphor for 'all three matter and weak links dominate,' not a formula the authors have used to compute or predict value. Downstream inferences drawn from its multiplicative structure (e.g. which factor deserves the next marginal dollar) are not licensed by anything demonstrated in the piece.
  Closing the Gap Between AI Investment and Financial Return — read next: Compare against a piece that attempts actual quantitative ROI modeling of AI investment, to see the gap between a metaphorical and an operational 'AI value equation.'
• incompleteThe 'moat' claim does not address that the article itself publishes the general four-step method to a mass HBR audience, nor does it engage with why competitors following the same publicly available method wouldn't erode the described advantage. It also never distinguishes alignment-as-analysis (replicable by any competent competitor reading this article) from alignment-as-proprietary-data (a genuine, durable moat).
  Closing the Gap Between AI Investment and Financial Return — read next: Read strategy literature on sustainable competitive advantage / the resource-based view to test whether 'knowing your own economics' is actually rare or inimitable once the method for finding it is public.
• incomplete#11's attribution problem — 'confusing outcomes with attribution' — is stated as an observation but not analyzed: the post doesn't distinguish cases where attribution is genuinely impossible (overdetermined outcomes) from cases where it is merely hard but tractable with better instrumentation (e.g. A/B tests, holdout groups), which would call for different remedies.
  John Cutler's Post — read next: This corpus's extract-tbm-437-tokens-hours-points-proxies.json works this exact problem for AI-spend ROI, proposing 'the team as the indivisible cost unit' plus an explicit causal model instead of attributing outcomes to individual units of work — a template for resolving Cutler's #11 in a specific domain.
• incompleteNo mention of psychological safety or incentive/reward-system design, even though several failure modes (#4 credit-seeking, #9 cherry-picking around promo time, #11 fabricating causal stories) are natural responses to a reward system that punishes honest reporting of ambiguity, setbacks, or unattributable work. The post diagnoses the symptom (behavior) without naming the likely structural cause (what gets rewarded).
  John Cutler's Post
• incompleteThe article never states the model's functional forms, parameter values, or sensitivity analysis (how the appreciation curve is shaped, what noise levels were tested, how many Monte Carlo trials produced the reported effect sizes). Without these, a reader can't judge how robust the 'moderate noise helps' conclusion is to the specific modeling choices — a purely theoretical model's conclusions can be highly sensitive to functional-form assumptions.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Read the underlying paper directly — Samsun Knight, "Engagement-based curation and the evolution of taste," Journal of Cultural Economics (2026), https://doi.org/10.1007/s10824-026-09591-3 — for the actual model specification and robustness checks.
• incompleteThe article treats platforms as having essentially one dial (prediction accuracy vs. noise), but real recommendation systems already blend multiple discovery mechanisms — curated new-release rows, human-programmed playlists, explicit 'diversity' injections, cold-start exploration bonuses. It never asks whether real-world systems already implement something functionally like the paper's proposed fix, which would make the normative recommendation partially moot rather than novel.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Platform-side accounts of recommender-system design (e.g., published descriptions of Spotify's Discover Weekly / exploration mechanisms, or general recommender-systems literature on the exploration-exploitation trade-off) to check how much of this gap is already closed operationally.
• uninformedThe study is a purely theoretical, simulated model with no human-subject or real-usage data — the article flags this directly, but doesn't examine whether the inverted-U 'consumption capital' curve (moderate exposure raises liking, excess exposure causes satiation) is itself empirically well-established for genre-level entertainment tastes specifically, as opposed to being carried over from general cultural-economics theory (Stigler & Becker) or adjacent phenomena like the mere-exposure effect and hedonic adaptation.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Empirical psychology/economics literature on hedonic adaptation and the mere-exposure effect (Zajonc) to check whether an inverted-U specifically holds at the genre/style level for music, film, and fiction — not just for individual songs or images in lab settings.
• illogicalThe paper's proposed fix is framed as making the algorithm 'less accurate' or 'noisier,' but the actual causal lever the article describes is forced exploration regardless of predicted engagement — a distinct design choice from genuinely degrading prediction quality. Conflating 'noisy predictions' with 'committed exploration' muddies the practical implication: a platform could keep maximally accurate predictions everywhere and still reserve a fixed fraction of recommendation slots for exploration, capturing the same long-term benefit without the framing cost of 'deliberately recommend worse content.' This is exactly the systems-thinking move the article skips: it treats accuracy and exploration as one dial when the model's own results imply they're two.
  Recommendation algorithms might be making your entertainment boring, new research suggests — read next: Multi-armed bandit / exploration-exploitation literature (e.g., Sutton & Barto, *Reinforcement Learning*) which cleanly separates prediction accuracy from exploration policy — useful for reframing the paper's own proposed fix as a policy-layer change rather than an accuracy trade-off.
• incompleteThe essay asserts that PR count 'reflected' productivity pre-AI because engineers produced PRs 'consistently and logically,' and frames AI as the disruptive cause that broke this. But it never establishes that this pre-AI correlation was itself validated (versus a convenient habit-based convention) — leaving unclear whether AI broke a previously-sound metric, or merely removed the human regularity that had been propping up an already-unvalidated one.
  Are You Getting Your Money's Worth from AI? — read next: Compare against extract-tbm-437-tokens-hours-points-proxies and extract-8-engineering-metrics-ai-broke, which interrogate proxy validity independent of AI as the disruptive cause.
• incompleteThe post asserts 'anything focused on value judgment survives' but gives no account of who makes that judgment, on what basis, or how to prevent it from degrading into the same gameable, effort-disguised-as-value proxy that story points became — i.e. it names the right kind of metric (value, not effort) without addressing how to keep a value judgment itself honest and well-calibrated over repeated 2-day cycles.
  Erik van der Pluijm's Post — read next: Read against the corpus's broader treatment of proxy validity and Goodhart's Law for AI-era engineering metrics (extract-8-engineering-metrics-ai-broke and related) for frameworks on how teams keep qualitative/value-based signals from degrading the way quantitative ones did.
• incompleteThe 2-day cycle is presented from the inside of a single, apparently small team ('we've been trying'); the post doesn't address whether a 3-day cadence (plan/build/demo) scales to multiple interdependent teams, where cross-team dependencies, shared architecture decisions, and release coordination historically required longer sprint boundaries to synchronize.
  Erik van der Pluijm's Post — read next: Cross-read against the corpus's coordination-cost and multi-team scaling material to test whether compressed cycles survive beyond a single small team.