Are You Getting Your Money's Worth from AI?

James Shore · rss · 2026-09-08 Reader 🖥️

practical · mixed — Practical: the essay argues for an action (measure AI's ROI rigorously; don't measure X, Y, Z) rather than for a theoretical understanding of AI. Form is mixed — expository framing of the problem, argumentative critique of bad metrics (LOC, PRs, self-assessment), and short practitioner anecdotes used as evidence throughout. This is part 1 of a 5-part series (parts 2-4 plus a CFO epilogue, publishing 15/22/29 Sep 2026); it is extracted here on its own terms — as the 'why, and what not to do' essay — but its argumentative shape (setup + negative case) is incomplete by design and should be read as scaffolding for the constructive methodology promised in later parts.

Structure

Key terms

• Spending Efficiency / Cost per Value-Add Day
  Shore's own proposed metric (shown in a graph, not yet defined in prose): dollars spent per day of genuinely value-adding work, as opposed to overhead or maintenance time. Used to argue that AI can look cost-efficient in the near term and much less efficient over a longer horizon.
• value-add work / delivering value
  Work whose output actually matters to a business outcome, as distinct from any completed ticket, PR, or line of code. Used to argue that speed metrics and value are not interchangeable.
• approach (as the unit to measure)
  A specific way of using AI — a tool, workflow, or policy choice — as opposed to a person or team. Shore's proposed antidote to Goodhart's Law: measure and compare approaches, not the people using them.
• slop AI Slop
  AI-generated code that is voluminous but not valuable — Shore's shorthand for what lines-of-code actually measures once AI is doing the writing.

Propositions

• explicitOne CIO's 170-person organization saw a 2.1x productivity increase from AI, but top individual users spent as much as $21,000/month on tokens, and a separate engineering leader reported $6,000-$8,000 per engineer per month.conf: high
• explicitAI's costs will likely come out of the personnel budget, creating real consequences for real people's livelihoods, while failing to act on AI also carries real business consequences.conf: high
• explicitAnecdotal reports about AI's impact are unreliable — the reasoning behind them is often poor, and some claims are outright fiction.conf: high
• explicitcorroboratedSelf-assessment of AI's speed benefit cannot be trusted: in a METR study, engineers believed AI made them faster (self-rated 20% faster) while being measured as slower (19% slower).conf: high
  supportsComprehension at AI Speed: Building a Context Store for Evolutionary Architecture · p2 — Both cite the same METR 2025 RCT finding that developers' self-assessed AI speed gains diverge sharply from measured performance.
  supports ←Measuring AI's Impact on Delivery Speed · p8 — Both cite the same METR finding that engineers' self-assessed AI speed gains diverge sharply from measured performance; part 2 restates it via the 'nearly 50% overestimate' framing.
• explicitOrganizations cannot make sound AI cost/benefit tradeoffs from gut feel alone; impact must be measured.conf: high
• explicitcorroboratedA faster AI-assisted approach is not automatically better ROI — if its token cost is high enough to force offsetting staff cuts, the apparent speed gain may not be a real net gain.conf: high
  supportsTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p2 — Shore's staff-cuts example is a concrete instance of the general proxy failure mode tbm-437 names: overweighting the input/speed side of ROI over the net return.
  supports ←Measuring AI's Impact on Delivery Speed · p9 — Part 1 argues a faster AI-assisted approach isn't automatically better ROI if its cost forces offsetting cuts; part 2's stakes framing (order-of-magnitude spending differences across approaches) reinforces why getting the speed measurement right matters enough to justify this much rigor.
• explicitThere is no single best approach to AI in software development; the right approach depends on organizational context (e.g. internal IT shop vs. startup, new product vs. legacy system).conf: high
• explicitToday's AI token prices are artificially depressed by vendor subsidies competing for market share and should not be assumed to hold when projecting ROI.conf: high
  supportsHow to Respond to the Coming AI Cost Shock · p2 — Both identify current AI/token pricing as vendor-subsidized and therefore an unstable basis for projecting future costs.
• explicitdevelopedThe future direction of AI pricing is genuinely uncertain — it could rise as subsidies end, or fall due to open-weight models, local inference, cheaper inference chips, routing, and more token-efficient models — so ROI models should state cost assumptions explicitly and test both optimistic and pessimistic scenarios.conf: high
  supportsTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p10 — Both warn that token cost's apparent measurability/precision is illusory and unstable, and shouldn't be mistaken for a solid basis for ROI projection.
  refinesHow to Respond to the Coming AI Cost Shock · p12 — Shore's explicit instruction to model both optimistic and pessimistic pricing scenarios directly corrects the flaw the other extraction flags: that piece's own ROI math treats current subsidized rates as a stable 'maximum justifiable per-unit cost' even though its own thesis says pricing and usage are volatile.
  supportsAgentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p4 — Tokenmaxxing's own history (subsidies vanished, API prices rose, unlimited-spend policies got rolled back once companies went public) is a concrete precedent for the upward-price scenario Shore says organizations should model.
  extends ←Measuring AI's Unintended Consequences · p13 — Part 1 argued future AI pricing direction is genuinely uncertain and ROI models should test optimistic/pessimistic scenarios; part 3's lock-in model operationalizes that recommendation specifically for token-price increases forcing headcount cuts or reduced AI use.
• explicitAI adoption carries a change-management cost: productivity often dips before it rises as people learn to apply AI effectively, which can make early ROI measurements misleading.conf: high
• explicitdevelopedLines of code is not a valid measure of AI-era productivity: AI's verbosity means more code measures 'slop,' not value, and all else equal, more code is more expensive to maintain and more bug-prone.conf: high
  supportsThe 8 software engineering metrics AI broke · p6 — Both identify AI's code verbosity as turning lines-of-code into an input/volume signal correlated with token spend rather than a value signal.
  extends ←Measuring AI's Impact on Delivery Speed · p2 — Part 1 rules out lines-of-code as a productivity metric; part 2 explicitly cross-references that argument and generalizes the 'no standardized unit' critique to features, bugs, stories, velocity, and story points.
• explicitdevelopedPull-request count is not a valid measure of AI-era productivity, because its former loose correlation with development pace depended on engineers producing PRs at a human, 'consistent and logical' rate — a regularity AI breaks by producing more code than engineers can review.conf: high
  supportsThe 8 software engineering metrics AI broke · p1 — Both flag PR-based (and other activity) metrics as inflated by AI without a corresponding improvement in engineering quality.
  extends ←Measuring AI's Impact on Delivery Speed · p2 — Part 1 rules out pull-request count on the same 'no stable unit' logic; part 2 folds PR count into a broader list of arbitrary-sized output units that can't measure speed.
• explicitOnce people believe pull-request count is being used to measure productivity, they will be tempted to produce more PRs than necessary to game the measurement.conf: high
  extendsThe 8 software engineering metrics AI broke · p7 — Shore supplies a concrete mechanism — engineers producing unnecessary PRs once PR count is known to be watched — for Goodhart's Law becoming 'inescapable' under AI-era tooling.
• explicitdevelopedMetrics of AI's impact should be used to evaluate a particular approach to using AI, then retired once learned from — never used as an ongoing measure of individual or team performance — per Goodhart's Law.conf: high
  supportsThe 8 software engineering metrics AI broke · p10 — Both argue metrics were originally meant as feedback mechanisms, not tools for measuring individual/team performance or comparison, and that this got lost.
  refines ←From dashboards to decisions · p7 — Shore's Goodhart's-Law discipline says metrics should evaluate an approach and then be retired, implicitly treating gameability as a universal risk; this piece's Time-to-First-PR case refines that by showing a metric can be deliberately gamed on purpose and still produce a genuine, durable improvement, when gaming and improving are structurally the same action.
  extends ←John Cutler's Post · p2 — Cutler's 'metrics are hypotheses, not perfection targets' generalizes the shore-ai piece's Goodhart's-Law-motivated rule that AI-impact metrics should be used to evaluate an approach then retired, not treated as an ongoing performance target.
  extends ←Measuring AI's Unintended Consequences · p16 — Reapplies part 1's Goodhart's Law discipline (measure approaches, not people) to a new metric, value-add percentage, while explicitly admitting its limits in this new case.
• explicitdevelopedDelivering software faster is not the same as delivering value faster: doubling the bug-fix rate does not have the same business value as doubling the rate of competitive new features, and speed applied to poorly-conceived features can make the product worse.conf: high
  supportsAI isn't making developers more productive – it's making them busier · p1 — Both distinguish raw output/speed from delivered business value, warning that faster activity doesn't guarantee more value shipped.
  extends ←Measuring AI's Impact on Delivery Speed · p1 — Part 1 asserts that delivering software faster isn't the same as delivering value faster; part 2 operationalizes that distinction into a strict productivity = value/money formula and explicitly narrows this essay's scope to the speed leg alone, deferring value to part 4.
• explicitdevelopedIf AI simultaneously increases feature delivery and maintenance burden, a team's overall long-run productivity can spike and then fall below its pre-AI baseline, even though the short-term effect looks like a clear win.conf: medium
  extends ←Measuring AI's Unintended Consequences · p10 — Part 1 previewed a 'Spending Efficiency' graph and the claim that AI-driven productivity can spike then fall below baseline, without defining the underlying model; part 3 supplies and fully works the maintainability-tax model that produces exactly that spike-then-cross-under shape, with concrete numbers.
• explicitdevelopedBurnout is a live risk of intensive AI use (e.g. under spec-driven development); burned-out engineers produce worse work, don't push back or report problems, and eventually leave, taking institutional knowledge with them.conf: medium
  supportsAI productivity is burning out your best engineers · p7 — Both treat apparent AI speed gains as potentially borrowed against a depletable human resource (burnout / sustainability).
  extends ←Measuring AI's Unintended Consequences · p14 — Part 1 flagged burnout risk via a single anonymized anecdote; part 3 supplies a concrete (if explicitly unmodelable) measurement protocol — a mental-exhaustion scale folded into the speed study, periodic surveys, turnover tracking, and exit-interview attention.
• implicitdevelopedRigorous measurement of AI's ROI is practically achievable by a typical engineering organization, not merely desirable in principle.conf: medium
  extends ←Closing the Gap Between AI Investment and Financial Return · p12 — Supplies an enterprise-strategy-level method (define financial-return metrics via value drivers, before building) for Shore's general claim that rigorous AI ROI measurement is practically achievable, at a different (organizational rather than engineering-team) level of analysis.
• implicitBecause AI-adoption decisions can cost people their jobs, they carry a distinct ethical weight that obligates a higher standard of evidence than an ordinary tooling decision would.conf: medium

Arguments

Narratives

• An unnamed CIO with a 170-person organization measured a 2.1x productivity increase from AI, but his top individual AI users each spent up to $21,000/month on tokens; a second unnamed engineering leader reports $6,000-$8,000 per engineer per month.
  → Grounds the abstract claim that 'AI has real, substantial costs and benefits' in concrete, practitioner-sourced dollar figures rather than vendor marketing — supports the essay's opening propositions and its overall stance against anecdote-free hand-waving.
• An engineering leader reports that each team member was spending thousands of dollars in tokens against the cost of a $200 subscription.
  → Makes the subsidy argument concrete — illustrates that today's sticker prices for AI tools understate the real per-user token cost, supporting the claim that current costs are not a stable basis for ROI projection.
• The METR study footnote: participants self-rated as 20% faster with AI, but were measured as 19% slower.
  → Supplies the essay's single hardest piece of controlled evidence against trusting self-assessment or gut feel — the load-bearing example for the whole 'you must measure, not guess' argument.
• An engineering leader using spec-driven development tells Shore, 'every team is reporting burnout.'
  → Supports the claim that AI intensifies burnout risk with real downstream costs (quality, attrition, institutional knowledge loss) — but notably, this is itself an unsourced, anonymized anecdote of exactly the kind the essay opens by warning readers not to trust.

Questions addressed

• ⭕ How do you know whether you're getting a good return on AI investment in software development?
  Not answered in part 1 — the essay argues only that the question matters and that gut feel/anecdote can't answer it; the measurement methodology is deferred to parts 2-4.
• ✅ What should you NOT measure when trying to quantify AI's impact?
  Lines of code, pull-request count, and any metric used to measure individual or team performance rather than an approach.
• ✅ Should you assume today's AI token costs will hold when projecting ROI?
  No — today's costs are subsidized and unstable; model both optimistic and pessimistic pricing scenarios with explicit assumptions.
• ✅ Is a bigger AI-driven productivity multiplier automatically better?
  No — not if its cost forces offsetting cuts, or if it isn't sustainable once maintenance costs and burnout are factored in over the long term.
• ⭕ If metrics should be tied to evaluating an approach and then retired ('learn and stop'), how do you know when a shifting cost/capability landscape (which the essay itself says to expect) warrants re-measuring an approach you'd already stopped measuring?

Gaps

• incompleteThe essay repeatedly relies on unsourced, anonymized anecdotes (an unnamed CIO with 170 people, an unnamed engineering leader on subsidy pricing, an unnamed spec-driven-development practitioner on burnout) to ground its claims about costs and burnout — without stating how these were collected, verified, or whether they're representative. This is the same evidentiary category ('anecdotal reports... reasoning... not ideal') the essay explicitly warns readers against trusting elsewhere in the very same piece.
  Read next: Watch whether parts 2-4 of this series (publishing 15/22/29 Sep 2026) ground the promised measurement methodology in named, checkable data sources rather than anonymized practitioner conversations, and compare its rigor against the controlled METR RCT Shore himself cites as the gold standard.
  closed byMeasuring AI's Unintended Consequences — The excess-lines-of-code model is partially validated against real, named, checkable commit data from the open-source tmux project (provided by Christophe Stjernlöf, LLM-classified as value-add/maintenance across a 19-year history) — giving one component of the series' promised methodology a checkable data source rather than an unsourced anonymized anecdote. This is a partial closure: the model's other key inputs (maintenance-cost multipliers, headcount-reduction assumptions) remain estimated via 'wisdom of the crowd,' and the value-add-percentage recommendation still rests on Shore's own unreplicated anecdote (see this extraction's own gap 1).
• incompleteShore's remedy for Goodhart's Law ('measure the approach, learn from it, then stop measuring') is not reconciled with his own point that AI's pricing and capability landscape is actively shifting (subsidies ending, prices potentially dropping, models improving). If an approach's cost/benefit profile can change after the fact, 'stop measuring' risks leaving an organization blind to exactly the kind of change the essay warns about earlier in the same piece.
  Read next: Watch for whether later parts of the series (or Shore's broader writing) specify a cadence or trigger condition for re-measuring an approach, rather than a one-time 'learn and stop.'
• incompleteThe essay asserts that PR count 'reflected' productivity pre-AI because engineers produced PRs 'consistently and logically,' and frames AI as the disruptive cause that broke this. But it never establishes that this pre-AI correlation was itself validated (versus a convenient habit-based convention) — leaving unclear whether AI broke a previously-sound metric, or merely removed the human regularity that had been propping up an already-unvalidated one.
  Read next: Compare against extract-tbm-437-tokens-hours-points-proxies and extract-8-engineering-metrics-ai-broke, which interrogate proxy validity independent of AI as the disruptive cause.
• incompleteAs explicitly signaled by the essay's own roadmap, part 1 argues only for the necessity of measurement and what not to measure — it does not itself deliver a constructive methodology for measuring AI's benefits. A reader can evaluate the critique here, but has no basis yet to judge whether the series' eventual positive methodology (parts 2-4, plus a CFO epilogue) will hold up.
  Read next: Read and extract part 2 (measuring delivery speed, 15 Sep 2026), part 3 (unintended consequences, 22 Sep), part 4 (business outcomes, 29 Sep), and the CFO epilogue (29 Sep) when published, and check whether they resolve the gaps flagged here.
  closed byMeasuring AI's Unintended Consequences — Part 3 supplies constructive methodology for the unintended-consequences dimension specifically named in g4's reading direction: the cost-of-change construct, value-add percentage/DORA/token-usage as indicators, and three explicit scenario models (maintainability-as-tax, excess-lines-of-code, lock-in). This is a partial, not full, closure of g4 — the models explicitly consume part 2's speed-trial output as an input (not yet extracted in this corpus), and part 4 (business outcomes) and the CFO epilogue are still needed to complete the series' promised picture.

Closes gaps elsewhere

• The 8 software engineering metrics AI broke · g3 — The same 'measure approaches, not people, and stop once you've learned' discipline addresses the gap's concern that recommended outcome metrics are themselves targetable — Shore's answer is not a metric that resists gaming, but a discipline (approach-level, time-boxed measurement) that reduces the incentive to game any metric.
• TBM 437: Tokens, Hours, Points, and Other Curious Proxies · g2 — Shore's 'measure the approach, then stop measuring' discipline — never tying a metric to individual/team performance, and retiring it once you've learned from a given approach — is a concrete, general mitigation for the Goodhart's-Law risk this gap flags in leading-indicator proxies, applicable regardless of which metric (tokens, PRs, or a validated leading indicator) is chosen.

Related documents

• AI isn't making developers more productive – it's making them busier
• Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture
• Measuring AI's Unintended Consequences
  AI Unit Economics Burnout & Recovery Goodhart's Law Measuring AI Impact
• The 8 software engineering metrics AI broke
  Goodhart's Law Measuring AI Impact Technical Debt
• The Agentic P&L: Beyond the Empire of Headcount
  AI Unit Economics Enterprise AI Adoption & Readiness Technical Debt
• 2024_final_dora_report
  Burnout & Recovery Enterprise AI Adoption & Readiness Measuring AI Impact
• Closing the Gap Between AI Investment and Financial Return
  Enterprise AI Adoption & Readiness Measuring AI Impact Proxy Validity (Construct Validity of Measurement Proxies)
• Research: How AI Agents Broaden the Scope of Knowledge Work
  AI Unit Economics Enterprise AI Adoption & Readiness Measuring AI Impact
• How People Are Really Using AI in 2026
  AI Slop Enterprise AI Adoption & Readiness Measuring AI Impact
• Engineering Leadership Report 2026
  Burnout & Recovery Enterprise AI Adoption & Readiness Measuring AI Impact