TBM 437: Tokens, Hours, Points, and Other Curious Proxies

John Cutler · email · 2026-08-17 Reader 🖥️

AI Unit Economics Speaking the Language of Capital Allocation DORA & Engineering Metrics Measuring AI Impact Proxy Validity (Construct Validity of Measurement Proxies)Token Throughput

Tokens are the latest in a lineage of seductive measurement proxies -- hours, story points, flow metrics -- that organizations mistake for a theory of value because they happen to be unusually easy to measure, and the fix is not a better proxy but making the underlying causal model of value explicit (treating the team as the cost unit, funding durable lanes, stating falsifiable hypotheses) so that ROI becomes the *output* of that model rather than a number computed directly from an easy-to-meter input.

practical · mixed — Argumentative in structure (a thesis defended through a sequence of case studies) but practical in aim: Cutler is arguing for a specific way of doing something -- fund teams and judge AI/engineering investment through an explicit causal model rather than through hours, story points, or tokens. The theoretical scaffolding (construct validity, manufacturing-vs-R&D economics) is in service of that prescription.

Structure

Part	Role
Opening: the 'return on tokens' moment and the dumpling/buffet metaphor	context
Hours as a measurement proxy -- six construct-validity caveats (fungibility, quality-blindness, context-switching, non-linearity, long-term option value, non-independence of buckets)	evidence (case study 1)
Lean/flow metrics and story points -- their manufacturing lineage, the conditions (classes of service, stable demand, homogeneous units) under which they work, and why product/R&D work usually violates those conditions	evidence (case study 2)
The 'Alternative' -- treat the team as the cost unit, connect it to a causal model, fund durable lanes, maintain a business case, judge return through movement in validated proxies	conclusion / practical prescription
Tokens -- applying the same construct-validity critique to AI spend specifically, via a point-by-point comparison against human-capital investment	evidence (case study 3, the target case)
'What theory of the system are we betting on' -- ten fill-in-the-blank falsifiable hypothesis templates, contrasted against a naive point-ROI sentence	conclusion / practical prescription, applied specifically to AI spend
Conclusion -- team funding as salaries + token spend, and the same 'what demand, what causal model, what durable investment' questions applied to that combined figure	conclusion

Key terms

return on tokens
The vendor- and executive-favored framing that treats AI token consumption as a clean, ROI-ready denominator -- used ironically by Cutler as the latest instance of a recurring measurement mistake, not as a framework he endorses.
theory of the system Proxy Validity (Construct Validity of Measurement Proxies)
A set of explicit, falsifiable causal hypotheses ('we believe X will reduce Y without materially increasing Z') that a team or organization commits to about how an investment produces value -- the thing a serious business case actually consists of.
classes of service
The manufacturing/Kanban practice of segmenting work by sufficiently different work processes or value profiles before comparing flow metrics across it -- without this segmentation, changes in throughput or cost may just reflect a changed mix of what was produced.
capacity (as distinguished from hours spent)
An emergent, designed property of a system (a factory, a team) representing sustainable ability to meet demand -- not a fungible resource that gets 'spent' unit by unit at the moment of use.

Propositions

explicitThe current surge of interest in 'return on tokens' is driven at least as much by organizations wanting cover for layoffs they couldn't otherwise justify as by a genuine effort to understand how AI augments human capability.conf: high
explicitcorroboratedHours, flow/lean metrics (including story points), and tokens recur as measurement proxies because they share three failure modes: overweighting the input ('I') side of ROI over the return side, favoring shorter-term easier-to-attribute use cases, and gravitating toward whatever is easiest to measure.conf: high
refines ←Measuring AI's Impact on Delivery Speed · p4 — tbm-437 names three general failure modes shared by proxy metrics (hours, story points, tokens); part 2 itemizes four specific, concrete methodological confounds for one particular proxy (the actual:estimate ratio) and gives a design fix for each, which is more operational than tbm-437's general critique.
supports ←Are You Getting Your Money's Worth from AI? · p6 — Shore's staff-cuts example is a concrete instance of the general proxy failure mode tbm-437 names: overweighting the input/speed side of ROI over the net return.
explicitHours are not a fungible resource that can be infinitely reallocated -- skill-context specificity, time-of-day effects, and specialist bottlenecks mean who spends an hour matters as much as how many hours are spent.conf: high
explicitThe relationship between time invested and outcomes produced is non-linear and threshold-based, so metrics that treat hours as a continuous, linearly scaling input misrepresent the underlying production function.conf: high
explicitManufacturing-derived flow metrics (cycle time, throughput, WIP) require 'classes of service' segmentation to remain meaningful; without it, changes in throughput or cost may simply reflect a changed mix of work rather than a real performance change.conf: high
explicitAny use of story points beyond a disposable tool to spark discussion, right-size, or reality-check a near-term commitment is a 'dereliction of duty.'conf: high
extendsTBM 416: Investment Stewardship (As Habit) · p5 — Generalizes Cutler's own earlier 'a bad proxy is worse than no proxy' point from story points specifically to the full lineage of proxies (hours, flow metrics, tokens) sharing the same failure mode.
explicitIn R&D/discovery-style product work, 'waste' (failed experiments) is not inherently bad, because a failed attempt purchases knowledge for the next attempt -- the manufacturing framing of waste-as-pure-loss doesn't transfer.conf: high
explicitcorroboratedFlow/lean metrics are genuinely valuable, but only for continuous improvement -- understanding bottlenecks and system health -- and should not be used as a proxy for investment or value itself.conf: high
supports ←Measuring AI's Unintended Consequences · p5 — Both endorse flow/lean-style metrics for continuous improvement and internal trend-watching, explicitly not as a precise external ROI proxy — the same discipline Cutler prescribes generally, applied by Shore specifically to value-add percentage.
explicitcorroboratedThe workable alternative to precise-but-invalid proxies is to treat the team as the indivisible cost unit, connect it to an explicit causal model of value, fund durable lanes of focus, maintain a documented business case (causal assumptions, proxy validity, expected returns, uncertainties), and judge return through movement in validated proxies rather than by attributing outcomes to individual units of work.conf: high
extends ←John Cutler's Post · p11 — Cutler's diagnosis that attribution-demands push teams to avoidance or fabricated causal stories extends the tbm-437 prescription to judge return via validated proxies and a causal model 'rather than attributing outcomes to individual units of work' — same underlying critique of forced attribution, applied to team outcomes rather than AI spend.
supports ←TBM 437: AI and the Recontextualization Tax · p2 — Both pieces (published in the same TBM 437 issue) share Cutler's broader concern with organizations substituting an easy/legible proxy (a flattened report, a token metric) for the harder work of maintaining an explicit, faithful model of underlying reality/value.
explicit⚡ disputed & corroboratedTokens appear to solve the ROI-measurement problem because they are unusually precise and easy to meter, but this precision is illusory: it measures only the input side while the underlying value-creation system (augment vs. replace, commodity vs. differentiated capability, complementary assets) remains as uncertain as it was for hours or story points.conf: high
contradictsThe Agentic P&L: Beyond the Empire of Headcount · p14 — Cutler argues token precision is a metering illusion masking a fuzzy underlying value system; this directly opposes the PNL piece's claim that cost-per-cognitive-outcome is meaningfully measurable and stable enough to serve as a managerial control metric.
supportsMeasuring AI impact, assessing readiness, and new data trends · p6 — Supplies the construct-validity mechanism explaining why DX's early warning -- that 'every token dollar is a good dollar' is starting to break down -- is correct.
supports ←CannotMeasureProductivity · p5 — Fowler's point that even accurate function-point counts miss true value (30 of 100 FP actually useful) is an early instance of the general pattern the newsletter later names explicitly: an easy-to-meter input proxy (FP, tokens) is mistaken for a measure of the unmeasured value it's supposed to represent.
supports ←Are You Getting Your Money's Worth from AI? · p9 — Both warn that token cost's apparent measurability/precision is illusory and unstable, and shouldn't be mistaken for a solid basis for ROI projection.
supports ←Measuring AI's Unintended Consequences · p8 — Both treat token-usage metrics as far noisier and less validly precise than their apparent measurability suggests, warning against using tokens as a standalone ROI or cost-of-change signal.
explicitAI spending is not straightforwardly comparable to an equivalent amount of human-capital spending, because of structural disanalogies: employees generate positive externalities (knowledge, relationships, institutional memory) that token consumption does not automatically generate; salaried labor behaves as a fixed cost over the planning horizon while token consumption behaves as a variable cost; and the residual asset left behind by AI spend depends entirely on whether the organization deliberately captures it.conf: high
refinesThe Agentic P&L: Beyond the Empire of Headcount · p2 — Complicates the PNL piece's clean 'cost per cognitive outcome replaces headcount' substitution by detailing structural disanalogies (externalities, fixed vs. variable cost, residual-capital capture) that make token spend and human-capital spend not directly interchangeable.
explicitMarginal returns to AI spend likely decline as spend scales, because the first dollars target obvious, high-value use cases while later dollars chase increasingly marginal applications.conf: high
explicitdevelopedA defensible AI business case is not a single point-ROI estimate ('$10M spent -> $30M value') but a set of explicit, falsifiable 'theory of the system' hypotheses, from which ROI is properly the observed output rather than the initial pitch.conf: high
extendsTBM 416: Investment Stewardship (As Habit) · p6 — Extends Cutler's Bayesian-updating framing for engineering ROI into a concrete template of falsifiable 'theory of the system' hypothesis statements specifically for AI spend.
extends ←John Cutler's Post · p2 — Both treat a business/ROI claim as a falsifiable hypothesis to be tested and retired rather than a fixed pitch to be perfected; Cutler generalizes the same epistemology from AI-ROI business cases to team metrics in general.
implicitTokens do not create a new measurement crisis so much as expose a pre-existing one -- many companies had already abandoned rigorous product-work ROI or value-architecture thinking well before AI spend became visible and metered.conf: medium
implicitThe ease with which an input can be measured is inversely related to, not evidence for, how well-understood that input's actual economic return is -- an easy-to-meter cost can sit inside a value system that is unusually hard to pin down.conf: medium

Arguments

'Return on tokens' is not a solution to the ROI-measurement problem; it is the newest instance of an old, unsolved problem. strength: high

Explicit premises:

Hours failed as an investment proxy due to fungibility, quality-blindness, context-switching cost, and non-linearity.
Lean/flow metrics and story points failed as investment proxies because product work usually violates the manufacturing preconditions (classes of service, stable demand, homogeneous units) those metrics require.
The same three failure patterns (I-side focus, easy short-term attribution, gravitation toward what's easiest to measure) recur across hours, flow metrics, and tokens.

Implicit premises:

If a new metric shares the causal structure of prior failed metrics, it will fail in the same way regardless of how precisely it can be measured.
Measurement precision of an input is orthogonal to the validity of using that input as an ROI denominator.

Inferences:

Organizations and vendors currently embracing 'return on tokens' will eventually encounter the same attribution, non-linearity, and boundary-setting problems that undermined hours and story points, unless they change what they're measuring rather than just how precisely they measure it.

A serious AI investment case should be framed as an explicit, falsifiable 'theory of the system,' not as a point ROI estimate.