CannotMeasureProductivity

Martin Fowler · article · 2003-08-29 Reader 🖥️

Changeability & Optionality Code Was Never the Bottleneck Code Quality DORA & Engineering Metrics Measuring AI Impact Outcome Engineering Software Design Fundamentals

Software productivity cannot be measured because its true output — delivered business value — cannot be measured, so every popular proxy (lines of code, function points, feature counts, project 'success' rates) either measures the wrong thing or measures it too late to be useful.

theoretical · argumentative — A short bliki essay building a chain of counterexamples (LOC, function points, FP-adjusted-for-usefulness, success rates) to establish a negative epistemic thesis: software productivity cannot be measured. Practical in motivation (it's aimed at ending futile process/design arguments) but the content itself is conceptual analysis of a measurement problem, not a technique or how-to.

Structure

Part	Role
Opening framing	Sets up the stakes: emotional, unresolvable industry debates persist because we can't measure the basic effectiveness of software development
Productivity = output/input; we can't measure output	States the core definitional move that the rest of the essay unpacks
Lines of code critique	First counterexample: LOC rewards poor design (duplication) and varies with language/style, so it isn't a valid output measure, only a rough size measure
Function points critique (counting unreliability)	Second counterexample: FP counts vary by a factor of three between counters even before getting to what FPs mean
Function points critique (usefulness)	Third counterexample, escalating: even a perfect FP count conflates 'functionality delivered' with 'functionality that matters to the customer' — introduces Jeff Grigg's reuse-asymmetry point
Business value as true productivity	The essay's positive claim: true productivity must be measured in delivered business value, illustrated with the $10M vs $5M example
Success-rate critique	Extends the business-value argument to project 'success', showing raw success counts are as misleading as raw output counts
'If you can't measure it, you can't manage it' rebuttal	Pre-empts the standard objection by citing analogous unmeasurable-but-managed functions (law, marketing, education)
Individual vs. team productivity	Adds a further layer of difficulty: even if team output were measurable, individual contribution (especially supporting/enabling roles) is harder still
The Economist lag citation	Adds a temporal complication: even business-value measurement is confounded by a multi-year lag between investment and payoff (citing computers and electricity)
Closing admission of ignorance	Explicitly refuses the seductive false-measure alternative and ends on intellectual honesty about the limits of current knowledge

Key terms

productivity
Output divided by input for a software development activity — but Fowler insists this is only coherent once 'output' is defined correctly (as delivered business value), which for him makes productivity currently unmeasurable in practice.
output
The true deliverable of software development, which Fowler equates with delivered business value, not any artifact (code, function points) produced along the way.
direct productivity vs. true productivity
Direct productivity = functionality delivered per unit time (even if perfectly and accurately counted); true productivity = business value delivered per unit time. Fowler explicitly separates these and asserts only the latter is the real measure.
function points
A more sophisticated functionality-counting method than LOC, for which Fowler has 'a little more sympathy' but still rejects, both because counting itself is unreliable (3x variance between counters) and because even accurate counts don't capture business value.
success (project success)
Fowler redefines project success as delivering more business value than the project's cost — not simply completing/shipping — which makes naive success-rate counting (e.g., '4 of 5 projects succeeded') as misleading as output counting.

Propositions

explicitcontestedThe software industry cannot reasonably measure productivity because it cannot measure the true output of software development.conf: high
supportsThe 8 software engineering metrics AI broke · p10 — Fowler's core thesis — productivity is unmeasurable because output is unmeasurable — supports and predates the corpus's independently-argued claim that metrics were never meant to measure productivity or enable cross-team/individual comparison, only to serve as local feedback.
contradicts ←DORA Community Discussion - DX Core 4 · p16 — DX Core 4's DevEx Index claims a precise, validated 10-hours-saved-per-point conversion rate; Fowler argues the industry cannot currently measure true software productivity at all. These are direct, unreconciled opposing positions on whether productivity is currently measurable.
explicitLines of code (LOC) is not a valid measure of output because well-designed, well-factored code is shorter, while poorly designed code with duplication produces artificially inflated LOC counts.conf: high
extendsThe 8 software engineering metrics AI broke · p6 — Fowler's 2003 claim that LOC rewards poor design and can be mechanically inflated (Inline Method doubling LOC) is directly extended by the AI-era finding that AI-generated code is ~20% larger than human code for the same task, making LOC an input/token-spend proxy rather than an output-value measure.
explicitLOC remains a reasonably good indicator of system size (e.g., that a 100 KLOC system is larger than a 10 KLOC system), even though it is not a valid productivity measure.conf: high
explicitFunction point counts are unreliable in practice, with reported variance of up to a factor of three between different counters assessing the same system.conf: medium
explicitEven a perfectly accurate function-point count would not measure true productivity, because it conflates total functionality delivered with functionality that is actually useful to the customer.conf: high
supportsTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p10 — Fowler's point that even accurate function-point counts miss true value (30 of 100 FP actually useful) is an early instance of the general pattern the newsletter later names explicitly: an easy-to-meter input proxy (FP, tokens) is mistaken for a measure of the unmeasured value it's supposed to represent.
explicitInternal factors such as the degree of reuse available across a set of features materially affect how much functionality a developer can deliver in a given time, independent of skill.conf: medium
explicitTrue software development productivity must be defined in terms of delivered business value, not functionality or code volume.conf: high
supportsMeasuring AI impact, assessing readiness, and new data trends · p1 — Fowler's assertion that true productivity must be based on delivered business value, not functionality/code volume, is the same claim later restated as 'code generation was never the real bottleneck' — both locate the real measurement target upstream of code output.
explicitA project should be considered successful only if it delivers more business value than its cost, making simple success/failure counts across a portfolio of projects misleading as a productivity signal.conf: high
explicitOrganizations routinely and legitimately manage functions (e.g., legal, marketing, education) whose value they cannot measure, refuting the claim that unmeasurable things cannot be managed.conf: medium
explicitcorroboratedIndividual contribution within a team is harder to measure than team-level output, especially for people in supporting/enabling roles who raise others' productivity rather than shipping features themselves.conf: high
supports ←DORA Community Discussion - DX Core 4 · p4 — Both single out supporting/enabling roles (mentors, reviewers) as the clearest case where individual-level output measurement fails and team-level measurement is required.
explicitBusiness-value payoff from software investment can lag the investment itself by years, by analogy with computerization and electrification in the broader economy.conf: medium
explicitAdopting a false or invalid productivity measure is worse than admitting the industry currently lacks the ability to measure productivity at all.conf: high
implicitFowler implicitly believes that respected outlets (he names IEEE Software) publishing LOC-based productivity studies reflects an industry failure of rigor, not just an occasional lapse.conf: medium

Arguments

Software productivity cannot currently be measured. strength: high

Explicit premises:

Productivity = output / input.
Software's true output is delivered business value.
Delivered business value cannot currently be measured (accurately, without long lag, and separated from confounds).

Implicit premises:

A quantity that cannot be measured cannot be used to compute a ratio (productivity) that depends on it.
No proxy measure available today (LOC, function points, raw functionality, project success counts) tracks business value closely enough to substitute for it.

Inferences:

deductive chain from the definition of productivity to the unmeasurability of its numerator

Lines of code is an invalid productivity measure. strength: high

Explicit premises:

Good design eliminates duplication and therefore produces less code for the same functionality.
Copy-paste programming (poor design) produces more code (duplication) for the same functionality.
A refactoring tool (Inline Method) can mechanically double LOC while making design strictly worse.

Implicit premises:

Any measure that a mechanical, value-neutral transformation can arbitrarily inflate is not tracking value or effort.

Inferences:

counterexample/reductio: if LOC tracked productivity, this transformation would have to mean productivity doubled, which is absurd

Even an accurate functionality count (function points) is not a valid productivity measure. strength: medium

Explicit premises:

Two developers can each deliver a different quantity of function points in the same time.
Only a fraction of delivered function points may be useful to the customer.
Different features carry different levels of implementation difficulty depending on reuse potential.

Implicit premises:

Productivity should track value delivered to the customer, not raw output regardless of usefulness or difficulty.

Inferences:

thought experiment (Fowler vs. 'Joe') showing higher raw output can coexist with lower true productivity

Narratives

Fowler vs. 'Joe' delivering 100 FP (30 useful) vs. 50 FP (all useful) in the same year.
→ Demonstrates that raw functionality output can diverge from useful/valuable output, motivating the direct-vs-true-productivity distinction.
Jeff Grigg's quoted elaboration: Fowler's 100 FP were similar (high reuse potential, low leverage achieved), Joe's 50 FP were all different (no reuse possible), yet Joe still delivered in a year.
→ Adds a mechanism (reuse asymmetry) that would make Joe's output even more impressive, deepening the point that raw counts obscure real differences in difficulty and skill.
Fowler's four $1M-profit successes vs. Joe's single $10M-net success, out of five projects each.
→ Extends the business-value argument from output measurement to project 'success' measurement, showing a naive success-rate count (4-of-5 vs. 1-of-5) inverts who actually performed better.
The Economist's (Sept 2003) finding that 1990s computer investment only later showed up in productivity statistics, paralleling the historical lag after electrification.
→ Introduces a temporal confound into even a hypothetical business-value measure: value may not be observable for years after the software is built.

Questions addressed

✅ Why can't the software industry resolve its recurring, emotional debates about process and design practices?
Because the industry lacks a valid way to measure the effectiveness (productivity) of software development, so competing claims can't be adjudicated with evidence.
✅ Is lines of code a valid productivity measure?
No — it's a reasonable proxy for system size but rewards poor design and can be mechanically inflated without any change in value delivered.
✅ Are function points a valid productivity measure?
No — counting is unreliable in practice, and even a perfect count only measures direct functionality, not the business value that functionality carries.
✅ What would a true measure of software productivity actually have to be based on?
Delivered business value, per unit of input — though Fowler stops short of proposing how to measure this in practice.
⭕ How should individual (as opposed to team) productivity be assessed?
Fowler states it's harder than team assessment, especially for supporting/enabling roles, but offers no method — this is left as an open problem.
⭕ Given that we can't measure productivity, how should organizations manage software development at all?
Fowler only argues by analogy (law, marketing, education are managed without measurable productivity) that unmeasurability doesn't preclude management, but does not describe what that management should look like.

Gaps

incompleteFowler asserts that unmeasurable functions like law, marketing, and education 'are managed' without explaining how — leaving the practical alternative to measurement (what replaces metrics in actual management decisions) completely unaddressed.
Read next: The DORA/SPACE/Core-4 tradition and this corpus's own 'metrics as feedback mechanism, not comparison tool' framing (extract-8-engineering-metrics-ai-broke) is arguably the field's 20-years-later answer: use flow/quality metrics as local feedback signals for a team rather than as cross-team/cross-individual productivity rankings — exactly the distinction Fowler gestures at but never develops.
incompleteThe essay treats 'business value' as if it were a single, well-defined, eventually-knowable quantity, but never grapples with whose valuation counts, how value is attributed when multiple systems/teams/market factors combine, or how to handle value that is negative, deferred, or optionality-based (e.g., a platform investment whose value is future flexibility).
Read next: This gap becomes sharper, not smaller, twenty years on: modern outcome-vs-output debates (e.g., the corpus's John Cutler outcomes/outputs material) show that 'outcome' metrics are themselves gameable and contested, suggesting Fowler's proposed alternative to LOC/FP is not obviously more tractable than the problem it replaces.
uninformedWriting in 2003, Fowler could not have anticipated generative AI's effect on code volume: AI-assisted development now produces measurably more code per unit of human effort (with reported ~20% larger AI-generated code than human-written code for equivalent tasks), which would further inflate LOC-based studies and make his critique more urgent, not less.
Read next: Directly extended by this corpus's AI-era engineering-metrics critiques (extract-8-engineering-metrics-ai-broke, extract-tbm-437-tokens-hours-points-proxies), which independently rediscover that easily-measured, easily-inflated proxies (LOC, tokens, story points, PR counts) systematically substitute for the still-unmeasured true output.
incompleteFowler doesn't consider that measurement problems could be partially decomposed rather than solved wholesale — e.g., using narrower, falsifiable proxies for specific sub-questions (defect escape rate, time-to-recover, customer usage of a shipped feature) even while abandoning the goal of a single 'productivity' number.
Read next: The DORA Core-4/SPACE frameworks and the AI-era 'metrics were always meant as feedback, not comparison' reframing (extract-8-engineering-metrics-ai-broke#p10) represent exactly this move: give up on 'productivity' as a scalar while keeping several targeted, harder-to-game signals.

Closes gaps elsewhere

The Pulse: Interesting AI coding stats from Cursor · g1 — That report measures LOC volume per developer but never links it to delivered business value, leaving the author's own question ('are they adding a lot of business value?') unanswered. Fowler's essay supplies the exact theoretical explanation for why that link is missing and hard to make: LOC (and by extension AI-assisted LOC volume) was never a valid proxy for output, and only delivered business value counts as true productivity — a diagnosis of the report's blind spot, two decades early.

CannotMeasureProductivity

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents