Measuring AI's Impact on Delivery Speed
James Shore · article · 2026-09-15 Reader 🖥️
Because no unit of software output is standardized and raw on-time-delivery rates are corrupted by chronic estimate over-optimism, an organization can only find out whether AI is really making delivery faster by running a randomized, within-subject, repeated-measures trial that compares each person's or team's own 'with AI' and 'without AI' actual-to-estimate ratios, collected before assignment is known, under an otherwise identical process.
practical · mixed — Practical: the essay argues for a specific action (run a randomized, within-subject, repeated-measures trial to measure AI's effect on delivery speed) rather than for theoretical understanding of AI. Form is mixed — expository redefinition of terms ('delivery speed isn't productivity'), argumentative elimination of bad candidate metrics (feature/bug/story counts, on-time-estimate rate), and a constructive experimental-design walkthrough illustrated with a real dataset (Star Citizen). This is part 2 of James Shore's 4-part series (plus a CFO epilogue) quantifying AI's impact on software development: part 1 ('Are You Getting Your Money's Worth from AI?', already extracted as extract-shore-ai-money-worth-part1.json) established why rigor is needed and what not to measure (LOC, PRs, individual performance); this part supplies the constructive methodology for the delivery-speed leg specifically; part 3 (forthcoming, unintended consequences), part 4 (business outcomes), and a CFO-facing epilogue complete the series.
Structure
| Part | Role |
|---|---|
| Opening framing and series roadmap: restates the question ('is AI making software development faster, and by how much'), recaps part 1, and previews part 3 (unintended consequences, Sep 22), part 4 (business outcomes), and a CFO epilogue. | context / scope-setting |
| 'What We're Measuring' — redefines productivity as value produced divided by money spent, with delivery speed as only one component of it; explicitly defers business value to part 4. | definitional groundwork |
| 'The Core Approach' — rules out counting requests (features/bugs/stories/points, all arbitrary in size) and rules out on-time-estimate rate (estimates are chronically over-optimistic, or gamed via padding when they're not) as speed measures; proposes comparing a team's actual:estimate ratio to itself over time. | elimination argument narrowing the field to one workable proxy |
| Four 'snags' with the naive actual:estimate-ratio approach: (1) ratios follow a log-normal/S-curve distribution rather than a single consistent number, illustrated with real Star Citizen alpha 3.0 data; (2) people/teams vary from each other, so always compare an entity to itself; (3) knowing you'll use AI changes your estimate, so collect estimates before random assignment; (4) concurrent process changes (workflow streamlining) confound AI's specific effect, so interleave conditions rather than comparing time periods. | identification of confounds that any valid design must control for |
| 'The Complete Measurement Approach' — the randomized, within-subject, repeated-measures trial protocol: define both approaches, practice both, collect the estimate before randomly drawing the condition from a 'virtual bag,' track estimated time, actual time, task type and spending, attribute rework back to its causing task, and compare each person's own with/without data before aggregating; a caution that a 2x coding-speed gain is not a 2x overall gain if coding is only part of the week. | constructive methodology — the essay's positive contribution |
| 'Is It Worth It?' — acknowledges the design is a lot of work, but argues it's justified by the unusually high stakes (order-of-magnitude spending differences across approaches) and the METR finding that engineers overestimate AI's speed benefit by nearly 50%; argues sample-size burden is manageable and the hardest part is really defining the 'with AI' approach, which is worth doing regardless. | justification / cost-benefit closing argument |
| Closing transition: delivery speed is only part of the picture; sustainability of speed gains is deferred to part 3 (unintended consequences). | series transition |
Key terms
- delivery speed
How fast a team finishes assigned work, independent of whether that work was worth doing. Explicitly a subset of productivity, not a synonym for it. - productivity
Value produced divided by money spent — a strict, outcome-anchored ratio Shore contrasts with mere output or speed. - actual:estimate ratio
Actual time spent on a task divided by its pre-assignment estimate; Shore's chosen proxy for relative speed, valid only once compared to itself over time and purged of the four named confounds ('snags'). - snag
A specific way the naive before/after comparison of actual:estimate ratios can produce a misleading result — Shore identifies exactly four, each with a corresponding design fix. - randomized within-subject repeated-measures trial
The specific experimental design Shore proposes: each participant (or team) serves as their own control, doing both 'with AI' and 'without AI' tasks in randomized order, so comparisons are made within a person/team rather than across different people/teams — reducing the number of samples needed relative to a between-subjects design.
Propositions
- explicitDelivery speed is not the same as productivity; productivity is value produced divided by money spent, and going faster doesn't mean going in the right direction.conf: highextendsAre You Getting Your Money's Worth from AI? · p15 — Part 1 asserts that delivering software faster isn't the same as delivering value faster; part 2 operationalizes that distinction into a strict productivity = value/money formula and explicitly narrows this essay's scope to the speed leg alone, deferring value to part 4.
- explicitThere is no objective measure of software delivery speed because none of the outputs teams create are standardized — features, bugs, stories, velocity, and story points are all arbitrary in size and reflect what's being requested more than how fast the team is.conf: highextendsAre You Getting Your Money's Worth from AI? · p11 — Part 1 rules out lines-of-code as a productivity metric; part 2 explicitly cross-references that argument and generalizes the 'no standardized unit' critique to features, bugs, stories, velocity, and story points.extendsAre You Getting Your Money's Worth from AI? · p12 — Part 1 rules out pull-request count on the same 'no stable unit' logic; part 2 folds PR count into a broader list of arbitrary-sized output units that can't measure speed.
- explicitOn-time-estimate delivery is not a valid speed metric either: software estimates are famously inaccurate and teams are nearly always over-optimistic; a team that misses its estimates isn't necessarily slow, and a team that hits them consistently is often padding estimates, which causes the work to expand to meet the estimate.conf: high
- explicitA team's or person's actual:estimate ratio, tracked against itself over time, can reveal a genuine speed change — but only within a design that treats the ratio's whole distribution (an S-shaped, log-normal curve, not a single number) as the unit of comparison, always compares an entity to itself rather than to others, collects estimates before people know their task's AI condition, and interleaves 'with' and 'without AI' tasks under an otherwise-identical, already-streamlined process.conf: highrefinesTBM 437: Tokens, Hours, Points, and Other Curious Proxies · p2 — tbm-437 names three general failure modes shared by proxy metrics (hours, story points, tokens); part 2 itemizes four specific, concrete methodological confounds for one particular proxy (the actual:estimate ratio) and gives a design fix for each, which is more operational than tbm-437's general critique.
- explicitThe rigorous version of this design is a randomized, within-subject, repeated-measures trial: define clear 'with AI' and 'without AI' approaches, let people practice both, record each person's estimate before randomly drawing their task's condition from a mixed pool, and compare each person's own with/without data before aggregating across people.conf: high
- explicitRework caused by a task (e.g. code-review changes, bug fixes) must be explicitly attributed back to the task that caused it, or the measurement will misattribute its cost.conf: high
- explicitA raw improvement in coding speed does not translate one-to-one into overall delivery-speed improvement if coding is only part of the work — e.g. a 2x coding-speed gain yields only a 1.33x overall gain if 50% of the week is spent on other work.conf: high
- explicitIn a METR study, engineers overestimated AI's speed benefit by nearly 50%, illustrating that unmeasured self-assessment of AI's speed impact cannot be trusted.conf: highsupportsAre You Getting Your Money's Worth from AI? · p4 — Both cite the same METR finding that engineers' self-assessed AI speed gains diverge sharply from measured performance; part 2 restates it via the 'nearly 50% overestimate' framing.supportsComprehension at AI Speed: Building a Context Store for Evolutionary Architecture · p2 — Both point to the METR 2025 RCT finding that experienced developers were measured as slower with AI while believing themselves faster.
- explicitThis level of measurement rigor is worth the organizational cost specifically because the stakes are unusually high — order-of-magnitude differences in AI spending (hundreds vs. thousands of dollars per person per month) alongside inflated productivity expectations — unlike an ordinary process-improvement measurement exercise, which wouldn't justify this much rigor.conf: highsupportsAre You Getting Your Money's Worth from AI? · p6 — Part 1 argues a faster AI-assisted approach isn't automatically better ROI if its cost forces offsetting cuts; part 2's stakes framing (order-of-magnitude spending differences across approaches) reinforces why getting the speed measurement right matters enough to justify this much rigor.
- explicitThe study doesn't require a large number of samples if the targeted effect size and participant count are large enough, and teams can collect the data as part of their normal work rather than through a separate data-collection exercise.conf: high
- explicitThe hardest and most valuable part of running this study is not the statistics but explicitly, concretely defining how people are expected to use AI in the 'with AI' condition — work that is worth doing whether or not the study itself is ever run.conf: high
- implicitThe whole design assumes an organization can secure a statistician's involvement and enough authority and organizational buy-in to randomly assign live delivery work to 'with/without AI' conditions — a nontrivial organizational capability the essay assumes rather than argues for.conf: medium
- implicitThe design assumes tasks are ethically and practically interchangeable for random with/without-AI assignment, without addressing work where that assignment isn't feasible — safety-critical, contractually committed, or externally deadline-bound tasks.conf: medium
- implicitAs operationalized here, 'speed' still measures how fast assigned work gets done, not whether the work was worth doing — a distinction the essay explicitly defers to part 4, but one an organization could easily elide, treating a delivery-speed win from this study as if it already settled the ROI question.conf: medium
Arguments
To reliably measure whether AI increases delivery speed, an organization needs a randomized, within-subject, repeated-measures trial rather than any simpler before/after or cross-team comparison. strength: high
Explicit premises:
- No unit of software output (features, bugs, stories, points) is standardized enough to compare across teams or time.
- On-time-estimate rate is corrupted by chronic estimator over-optimism, or by padding when estimates are consistently hit.
- The actual:estimate ratio forms a log-normal distribution, not a single number, so only distributional shifts (not individual data points) are meaningful.
- People and teams vary systematically from each other, introducing noise unless each is compared only to itself.
- Knowing in advance whether a task will use AI changes the estimate itself, biasing any comparison unless the estimate is collected before assignment.
- Process changes introduced alongside AI adoption (streamlined workflows, closer business contact) are themselves productivity-improving and will be misattributed to AI unless interleaved rather than time-separated.
Implicit premises:
- The actual:estimate ratio, once purged of these four confounds, is itself a valid-enough proxy for speed — the essay treats the ratio as basically sound once the snags are handled, without addressing whether the underlying estimation process is stable enough across the length of a multi-week study to serve as a reliable denominator at all.
- Participants are able and willing to have their individual estimate/rework data collected this closely without the psychological-safety concerns Goodhart's Law implies (per part 1's own warning against tying metrics to individuals or teams).
Inferences:
- If teams' estimation skill itself drifts over the course of the study — e.g. people get better at estimating simply from practicing the protocol, independent of AI — that would reintroduce exactly the kind of within-team confound the four snags are designed to eliminate; the essay doesn't name this as a fifth snag.
This level of methodological rigor is worth the organizational cost, despite being 'a lot more complicated than just counting PRs.' strength: medium
Explicit premises:
- Productivity expectations and AI spending are both at unusually extreme, order-of-magnitude levels compared to a routine process-improvement question.
- The METR finding that engineers overestimate AI's speed benefit by nearly 50% shows that unmeasured judgment is unreliable at exactly the scale of decision this data would inform.
- The sample-size burden is manageable given a large-enough effect and participant count, and teams collect the data during normal work rather than a separate exercise.
- Defining precisely how people should use AI in the 'with AI' condition is valuable in its own right, independent of running the study.
Implicit premises:
- The value of knowing — even approximately — which approach is faster outweighs the risk that the study itself creates the kind of individual/team performance-measurement dynamic part 1 explicitly warned against ('Measure Approaches, Not People').
Inferences:
- There is an unresolved tension with part 1's own Goodhart's-Law discipline: this design is fundamentally a repeated, per-person, per-team comparison of actual vs. estimate, which is much closer to 'measuring people' than part 1's prescription admits — the 'approach, not people' framing survives only if results are reported and acted on strictly at the approach level and never surfaced per individual, a safeguard this piece doesn't explicitly state.
Narratives
- Real actual:estimate-ratio data from Star Citizen's publicly released alpha 3.0 task estimates and actuals (112 data points), plotted on a log scale and matched almost exactly by a best-fit censored log-normal curve, with a visible 'cliff' at the 1.0 ratio mark flagged as a classic sign of work expanding to meet its estimate.
→ Grounds the 'estimates aren't consistent' snag in a real, named, checkable public dataset rather than an anonymized anecdote — a notable methodological improvement over part 1's reliance on unnamed practitioner conversations — and visually demonstrates why a single before/after number would be misleading. - The Star Citizen data artificially split in half and manipulated to simulate a 'Without AI' vs. 'With AI (Simulated)' distribution shift, explicitly labeled by the author as not representing real-world AI impact.
→ Illustrates, without overclaiming real evidence, what a genuine AI-driven distributional shift in the actual:estimate ratio would look like on the same kind of chart — a pedagogical device the author is careful to flag as simulated. - An AI-use disclaimer footnote: Shore used OpenAI's GPT-5.6 'Sol' model in 'High' thinking mode to critique his own explanation and to generate the Python code that drew the article's graphs, with a public link to the full ChatGPT conversation.
→ Models, in miniature, the same 'named, checkable' rigor the essay argues organizations should apply to AI's impact — transparent process disclosure rather than an unverifiable claim about how the piece (or its figures) were produced.
Questions addressed
- ✅ How can you measure a change in delivery speed when no unit of software output is standardized across teams or time?
Compare each person's or team's own actual:estimate ratio distribution to itself over time, within a design that controls for the four named confounds. - ✅ Is on-time-estimate delivery a valid speed metric?
No — estimates are systematically over-optimistic, or gamed via padding when consistently hit, so hitting or missing them doesn't reflect team speed. - ✅ How do you separate AI's specific effect on speed from simultaneous process changes (workflow streamlining, closer business-developer contact) introduced alongside AI adoption?
Interleave 'with AI' and 'without AI' tasks under an identical, already-updated process, rather than comparing a pre-AI period to a post-AI period. - ✅ Is this level of measurement rigor proportionate, given the effort required?
Yes — because AI spending and productivity claims involve order-of-magnitude stakes far beyond a routine process-improvement question, and the sample-size burden turns out to be manageable. - ⭕ Does collecting per-person, repeated actual/estimate data for this trial create the same individual-performance-measurement risk that part 1's Goodhart's-Law discussion explicitly warned against?
Gaps
- incompleteThe essay never addresses estimation-skill drift over the course of the study — teams could get better (or worse) at estimating simply from practicing the protocol, independent of any AI effect, which would reintroduce exactly the kind of within-team confound the four 'snags' were designed to eliminate. It is a plausible fifth snag the piece doesn't name.
Read next: Look for discussion of calibration/learning effects in controlled measurement studies (e.g. how the METR study Shore credits for the design handled repeated-measures learning effects), or a statistician's treatment of practice effects in within-subject trial design. - incompleteThis design is a per-person, per-team, repeated comparison of actual to estimated time — functionally much closer to 'measuring people' than part 1's own prescription ('Measure Approaches, Not People,' grounded in Goodhart's Law) admits. The essay never states the safeguard that would preserve part 1's discipline (e.g., that results must be reported and acted on only at the approach level, never surfaced per individual), leaving open exactly the gaming risk part 1 warned about.
Read next: Watch whether part 3 or part 4 of this series addresses how the trial's per-person data should be governed, reported, or anonymized to avoid becoming an individual performance target. - incompleteThe essay doesn't address ethical or practical constraints on random assignment — work that can't be randomly denied AI assistance (or forced to use it) for safety, regulatory, or externally-committed-deadline reasons is left out of the protocol entirely.
Read next: Look for adaptations of within-subject experimental designs to constrained real-world engineering environments (e.g. regulated industries, on-call/incident work) rather than the idealized 'virtual bag' assignment described here. - incompleteThe essay asserts 'it doesn't take many samples' and that 'the exact number will depend on the number of participants, the amount of variance in your results, and how conclusive you want the results to be' (footnote 7), but gives no worked numbers, formula, or example power calculation — a real gap for a reader who wants benchmark rigor over reassurance, since the whole essay otherwise leads with concrete numbers (Star Citizen's 112 points, the METR nearly-50% overestimate, the 2x/1.33x example).
Read next: Look for the underlying METR study's own sample-size/power methodology (the design's stated inspiration), or a follow-up post from Shore with a worked example.
Related documents
- Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture
- Are You Getting Your Money's Worth from AI?
- Measuring AI's Unintended Consequences
- The 8 software engineering metrics AI broke
- Closing the Gap Between AI Investment and Financial Return
- John Cutler's Post
- Recommendation algorithms might be making your entertainment boring, new research suggests
- TBM 437: Tokens, Hours, Points, and Other Curious Proxies