Article: When Spec-Driven Development Pays Off

Nitin Garg · article · 2026-09-10 source ↗ Reader 🖥️

practical · argumentative essay structured as an empirical report: a governance framework is proposed, then tested via a controlled human study and an LLM replication, then qualified into an actionable targeting rule — Read as a governance argument disguised as a methods paper, or vice versa. The rhetorical move is to state an intuitive claim (specs help review), let a measurement falsify the intuitive half (recall) while confirming a narrower, less obvious half (attribution), and then convert the surprising result into a scoped recommendation. The honesty about limits (small n, confounds, single domain) is itself part of the argument's credibility strategy.

Structure

Key terms

• Attribution
  A binary property scored per review finding: whether the reviewer tied it to a specific named, approved requirement or invariant, as opposed to reporting it as generic 'something looks off.' This is the study's central dependent variable and the article's actual payoff.
• Drift / Drift Review
  The specific human act of judging whether AI-generated code diverges from an agreed, approved design baseline — the exact act the whole study is built to measure, distinct from the drift log artifact it produces.
• Governance Baseline
  The approved, versioned combination of specification + HLD + LLD, treated as an authoritative governing artifact that generated code, drift detection, and human review are all accountable to — explicitly contrasted with using a specification merely as inline prompt text.
• Delivery (vs. Presence)
  How a specification is introduced into the generation process — a staged, fresh-generation step vs. inline in a single prompt — as the variable that actually moves outcomes, contrasted with 'presence,' meaning merely whether a spec exists at all.
• Targeting Rule
  A coined decision heuristic for when the cost of specification governance is worth paying: specifically hard, multi-constraint, regulated, long-lived systems built by a capable-but-imperfect model — as opposed to a blanket 'always write specs' mandate.

Propositions

• explicitThe bottleneck in AI-assisted software delivery has moved from writing code to verifying it, which reframes the central problem as governance — accountability, divergence detection, and the human/model split of oversight labor — rather than a model-capability problem.conf: high
  supportsFrom AI Code to Trusted Software: Harness Engineering in Practice · p4 — Both state the same bottleneck-shift-to-verification thesis in near-identical terms; this article supplies a controlled study of the reviewer act (drift review) that the harness-engineering piece asserts more generally.
  supportsA cautionary tale about AI in tech writing: Why "Trust but Verify" is the only way to scale · p9 — An independent domain (technical writing, not code) converges on the same claim that verification, not generation, is now the scarce resource.
  supportsSam Hatoum's Post · p6 — Hatoum asserts the constraint moved from implementation speed to verification capacity; this proposition states the same shift in different vocabulary (writing to verification).
• explicitA specification baseline (spec + HLD + LLD treated as an approved, versioned governance artifact) does not improve a reviewer's ability to find bugs in AI-generated code: recall was statistically indistinguishable between Baseline and Code-only conditions (0.525 vs. 0.518, p=0.69).conf: high
  refinesSam Hatoum's Post · p6 — Hatoum names 'verification capacity' as the new constraint in the abstract; this study operationalizes and measures it directly, and the measurement narrows what that capacity actually consists of — attribution/accountability capacity, not raw bug-finding capacity, since recall itself did not move.
• explicitcorroboratedA specification baseline dramatically raises the attributability of review findings — reviewers tied 81% of findings to a named requirement with a baseline versus 0% without one (p=0.043) — which is the actual mechanism by which spec governance produces value.conf: high
  supports ←Spec-Driven Production Grade Development In The Age Of Vibe Coding · p8 — Both push review attention toward a structured, named artifact (a bundled AI-generated risk assessment / an approved spec baseline) that lets reviewers tie findings to something specific, rather than relying on free-floating line-by-line critique.
• explicitThe recall-null / attribution-gain pattern replicates when large language models, rather than humans, occupy the reviewer role across ninety machine reviews and three models, with attribution collapsing to exactly 0.00 without a baseline.conf: high
• explicitHow a specification is delivered into the generation process matters more than whether one exists: a single prompt combining spec-then-code performed no better than asking for code directly, while a staged 'author the spec, then implement in a fresh generation step' approach nearly doubled the pass rate on a weak model (23.8% to 45%) and halved failed builds.conf: medium
• explicitOn easy, single-function tasks, most of the apparent quality gain attributed to 'specifying first' is actually a reasoning effect: a chain-of-thought control with no specification at all captured nearly all of the same gain (59% direct code, 95% reason-first, 92% spec-first; spec-vs-reasoning not significant).conf: high
• explicitSpecification governance pays off specifically on hard, multi-constraint work built by a capable-but-imperfect model: the weaker model gained about 21 points from spec discipline on the complex banking task, while the already-strong model gained only about 2 points.conf: medium
• explicitSpecification governance imposes a real, measurable, recurring cost: spec-anchored review took about 48 minutes versus 27 minutes for code-only review (p=0.043), plus an ongoing up-front discipline of authoring and re-synchronizing the spec/HLD/LLD baseline as the system evolves.conf: high
• implicitA finding's attributability to an approved, named clause is what regulatory 'meaningful human oversight' substantively consists of — not some further, harder-to-observe property of the human's judgment.conf: medium
• implicitAccountability is irreducibly human: regardless of how much of the generation work a model performs, only a named human can occupy the Accountable role in a RACI split, and this allocation is treated as self-evidently correct rather than as one design choice among others.conf: high
• implicitReviewer-level proxies — recall and attribution rate, measured on a single review pass — are valid stand-ins for the actual governance value an organization cares about, even though the study never measures downstream outcomes such as defects escaping to production, incident rates, or how an attribution trail performs under an actual audit.conf: medium

Arguments

Narratives

• A worked banking-service example: a money-transfer business rule ('never leave a balance negative,' 'never create or destroy money') traced through all three layers — business requirement, HLD TransferService interface, and LLD invariants such as transfer_idempotent, transfer_atomic_on_fail, transfer_moves_funds, and assets_conserved — as one thread among twenty total invariants across core money operations.
  → Makes the abstract three-layer specification concept concrete and shows precisely what a reviewer checks against, grounding the governance framework before the study results are presented and making the later 'attribution' finding legible (attribution means citing exactly one of these named invariants).
• The five-reviewer, 2x2 within-subject drift-review study on two independently AI-generated banking services (Java and Python, with JUnit/Mockito test suites), where ground truth was adjudicated by source inspection because a single executable test suite missed real defects the reviewers correctly found.
  → Supplies the empirical backbone for the recall-null / attribution-gain claim and lends the argument methodological credibility through specific statistical machinery (Wilcoxon signed-rank, Fleiss' kappa) named in the text.
• A ninety-review automated replication placing three frontier LLMs (a mix including Claude Opus 4.8, GPT-5.2, DeepSeek V4, and Gemini 3.1 across generation and review) in the reviewer seat, reproducing 0.67 attribution with a baseline versus exactly 0.00 without one, across all three models.
  → Generalizes the human finding to a second population and preempts the objection that the recall-null/attribution-gain pattern is a quirk of five specific people rather than a structural property of what a baseline provides.
• Code-only reviewers repeatedly hedging in their reports that they 'could not tell whether the behavior was intended.'
  → A quoted line that crystallizes the quantitative attribution gap into a single memorable, human moment, functioning as the rhetorical payoff of the statistical finding.

Questions addressed

• ✅ Does giving reviewers an approved specification baseline help them find more bugs in AI-generated code?
  No — recall was statistically indistinguishable between Baseline and Code-only conditions in both the human study and the LLM replication.
• ✅ What, then, does a specification baseline actually buy a reviewer?
  Attribution — the ability to tie a finding to a specific approved requirement or invariant, turning a hunch into an accountable, defensible claim.
• ✅ Are the gains commonly attributed to 'specifying first' really about the specification, or about the extra deliberation it forces?
  On easy tasks, mostly the latter — a chain-of-thought-only control captured nearly all of the apparent spec-first gain.
• ✅ When, if ever, is specification governance worth its cost?
  On hard, multi-constraint, regulated or high-stakes, long-lived systems built by a capable-but-imperfect model — not as a universal practice.
• ⭕ Does the artifact trail produced by this governance loop actually satisfy what regulators (EU AI Act, ISO/IEC 42001, NIST AI RMF) mean by 'meaningful human oversight' and adequate record-keeping?
• ⭕ Would spec governance's benefits and the specific 21-point vs. 2-point capability gap hold up across more reviewers, more services, and more domains than one small banking simulation with one weak and one strong model?

Gaps

• incompleteThe article asserts that the five-control-point artifact trail 'can provide supporting evidence for the governance, traceability, record-keeping, and human-oversight controls emphasized by NIST AI RMF, ISO/IEC 42001, and the EU AI Act,' but never examines what these frameworks actually require to accept evidence as sufficient (e.g. independent audit, risk-tier-specific documentation depth, conformity assessment) — a compliance claim asserted without engaging the compliance literature it invokes.
  Read next: Read primary regulatory texts or a legal/compliance-focused analysis of the EU AI Act's Article 14 human-oversight requirements and ISO/IEC 42001's audit criteria to test whether 'attribution to a named invariant' meets the bar these regimes actually set.
• incompleteThe load-bearing 'delivery beats presence' claim — that a staged spec-then-generate step, not a spec's mere presence, is what worked — rests on a result the author himself reports as p≈0.18 at n=4 with an acknowledged second-pass confound, yet the surrounding prose states the causal claim in confident, unqualified language ('Only the one that treated the spec as a governing artifact... moved the outcome') that outruns the reported statistics.
  Read next: Look for a larger-n replication of the delivery-mode experiment that adds a control arm doing two generation passes with no specification, to disentangle the second-pass confound before treating 'delivery beats presence' as established rather than suggestive.
• illogicalThe RACI framing ('the model is Responsible but never Accountable... Accountability always rests with a human... That is the operational meaning of meaningful oversight') treats naming a human as accountable as sufficient to satisfy 'meaningful human oversight,' but the study's own recall-null result shows that human reviewers, even with a baseline, caught only about half of real drifts (mean recall 0.525) after spending 48 minutes doing so — evidence that the accountable human's actual detection capacity is limited, which sits in tension with using that same human's formal accountability as proof that oversight is substantively meaningful rather than nominal.
  Read next: Cross-read against literature on rubber-stamp or nominal accountability in human-in-the-loop systems (automation bias, 'moral crumple zones') to weigh whether formal RACI accountability without adequate detection capacity constitutes real oversight or just a liability-assignment ritual.
• incompleteThe targeting rule ('reach for governance on regulated, high-stakes, long-lived systems... skip it for throwaway scripts') is generalized from a single task family — one complex banking-service task compared across exactly one weak and one strong model — without testing other high-stakes domains (healthcare, aviation, safety-critical embedded systems) or a range of model capability levels, so the rule's generality across the broad 'hard, multi-constraint' category it is meant to cover is unverified.
  Read next: Look for replications of the weak-model-vs-strong-model spec-discipline gain across other multi-constraint domains and a wider spread of model capability before treating the 21-point/2-point contrast as a stable, generalizable pattern rather than one data point.
• incompleteThe article proposes an LLM pre-screen as the obvious next automation step ('the most automatable part of the loop... an LLM pre-screen that attributes likely drifts and routes them for regeneration before a human looks'), but never addresses that the pre-screen's triage decision — which drifts are 'likely' enough to route to a human — is itself an accountability-relevant decision the five-control-point/RACI model does not name or assign an accountable human to.
  Read next: Extend the RACI table explicitly to the proposed LLM pre-screen step (phase 5 of the staged-adoption Table 4) and ask who is Accountable for triage/routing decisions the pre-screen makes before a human ever sees the case.
• uninformedThe controlled experiment — does giving a reviewer a reference document (vs. none) change defect-detection recall on code being inspected? — is exactly the setup studied for decades in software-inspection research (Fagan's original inspection studies; Porter, Votta, and Basili's controlled experiments comparing ad-hoc, checklist-based, and scenario-based code reading), where checklist-vs-ad-hoc recall nulls are a recurring, previously documented result. The article shows no awareness of this literature, presenting its recall-null finding as a novel, 'honest' surprise rather than engaging with whether it replicates or contradicts an established result.
  Read next: Read Fagan's software inspection work and Porter/Votta/Basili's comparative code-reading-technique experiments to check whether this study's recall-null finding is a known effect being rediscovered, and whether that literature already has a construct comparable to 'attribution.'

Closes gaps elsewhere

• Sam Hatoum's Post · g1 — Hatoum's post assumes AI implementation is reliable enough to close the spec-implement-verify loop in minutes but offers no evidence; this article's generation-side data bears on that assumption in the negative direction for hard tasks — a staged spec-then-generate approach reached only 45% pass on a 20-invariant banking task even for the winning arm, with the other arms lower still, though gains were much larger and near-ceiling on single-function tasks. It qualifies rather than confirms Hatoum's optimism, and does not directly measure debugging/rework cycle counts.
• Sam Hatoum's Post · g2 — Hatoum's piece leaves the granularity, formalism, and format of a 'spec' entirely unspecified; this article supplies a concrete three-layer format (specification, HLD, LLD) with a worked banking example and a twenty-invariant baseline, directly filling that gap for one domain — though it is offered as one worked example, not a universal template.

Related documents

• Sam Hatoum's Post
• From AI Code to Trusted Software: Harness Engineering in Practice
• Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture
  Code Was Never the Bottleneck Spec-Driven Development Verifiability (as the Quality Bar)
• Fragments: February 18
  Code Was Never the Bottleneck Risk Tiering Spec-Driven Development
• Brownfield Agentic Engineering
  Agent Work Ledger Risk Tiering Verifiability (as the Quality Bar)
• We've Stopped Arguing About Whether AI Changes Software
  Code Was Never the Bottleneck Spec-Driven Development Verifiability (as the Quality Bar)
• Nils Brede Moe's Post
  Code Was Never the Bottleneck Spec-Driven Development
• Finding Comfort in the Uncertainty
  Agent Work Ledger Code Was Never the Bottleneck
• Podcast: Formal Methods for Every Engineer in an AI-Powered Future
  Code Was Never the Bottleneck Verifiability (as the Quality Bar)
• How we Claude Code
  Spec-Driven Development Verifiability (as the Quality Bar)