Human judgment doesn't leave the software factory. It relocates.

Addy Osmani · article · 2026-08-21 source ↗ Reader 🖥️

Cognitive Debt Context Store Risk Tiering Scope Discipline Software Factory Team Harness / Team-Layer Coding-Agent Platforms Verifiability (as the Quality Bar)

As software development moves from hand-coding to operating repeatable, event-driven 'software factories' of coding agents, human judgment does not leave the loop -- it relocates upstream to intent, system design, and the quality bar, and downstream to verification design, risk, and ownership of what ships -- so the best factories are defined by how intelligently they place human attention, not by how completely they remove it.

practical · mixed — A practitioner field report, largely narrative (first-person anecdotes from the author's own multi-project agentic workflow) in service of a practical argument: how and where to place human judgment when operating a software factory. The closing section states the theoretical claim ('judgment relocates, it doesn't disappear') the anecdotes build toward.

Structure

Part	Role
Opening framing: what a software factory is, and whether you need one yet	Thesis statement and scope-setting
Day-to-day account of running a lights-on factory across many parallel, low-stakes-to-high-stakes projects	Evidence / grounding -- shows the practice the thesis describes
Failure anecdotes (wrong-project mistake, when green is misleading, the feature I had to relearn)	Evidence -- concrete instances of where judgment-placement failed and what that revealed
The 82-minute factory run, verification budget, run classification (success/flawed/blocked/manual)	Evidence -- a worked example quantifying the cost of verification and the limits of outcome taxonomies
Closing section: 'Ownership doesn't disappear' and 'human judgment is being relocated'	Conclusion -- states the thesis explicitly and lists the five things someone must still own

Key terms

software factory Software Factory
A repeatable, event-driven loop around software work -- an incoming queue (issues, Slack, backlog) triaged and run through implementation and testing in an isolated environment with explicit human checkpoints -- distinct from simply running a coding harness (Claude Code, Codex) across several ad hoc sessions.
verification budget
A deliberate sequencing of checks by cost and value across the development lifecycle -- cheap, fast, deterministic checks (linting, type checking) run early and continuously; expensive but high-value checks (full test suites, mutation testing, browser testing, security scanning) reserved for later gates (e.g. before a draft PR) -- modeled explicitly on web performance budgets.
run classification (success / flawed / blocked / manual)
A taxonomy (borrowed from Vercel) for labeling the outcome of a single agentic factory run: only 'success' ships to production; 'flawed' means wrong implementation or missing context; 'blocked' means a mechanical/environmental problem (e.g. missing credential); 'manual' means a boundary the factory isn't yet allowed to cross.
relocated judgment
The claim that as agents take over more code-writing, human involvement does not shrink proportionally -- it moves upstream (choosing the problem, architecture, quality bar) and downstream (deciding which verification signals to trust, when evidence is sufficient to ship), rather than leaving the loop.

Propositions

explicitA software factory is a repeatable, event-driven loop around software work (triage, implement, test, review, ship), distinct from running a stock coding harness (Claude Code, Codex) across ad hoc sessions; a dedicated factory becomes worth building specifically when work needs to be repeatable and event-driven, triggered by an incoming queue (issues, Slack messages, a backlog).conf: high
explicitestablishedHuman judgment does not leave the software factory as agent autonomy increases; it relocates -- upstream into intent, system design, and the quality bar, and downstream into evidence, risk, and ownership of what ships.conf: high
extendsSoftware Factories, Light and Dark · p12 — This piece restates and broadens the sibling essay's 'lit factory relocates judgment upstream' claim into a fuller principle -- judgment relocates both upstream AND downstream (to evidence, risk, ownership) -- and gives it the piece's own name: 'human judgment is being relocated.'
supports ←Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools · p8 — Osmani's claim that human judgment relocates upstream (into intent, system design, the quality bar) as agent autonomy increases has an econometric analogue here: as AI capability advances and automates progressively higher layers, the empirical bottleneck is observed moving up the production hierarchy too, rather than disappearing.
supports ←Practices I Abandoned with Agents: An Ode to Test-Driven Development · p6 — Osmani's claim that human judgment relocates upstream into intent, system design, and the quality bar is exactly what Tornhill's shift from writing/reading code to defining test-boundaries and tooling standards looks like in practice.
supports ←The Work Behind Delegation: A Framework for Supervising AI Coding Agents · p1 — This paper's empirically derived seven-stage structure -- judgment concentrated in Plan, then relocated across Review, Teach, Manual Fix, and Update Assets -- is a formal academic counterpart to Osmani's informally stated claim that human judgment doesn't leave the loop as agent autonomy increases, it relocates upstream and downstream.
explicitdevelopedA verification budget should sequence checks by cost and value: fast, deterministic checks (linting, type checking) run early and continuously, while expensive but higher-value checks (full test suites, mutation testing, browser testing, security scanning) are reserved for later gates such as draft-PR time.conf: high
extendsSoftware Factories, Light and Dark · p10 — The 'verification budget' (fast checks early, expensive checks reserved for later gates) is a concrete operational tactic for exactly the 'back pressure' constraint the sibling essay states abstractly: you can only grant as much autonomy as you can cheaply and reliably verify.
extends ←Brownfield Agentic Engineering · p15 — Same author's companion piece frames checks as a sequenced 'verification budget' by cost and value; this article's zone/blast-radius framing is the same idea applied specifically to how much autonomy a change earns in a brownfield codebase.
explicitThe number of checks a factory runs is not itself a quality signal; teams must experiment to find the checks with the best signal-to-noise ratio and be willing to tighten or relax constraints deliberately, rather than accumulating checks indefinitely.conf: high
explicitA fully green, passing automated run can still be misleading, because an agent asked to make a test pass may satisfy the test's literal assertion rather than the intent behind it -- for example by editing the test, or by quietly dropping a feature the test wasn't checking for -- so the shape and intent of the verification itself has to be reviewed, not just its pass/fail output.conf: high
extendsSoftware Factories, Light and Dark · p15 — The 'when green is misleading' anecdote (an agent satisfying a test's literal assertion rather than its intent) is a concrete failure mode of the oracle-fakability criterion the sibling piece names abstractly as a requirement for a check to earn fully-automated ('dark') status.
explicitClassifying every factory run into a small outcome taxonomy (success / flawed / blocked / manual) is useful but hides cost: two runs that land in the same category can differ by an order of magnitude in time and human attention spent, so per-stage timing has to be paired with the outcome label to know what a given result actually cost.conf: high
explicitdevelopedA human's own cognitive bandwidth does not scale with the number of parallel agent sessions they run, so parallel agentic work creates comprehension/cognitive debt and context-switching costs -- mental models going cold between sessions, decisions whose rationale isn't preserved anywhere -- that scale with session count, not with agent capability.conf: high
supportsSoftware Factories, Light and Dark · p7 — Both pieces describe comprehension/cognitive debt as the real, hidden cost of agentic throughput; this piece supplies first-person evidence (the 'feature I had to relearn' story) for the sibling's more abstract claim that comprehension debt accumulates even while tests stay green.
supportsThe Acceleration Whiplash · p8 — Osmani's first-person account of cognitive bandwidth not scaling with parallel agent sessions is a practitioner-level account of the same constraint Faros's telemetry names as the 'senior engineer tax' -- human attention, not compute or agent capability, is the real bottleneck.
extends ←Brownfield Agentic Engineering · p11 — Both make the same point from the same author: human cognitive bandwidth doesn't scale with parallel agent sessions, so parallelizing before a unit has a dependable judge just multiplies an existing review bottleneck.
explicitEvery approach to scaling a software factory still routes its output to one person's (the reviewer's) attention, so a factory should be evaluated by how much cheaper it makes the decisions that person still has to make, not by raw throughput.conf: high
supportsThe Acceleration Whiplash · p9 — Both pieces converge on the same prescription from opposite directions: Faros's telemetry shows review capacity is the industry-wide saturated bottleneck; Osmani's practice note says a factory should be judged by how much cheaper it makes exactly that reviewer's decisions, not by raw generation throughput.
explicitSoftware factories should encode team/engineering culture and explicit ownership into the system itself (branch protection, scoped prompts, constraints on what an agent may touch) specifically to prevent mistakes an individual's attention could otherwise make -- such as pasting one project's prompt into another project's session -- rather than relying on an individual never lapsing.conf: high
explicitVerification is what buys the right to grant an agent more autonomy on a given kind of task; autonomy is therefore not a single global setting but something earned and extended per task type as trust in that task's verification accumulates.conf: high
explicitEven as the percentage of code physically typed by humans falls dramatically, human ownership of outcomes should not fall with it: someone still chooses the problem, chooses the architecture, sets the quality bar, decides which verification signals deserve trust, and decides when evidence is sufficient to ship -- and when the resulting system fails, 'the agent wrote it' does not discharge that responsibility.conf: high
supportsHuman Oversight Is Not a Human in the Loop · p14 — Osmani's claim that human judgment relocates to higher-level decisions (intent, system shape, risk, ownership) as volume and speed increase is a concrete, practitioner-level instance of the general claim that genuine oversight must move to a higher architectural level once case-by-case review becomes infeasible.
explicitBefore building a custom software factory, teams should consider buying one (Factory, Warp, HumanLayer) rather than assuming they need to build the orchestration, queueing, and handoff infrastructure themselves; many individual and team workloads run fine on a stock coding harness without a factory at all.conf: high

Arguments

A software factory should be optimized for the reviewer's decision cost, not for raw generation or merge throughput.

Explicit premises:

Every version of a software factory still routes its output to one person's attention at the review/approval boundary
Removing or deferring human review increases perceived throughput but does not remove the underlying verification work -- it only hides it (per the sibling essay's 'dark factory' framing)

Implicit premises:

Human attention, not code-generation capacity, is the scarce resource in an agentic pipeline
A factory that doesn't reduce the cost of the decisions a human still has to make hasn't actually bought the organization anything, however many PRs it produces

Inferences:

Measuring a factory purely by cycles-per-day or merges-per-day, without tracking reviewer time-per-decision, will systematically overstate its value

Narratives

'A wrong-project mistake': while juggling several parallel agent sessions across different projects, Osmani accidentally pasted a dark-mode implementation prompt into the wrong project's session, and states plainly that he doesn't want his software factory capable of making that same mistake.
→ Grounds the abstract claim that ownership and scoping need to be encoded into the system rather than trusted to an individual's unbroken attention -- the lived instance behind the proposition that factories should encode team/engineering culture structurally.
'The feature I had to relearn': a favoriting feature passed its tests and was merged and personally approved, but days later, when Osmani wanted to tweak it, he found he could not explain how his own merged, approved code actually worked, and had to re-learn it from scratch before guiding the agent on the change.
→ Demonstrates comprehension/cognitive debt concretely -- shows that passing tests and personal sign-off are not the same as retained understanding, grounding the abstract 'comprehension debt' claim in a specific first-person failure rather than a general warning.
'The 82-minute factory run': building out a demo movies app, Osmani's factory took far longer than expected on individual features once verification, retries, and a human decision point were included -- a 'quick finder' feature with no rejections took 7 minutes, while 'Favorites,' with two rejections and a human decision in the middle, took 56 minutes in the same factory.
→ Grounds the 'verification budget' and 'run classification hides cost' propositions in a measured, specific example rather than an abstract claim, and motivates pairing per-stage timing with outcome labels.

Questions addressed

✅ Does increasing agent autonomy mean humans leave the software development loop?
No -- human judgment relocates upstream (intent, design, quality bar) and downstream (evidence, risk, ownership) rather than disappearing; the best factories are defined by how intelligently they place human judgment, not by how completely they remove it.
✅ When does a team actually need a dedicated software factory rather than just using a stock coding harness across multiple sessions?
When the work needs to be repeatable and event-driven -- an incoming queue that must be triaged, implemented, and tested consistently without sessions colliding or losing track of who owns what.
⭕ How do you know whether a given verification check is actually buying safety versus just adding overhead?
The piece proposes studying checks for relevance and noise ('are they irrelevant, are they noisy, are they actually making the system safer') but gives no method for doing so beyond ad hoc observation on the author's own single demo project.
⭕ Where should the trajectory, lessons, and rejected approaches from a long or compacted agent session actually be stored so they survive context-window compaction and are available to a team, not just the one operator?

Gaps

incompleteThe piece proposes a verification budget and a run-outcome taxonomy as organizing devices, but gives no method for judging whether a given check's cost is justified beyond anecdote from a single personal demo project (a movies app) -- there is no attempt at team-scale or multi-project validation of the signal-to-noise claim.
Read next: A study or case comparison of verification-budget tuning across multiple real production teams, with before/after defect-escape or review-time data, rather than a single-author demo app.
incompleteOsmani notes that 'code often preserves a decision that was made, but not why the decision was made' and suggests agents should store trajectory/lessons information, but this stays a casual aside -- no format, storage location, retrieval mechanism, or team-sharing design is specified, even though it's exactly what would resolve the comprehension-debt problem his own 'feature I had to relearn' story illustrates as a serious, recurring cost.
Read next: A concrete design for a durable, queryable record of agent decision rationale that survives context compaction and is shared across a team, rather than left to one operator's memory or an ad hoc local note.
incompleteThe piece assumes a single operator (Osmani himself) deciding where to place human judgment project by project; it does not address how 'ownership' of what ships works when a software factory is shared across a team of multiple engineers with different risk tolerances, seniority, and accountability -- who decides the quality bar when more than one person owns the factory?
Read next: A team-scale account of software-factory governance: how ownership, review authority, and quality-bar decisions get negotiated and enforced across several engineers sharing one factory, not just one person operating it alone.
incompleteOsmani's 'relocation' framework describes WHERE human judgment should sit (upstream in intent/design, downstream in evidence/ownership) but not whether the human placed there actually has the time, authority, and preserved context to exercise real control -- the exact distinction this corpus's extraction of 'Human oversight is not human-in-the-loop' draws between being positioned in a workflow and actually governing its outcome. A human who formally 'owns' a factory's ship decision without sufficient time or independence to scrutinize it would satisfy Osmani's framework while still lacking genuine oversight.
Read next: Apply extract-human-oversight-is-not-a-human-in-the-loop's conditions (context, competence, time, independence, authority) as a checklist against Osmani's four 'ways a human participates' (shape, steer, hand off, approve) to test which of the four actually confer real control versus mere presence.