Human Oversight Is Not a Human in the Loop

Gustavo Bataller · article · 2026-08-27 Reader 🖥️

mixed — primarily theoretical (redefines what 'oversight' means and diagnoses why the folk concept 'human in the loop' fails), building in its second half toward practical design prescriptions (event-triggered escalation, governance-state representations, designing oversight in advance) · argumentative, with expository elements — Structured as a sequence of ~13 thematically titled sections, each introducing one load-bearing concept (a condition of oversight, or a borrowed framework — Sheridan/Parasuraman/Wickens, Bainbridge, Endsley, Elish) and using it to widen the central distinction between presence and control. Almost no concrete case studies or named incidents; the argument proceeds through conceptual analysis, cited academic frameworks, and short illustrative vignettes rather than documented examples.

Structure

Key terms

• human in the loop
  A purely spatial/positional description — where a person sits relative to an automated workflow (a checkpoint, an approval click, a named reviewer) — that says nothing by itself about whether that person has the actual capacity to understand or change the decision. Necessary evidence of formal involvement, not sufficient evidence of control.
• human oversight / meaningful oversight
  An institutional capability jointly produced by context (understanding of the situation), competence, sufficient time and attention, independence, and real, exercisable authority — the demonstrated capacity to actually govern what a system does. Framed explicitly as a capability, not a position on a diagram.
• authority
  The demonstrated, exercisable ability to alter what happens next — delay continuation, demand more evidence, reject an output, change execution conditions, or escalate. Defined functionally: the system must be obliged to respond when the human speaks, not merely permit them to speak.
• governance state (vs. conversation state) Governance State (vs. Conversation State)
  A durable, structured representation of what is currently true about delegated work — current objective, authorized scope, what changed since the last meaningful intervention, which decisions were made under delegated authority and on what evidence, and what uncertainty remains open — as distinct from conversational memory or narrative session history.
• event-triggered oversight (vs. permission-triggered)
  A design principle: human intervention should be invoked at meaningful decision boundaries (new authority required, scope expansion, irreversible consequences, evidentiary insufficiency, value conflict) rather than at every step or every novel-looking occurrence, which produces approval fatigue without adding judgment.
• moral crumple zone Moral Crumple Zone
  (Imported from Madeleine Clare Elish) The place where institutional and moral responsibility for an automated system's failure is absorbed by the human operator, even though meaningful control over the conditions that produced the failure was already distributed elsewhere in the system.

Propositions

• explicitHuman-in-the-loop describes only a workflow configuration (where a person is positioned), while human oversight is an institutional capability (whether that person can actually govern the outcome) — the two are not interchangeable, and organizational practice routinely conflates them.conf: high
• explicitPresence inside a workflow does not guarantee control: a person can be formally 'in the loop' while lacking the information, time, competence, independence, or authority to actually understand or change the decision.conf: high
• explicitMeaningful oversight is jointly produced by several conditions — context/understanding, competence, sufficient time and attention, independence, and real authority — and losing any single one weakens a human's actual capacity to govern even though their formal presence is unchanged.conf: high
• explicitcorroboratedAutomation (including AI) removes the routine practice through which human competence is normally sustained while still expecting the human to intervene expertly during rare failures — Bainbridge's 'ironies of automation' — and AI intensifies this because its outputs vary with context and can appear coherent even when built on unsupported assumptions.conf: high
  supportsDet skal være vanskelig · p3 — Both cite Bainbridge's 1983 'Ironies of Automation' for the same mechanism: better automation → less practice → harder recovery during rare failures.
  supportsDet skal være vanskelig · p16 — Same Bainbridge lineage: the essay's institutional-competence-maintenance point (p15) matches this doc's claim that the most successful automated systems require the largest investment in training humans.
  supportsEveryone I recommend on AI is optimistic. Except Cory Doctorow. · p9 — Same vigilance-decrement/Bainbridge mechanism, here applied to AI-assisted work generally rather than industrial control rooms specifically.
  supports ←The Work Behind Delegation: A Framework for Supervising AI Coding Agents · p11 — Developers periodically redoing a risky function themselves 'just to keep the map in my head' is a grassroots countermeasure to exactly the competence-erosion risk ('ironies of automation') that the oversight paper names as a structural effect of automation removing routine practice.
• explicitAttaching formal responsibility to whichever human approved an output does not prove that human possessed meaningful control over the conditions that produced the outcome (the 'moral crumple zone').conf: high
• explicitAuthority is only real when the system is obliged to change course in response to the human's objection — the ability to comment, question, or disagree without altering the outcome is not control, only an advisory function.conf: high
  refinesSoftware Factories, Light and Dark · p10 — Osmani's back-pressure framing treats verification-boundedness as an engineering constraint on autonomy; Bataller's 'authority must be real' supplies the organizational/human-factors half — the boundary must also be organizationally enforceable, not just technically defined.
• explicitOrganizational incentives (e.g. measuring reviewers primarily on throughput) can make formally granted authority to intervene progressively more costly to exercise, hollowing out its practical value while it remains intact on paper.conf: high
• explicitcorroboratedAs the volume and speed of AI-generated recommendations grows, the same nominal review position produces progressively weaker oversight, because human expertise cannot be scaled the way computing capacity can.conf: high
  supportsOwn the Outer Loop · p11 — Both argue that human oversight capacity, not verification technology, is the actual bottleneck constraining how much autonomy can safely be granted.
  supports ←Loop Engineering: Building Blocks, Adoption, and Impact · p9 — Gives a concrete, agent-era instance of the same claim: as AI-generated volume and speed grow, a nominally-unchanged human review position produces progressively weaker oversight — here, nightly agent PRs outpacing review capacity until the gate becomes a rubber stamp.
  extends ←The Work Behind Delegation: A Framework for Supervising AI Coding Agents · p2 — The paper's finding that developers shift from active Monitor to disengaged Wait as trust in a plan grows is a concrete, practitioner-level instance of the general claim that the same nominal review position produces progressively weaker oversight as the volume and speed of agent-generated work grows.
• explicitLong-running, agent-delegated work requires a durable 'governance state' representation (current objective, authorized scope, what changed, what evidence supports prior decisions, what uncertainty remains open) because AI-generated summaries of 'what happened' can be accurate yet still omit precisely the information that would change a human's decision.conf: high
• explicitcorroboratedA system that depends on human oversight is itself responsible for preserving the context (evidence, assumptions, uncertainty, decision history) needed to make that oversight possible; oversight cannot be treated as solely the supervising human's obligation.conf: high
  extendsAWS Releases Loom, an Open-Source Reference Platform for Governing AI Agents at Enterprise Scale · p8 — AWS Loom's point that human-in-the-loop review should be a first-class platform capability rather than an ad hoc afterthought is a concrete implementation of Bataller's claim that the system itself — not just the supervising human — must be engineered to preserve the conditions for meaningful oversight.
  supports ←The Work Behind Delegation: A Framework for Supervising AI Coding Agents · p7 — The 'Update Assets' stage -- externalizing lessons into persistent files like MISTAKES.md and CLAUDE.md that feed back into future Plan stages -- is a concrete practice-level answer to the theoretical requirement that a system depending on human oversight must itself preserve the context needed to make that oversight possible.
• explicitOversight is more effective when human intervention is triggered by meaningful decision boundaries — new authority required, expanded scope, irreversible consequences, insufficient or contradictory evidence, or conflicting values — than when it is triggered by requiring approval at every step or every novel-looking event.conf: high
  supportsArticle: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules · p4 — Both converge on escalation-by-boundary rather than escalation-by-default; the fitness-functions piece gives concrete triggers (low confidence, judge disagreement, high blast radius) that match Bataller's abstract categories (insufficient evidence, irreversibility, value conflict).
  supportsStop Automating Old Processes. Design New Ones Instead. · p11 — Both treat irreversibility specifically as a principled escalation boundary — a default 'Approve' gate before execution — rather than as one input into a blanket approval requirement.
  extendsResearch: How AI Agents Broaden the Scope of Knowledge Work · p11 — The HBR piece's prescriptive leader moves (an explicit human commit boundary for consequential/irreversible actions, routing by complexity/verifiability) are a practical instantiation of Bataller's event-triggered-oversight design principle.
• explicitIndependent review works specifically because the reviewer's incentives are not fully aligned with the process under review; when a reviewer is measured by the same success metrics as the system being reviewed (e.g. throughput), formal independence does not reliably translate into real critical challenge.conf: high
• explicitResponsibility for a failure should be mapped to wherever meaningful control over the system's conditions actually resided — which may implicate several actors — rather than defaulted to whichever human performed the final approval step.conf: high
• explicitcorroboratedWhen decisions occur at a pace or scale that makes serious case-by-case human review impossible, genuine oversight should move to a higher level of the system's architecture (objectives, authority boundaries, monitoring, reversibility, exception handling) rather than being preserved as a fiction of per-decision review — and this is a stronger, not weaker, form of human oversight.conf: high
  supportsAgentic Code Review · p12 — Bataller's architectural principle — move human judgment to a higher level when case-by-case review becomes infeasible, without that being a retreat from oversight — is the general claim of which 'human on the loop' code review (sampling/auditing rather than reading every diff) is a concrete instance.
  supports ←Human judgment doesn't leave the software factory. It relocates. · p11 — Osmani's claim that human judgment relocates to higher-level decisions (intent, system shape, risk, ownership) as volume and speed increase is a concrete, practitioner-level instance of the general claim that genuine oversight must move to a higher architectural level once case-by-case review becomes infeasible.
• explicitBecause automation removes the routine practice through which competence was sustained, institutions — not just individuals — bear responsibility for deliberately preserving the conditions under which exceptional-case competence continues to develop as work is redistributed to AI.conf: high
  supports11) AI-spesial med Vilde Opsal og Anders Sveen · p23 — Both identify the same durable bottleneck — evaluative judgment built through routine practice — as what erodes when AI absorbs the routine work that previously built and exercised it.
• explicitMeaningful oversight cannot be added retroactively as an emergency feature after an automated system has already been designed and deployed; the organization must decide in advance what forms of judgment the system requires and preserve the conditions under which those judgments remain possible.conf: high

Arguments

Narratives

• A composite set of four brief vignettes: a specialist who understands the problem but lacks the authority to intervene; a manager who has authority but receives too many automated recommendations to examine seriously; a reviewer with time and competence who sees only the final answer produced by the same system they must challenge; and someone technically free to disagree while working inside incentives that make disagreement professionally costly.
  → Illustrates that each of the five oversight conditions (context, competence, time, independence, authority) can fail independently while a human remains formally 'involved,' grounding the abstract taxonomy in concrete role failure modes.
• An organization begins by using an automated system to produce a small number of recommendations that a specialist reviews carefully; as trust and adoption grow, the same specialist is expected to review hundreds of outputs at the pace that once required a much larger team — with no formal change to their governance role.
  → Shows how time/attention erodes real oversight capacity through organic scale creep, without any deliberate governance decision ever being made — illustrating why 'nothing changed on paper' does not mean oversight is unchanged.
• A reviewer formally authorized to stop an automated process, but whose performance is measured primarily through throughput; every intervention reduces completed cases, creates extra work, and requires explanation to managers whose own objectives depend on the system continuing to run.
  → Demonstrates how organizational incentive design can nullify independence and authority even when both are formally intact, without requiring bad faith on anyone's part.

Questions addressed

• ✅ Why does 'human in the loop' fail as a safety/governance guarantee?
  Because presence in a workflow does not entail the context, competence, time, independence, and authority jointly needed for actual control; any one can be absent while formal presence persists.
• ✅ What does genuine, meaningful human oversight actually require?
  A jointly necessary set of conditions — context, competence, time/attention, independence, and real exercisable authority — framed as an institutional capability rather than a position in a workflow.
• ✅ Why does automation (including AI) make oversight harder over time, even when nothing in the formal governance structure changes?
  Bainbridge's ironies of automation (competence atrophies as routine practice is delegated away) plus volume/speed effects that outpace human review capacity, intensified by AI outputs that vary with context and can look coherent while resting on unsupported assumptions.
• ✅ How should responsibility/accountability be assigned when an AI-assisted decision goes wrong?
  Map responsibility to wherever meaningful control over the system's conditions actually resided — potentially several actors — rather than defaulting to whoever performed the final approval (avoiding the 'moral crumple zone').
• ✅ When, concretely, should an autonomous agent escalate to a human rather than continuing on its own?
  At meaningful decision boundaries: when new authority would be required, scope would expand, a consequence becomes difficult to reverse, evidence is insufficient or contradictory, or values conflict — not at every step or every novel-looking event.
• ⭕ How does an organization concretely build and maintain the 'governance state' needed for cognitive continuity across long-running agentic work?
  The essay defines what governance state must contain (objective, scope, what changed, evidence, open uncertainty) but does not specify how such a representation is constructed, validated, kept accurate, or protected from being gamed by the agent whose work it documents.

Gaps

• incompleteThe essay introduces 'governance state' as the solution to cognitive continuity across long-running agentic work but never specifies how such a representation is actually constructed, validated, kept accurate over time, or protected from being gamed by the very agent whose actions it is meant to make legible. It stays entirely at the level of requirements ('what the human needs to know'), not implementation ('how the system produces and guarantees that').
  Read next: Read alongside this corpus's decision-provenance material — e.g. tracking idea→spec→PR→production, or 'mirror' log concepts — for concrete architectural proposals that might fill this gap.
• incomplete'Authority' is defined functionally (the system must be obliged to respond) but the essay never distinguishes how that obligation is actually enforced — a technical control (the agent literally cannot proceed without an unblocking signal), a policy/process control, or a cultural/incentive norm. These have very different failure modes, yet 'real authority' is treated as roughly binary rather than as a spectrum of enforcement mechanisms with different robustness.
  Read next: Read alongside back-pressure/verification-bound-autonomy material that treats authority boundaries as engineered, mechanical constraints ('only as much autonomy as you can cheaply and reliably verify') rather than primarily an organizational/cultural property.
• incompleteNo engagement with how an outside party (a regulator, auditor, or customer) could verify — as opposed to a human simply asserting — that an organization's oversight is meaningful rather than ceremonial. The essay's diagnostic vocabulary (context/competence/time/independence/authority) is entirely internal; it gives no external verification method, which matters directly for the regulatory regime cited in its own bibliography (the EU AI Act) that requires organizations to demonstrate human oversight to a third party.
  Read next: Read against material on EU AI Act compliance and observability/audit-trail limits (e.g. why OpenTelemetry-style traces are necessary but not sufficient to demonstrate human oversight) to see whether any of this corpus's audit tooling closes the gap between internal capability and external proof.
• incompleteThe illustrative failure-mode vignettes (specialist without authority, overloaded manager, reviewer seeing only the final answer, someone discouraged from disagreeing) are presented as plausible scenarios rather than drawn from documented incidents. No specific, named case of an AI-oversight failure grounds the argument, which weakens falsifiability — a reader cannot check whether the five-condition taxonomy actually predicts or explains a known real failure.
  Read next: Compare against concrete incident write-ups elsewhere in the corpus (e.g. production-database-deletion incidents where an agent operated without an approval gate, or major automated-decision failures with public postmortems) to test whether Bataller's framework actually accounts for those specific failures.

Closes gaps elsewhere

• Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident · g3 — Bataller's Bainbridge-irony argument (p4) explains precisely why delegating routine first-pass diagnostic work erodes the intuition needed for rare/novel failures, and the institutional-competence-maintenance claim (p15) extends it: organizations must deliberately design to preserve practice opportunities rather than assume the deskilling risk resolves itself.
• Developer Newsletter: higher usage limits, dreaming for Claude Managed Agents, and Claude Security · g6 — Bataller's 'Time is part of authority' section (p8) directly supplies the human-factors substrate the newsletter piece leaves unexamined: human expertise and attention cannot scale the way computing capacity can, so nominal reviewer/fleet-supervisor authority weakens as review volume and concurrency grow even without any formal governance change.
• Google Introduces Middleware Architecture for Genkit Applications · g5 — Bataller's essay is precisely an unpacking of what 'a human-in-the-loop is available' requires beyond mere availability — context, competence, time, independence, and enforceable authority (p1-p3, p6) — supplying the deeper analysis the Genkit piece flags as an unexamined design assumption.
• This Week in AI: When agents outnumber people · g1 — Same mechanism as above (p4, p15), applied generally rather than to incident response specifically: foundational/evaluative knowledge is sustained by routine practice, which is exactly what gets delegated to AI first, so 'foundational knowledge lets people recognize a wrong result' cannot be assumed to persist without deliberate institutional preservation of practice.

Related documents

• 11) AI-spesial med Vilde Opsal og Anders Sveen
• Own the Outer Loop
• Everyone I recommend on AI is optimistic. Except Cory Doctorow.
• Det skal være vanskelig
• Agentic Code Review
• Your AI agent just blamed the network team. Now what?
  Agent Governance (Internal) Human-in-the-Loop Risk Tiering
• AWS Launches Amazon GuardDuty Investigation Agent to Automate Threat Triage
  Agent Governance (Internal) Agentic AI Human-in-the-Loop
• Stop Automating Old Processes. Design New Ones Instead.
  Agent Governance (Internal) Human-in-the-Loop Risk Tiering
• This Week in AI: When agents outnumber people
  Agentic AI Capacity to Verify AI Output Human-in-the-Loop
• 5 Signs That Authenticated API Traffic Is Actually Malicious
  Agent Governance (Internal) Agentic AI