Ironies of Automation

Bainbridge's paradox intensified by AI: the more capable the automation, the more crucial — and more deskilled — the human operator becomes, especially in incidents.

Also known as: Ironies of automation, Ironies of AI, joint cognitive system, Joint cognitive system, ETTO / TETO (Efficiency-Thoroughness Trade-Off), ETTO, animacy paradox, joint activity testing, buy time, interpredictability, directed attention

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (9)

• AI Is Undermining Leaders’ Judgment. Here’s What to Do About It. Leonid Sudakov, Nathan Furr · 2026-08-19 source ↗ Reader 🖥️
  As AI-driven decision-support systems make organizations more informationally “intelligent,” they simultaneously train leaders out of original judgment through a three-stage erosion of cognitive offloading, deference to tools, and organizational monoculture — so leaders must deliberately practice structured curiosity and intentional dissent to preserve the independent judgment that produces competitive breakthroughs.
• Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 source ↗ Reader 🖥️
  Progressive collapse — civil engineering's term for a small, localized failure cascading into a system-wide catastrophe — is a productive lens for distributed-systems resilience: mitigation requires reducing hazards, strengthening components, and reducing interconnection together, but each mitigation (redundancy, multi-region, circuit breakers) carries real cost and complexity and can itself introduce new interconnection and new failure modes, so teams must apply these techniques as conscious, business-contextualized trade-offs rather than default best practice.
• Everyone I recommend on AI is optimistic. Except Cory Doctorow. Ernie Svenson · 2026-08-04 Reader 🖥️
  Cory Doctorow deserves to be the one skeptical voice in an otherwise optimistic reading list because his critique is disciplined by tracing where money and accountability actually flow rather than by reflexive contrarianism, and that discipline yields two portable diagnostic tools — centaur vs. reverse centaur, and the accountability sink — for judging whether a given AI tool serves the person using it or the vendor selling it.
• Det skal være vanskelig Erik Børve Rasmussen · 2026-07-10 source ↗ Reader 🖥️
  Difficulty is the mechanism of learning, not a defect in it, so AI — a machine for removing difficulty — destroys learning wherever the process rather than the product was the whole point (school, unlike work); the remedy is therefore to redesign AI and assessment as skill-forcing 'simulators' rather than to ban them.
• AI Is Changing Software Engineering. The Fundamentals Still Matter. Abby Bangser · 2026-07-07 source ↗ Reader 🖥️
  AI is transforming who produces, consumes, and maintains software, but the organisations moving fastest succeed by doubling down on engineering fundamentals — right-sized boundaries, modular abstractions, clear interfaces, and operational discipline — as the foundation that lets humans and AI build together safely and sustainably.
• Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident Sudhakararao Sajja · 2026-07-07 source ↗ Reader 🖥️
  The mechanical, judgment-free first phase of incident response — SSHing in, pulling logs from multiple sources, scanning for known failure patterns, and mapping them to the Four Golden Signals — is repetitive search rather than debugging, so it can be automated into a deliberately read-only, locally-run CLI that hands the responder a structured incident brief in under a minute, reserving the actual diagnosis and any state-changing action for the human.
• Software Design in the Agentic Age: Place Your Bets Mathias Verraes · 2026-07-07 source ↗ Reader 🖥️
  In the agentic age, code-level quality becomes automatable but high-level software design and rigorous (ideally executable) specifications remain decisive human-led work, so under deep uncertainty about AI's payoffs and externalities teams should bet on design rigour, executable specifications, and the ability to revert from agent generation back to human engineering.
• Should You Still Become a Software Engineer in 2026? GitHub VP Jean Lee (interviewer); Scott Hanselman (guest, VP & Member of Technical Staff, Microsoft/GitHub) · 2026-06-25 source ↗ Reader 🖥️
  AI coding agents are the latest in a recurring cycle of developer 'power tools' that change the tools but not the craft or its human accountability, so the real risk of the AI era is not that engineers become obsolete but that the profession fails to deliberately re-engineer its junior-to-senior training pipeline (via something like nursing's preceptor model) before AI hollows out the tasks juniors traditionally learned on.
• Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) J. Paul Reed · 2026-05-22 Reader 🖥️
  The ironies Bainbridge identified in 1983 — that automation makes the human operator more, not less, crucial while simultaneously eroding the skills needed to intervene — persist and intensify under AI, so engineering organizations must treat AI as an efficiency lever that demands deliberate cultivation of human thoroughness, mental models, and coordination practices rather than as a substitute for them.

Propositions touching this concept (6)

By publication month, oldest first (extraction month when no publication date is known).

2026-05

• explicitAutomation and AI possess autonomy and authority but lack directed attention, redirectability, and interpredictability — and these last three are the foundations of coordination.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p8
• implicitAI agents that participate in incident-response channels without explicit identification structurally erode interpredictability and should be banned or labeled.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p22
• implicitDeterminism is a precondition for the human-machine interpredictability that coordination relies on; non-deterministic AI is therefore structurally less coordinable than classical automation, not merely opaque.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p25

2026-07

• explicitestablishedBainbridge's 'Ironies of Automation' (1983) is the governing precedent: the better the automation, the more crucial and difficult the human's role becomes, and unused skills atrophy — as Air France 447 showed 26 years after she described it.
  Det skal være vanskelig · p3
• explicitAs agentic code generation improves, the share of easily-solvable production issues will fall in relative terms even as the absolute volume of production code (and therefore of issues) grows exponentially — leaving humans disproportionately with the most complex, highest-consequence issues (an instance of Bainbridge's Ironies of Automation).
  Software Design in the Agentic Age: Placing Your Bets · p2

2026-08

• explicitcorroboratedAutomation (including AI) removes the routine practice through which human competence is normally sustained while still expecting the human to intervene expertly during rare failures — Bainbridge's 'ironies of automation' — and AI intensifies this because its outputs vary with context and can appear coherent even when built on unsupported assumptions.
  Human Oversight Is Not a Human in the Loop · p4

Tensions

Open gaps (11)

• incompleteThe article asserts that removing humans from the loop removes organisational and product learning but does not specify the mechanism, nor how to preserve that learning while still automating parts of operations.
  AI Is Changing Software Engineering. The Fundamentals Still Matter. — read next: Read on the ironies of automation, joint cognitive systems, and how SRE teams learn from incidents while automating remediation.
• incompleteNo evaluation or false-positive analysis of the pattern detection. The tool surfaces 'likely causes' and 'what the evidence suggests', but there is no accuracy/precision/recall figure, no false-positive-rate discussion, and no account of the downside case — a fast, confident, but wrong brief. This matters acutely for incident automation specifically: the human-factors literature finds the risk is asymmetric — automation that surfaces misleading patterns can make outcomes far worse than no automation, and responders cannot detect in real time which regime they are in — precisely the analysis a single-practitioner build log omits.
  Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident — read next: Read J. Paul Reed on the ironies of automation and AI in incidents (the asymmetric-risk / nurses-study finding), plus eval methodology for automated triage/alerting: precision/recall, false-positive cost, and joint human-AI testing rather than testing the detector in isolation.
• incompleteParker frames the joy gap as greenfield vs. legacy but never examines whether the 'joy' experienced by greenfield developers is sustainable or genuine. Rapid AI-assisted prototyping may produce initial excitement that fades as technical debt accumulates or as developers realize they've lost depth of understanding.
  Podcast: The AI Joy Gap: Why Some Developers Thrive While Others Struggle — read next: Research on skill atrophy in automation-heavy domains (aviation, healthcare); Bainbridge's 'Ironies of Automation' for the classic formulation
• incompleteTreats manager decision-rights over AI-produced output as unproblematic — but if AI generates faster, broader, and (with verification) better output than the manager's review can keep up with, the manager's authority is conventional, not epistemic. The piece never asks whose judgment is actually load-bearing.
  Managers Are Struggling to Keep Up with the AI Productivity Boom — read next: Cathy O'Neil on algorithmic accountability; Bainbridge 'Ironies of Automation' for what happens to operator authority when automation outpaces oversight (Paul Reed extract).
• incompleteThe analogy between Bainbridge's control-room operators and software engineers is taken as given but never defended. Bainbridge studied operators of physical, bounded, regulated, slow-dynamics systems (nuclear plants, chemical processes, aviation cockpits) where (a) the human's role is fundamentally MONITORING-AND-OVERRIDE, (b) the system's correct behavior is specified ex ante, (c) regulators audit the human-machine boundary, and (d) the operator is licensed and trained on simulators. Software engineers are CONSTRUCTORS — they design the system itself, not just operate it; the system's correct behavior is partially undefined; no licensing or regulator polices the boundary; and the 'incident' role is only one slice of the work. Reed's mapping works best for the incident-response slice (which is what his job is) and is weakest for routine development. He never disambiguates these. The argument that 'AI ironies for engineers = Bainbridge ironies for operators' may overgeneralize from the incident-response edge case to the whole of engineering practice.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Read the original Bainbridge 1983 paper directly. Then Hollnagel & Woods on resilience engineering, and David Mindell's 'Our Robots, Ourselves' (history of human-machine collaboration). Pair with Naur's 'Programming as Theory Building' for the contrasting view that software engineering is theory construction, not operation.
• incompleteReed's prescription assumes organizations will reinvest AI-driven efficiency gains in human thoroughness, but the same talk reports that the current pattern is the opposite: 'AI's main impact on my life is more incidents because we're generating more code.' The systems-thinking question of how to design feedback loops that prevent capture of efficiency-as-throughput is left untouched. Without that loop, the 'AI does efficiency, humans do more thoroughness' division is a moral exhortation, not a mechanism.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Donella Meadows on leverage points (especially feedback loops, rules of the system, and goals of the system). John Cutler on institutional overload and on how organizations metabolize efficiency gains. Possibly Christensen on capabilities-as-process — i.e., the org's existing process consumes new capability.
• incompleteThe empirical evidence is thin and uneven. The nurses study (Woods et al., July 2025) is presented as decisive but is a single study, in a different domain (clinical prediction), with a likely small N; the developer study (METR-style) is cited from memory. Reed does not engage the contrary evidence (e.g., positive AI productivity findings from DORA, GitHub Copilot studies) or distinguish task types where AI does and does not pay off. The asymmetric-risk frame is compelling and probably right but is underdetermined by the cited data.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: DORA 2024/2025 AI findings, the Princeton/MIT studies on developer AI productivity, METR's actual published methodology, Woods's published paper (rather than the slide). Also Cal Newport / Bainbridge-skeptics on whether the operator analogy fits knowledge work.
• incompleteThe animacy paradox is introduced as a phenomenological observation but is not connected to design recommendations. If interpretation flips with situational stakes, what design pattern reliably keeps the deterministic-machine interpretation alive in high-tempo contexts? Reed gestures at attribution and explainability but doesn't develop this.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Lucy Suchman 'Plans and Situated Actions'; Dave Woods's papers on 'envisioned worlds' and on coordination breakdowns; possibly the air-traffic-control HCI literature on 'mode confusion'.
• incompleteReed touches the systems-thinking question (juniors who never built the mental model, attrition of senior engineers carrying tacit knowledge) but doesn't push it to its leverage-point: the org's HIRING and TRAINING flows may be where the irony actually lives, not in the engineer's individual relationship to AI. The dynamic he describes is a stock-and-flow problem (the stock of tacit knowledge is depleting faster than it is being replenished) and warrants stock-and-flow analysis.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Meadows again on stocks-and-flows; the National Research Council's literature on knowledge attrition in nuclear and aerospace; and possibly the apprenticeship literature (Lave and Wenger, situated learning).
• illogicalReed conflates 'AI is non-deterministic' with 'AI lacks interpredictability.' These are related but distinct. Humans are non-deterministic and have high interpredictability with their teammates. The reason AI lacks interpredictability is not non-determinism per se but lack of a SHARED MODEL of context, goals, and norms. This matters because it suggests the fix is not 'make AI more deterministic' but 'invest in shared context and norm-conformance' — a different research and product direction.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Herbert Clark on common ground in conversation; the multi-agent systems literature on shared mental models; recent work on AI alignment through context-engineering.
• illogicalThe 'know the basics because the automation will break' argument (Waymo/stick-shift analogy) is asserted as self-evidently true, but is never reconciled with his own opening anecdote mocking the 1989 claim that 'you're not a real programmer without assembler' as mere panic — he doesn't say which fundamentals are actually load-bearing today versus which are the modern equivalent of the assembler gatekeeping he dismisses.
  Should You Still Become a Software Engineer in 2026? GitHub VP — read next: extract-det-skal-vaere-vanskelig or other ironies-of-automation sources for a more rigorous account of which specific skills survive automation versus which atrophy safely.