SRE & Incident Response

Operating and learning from production incidents — reliability engineering practice and its human factors.

Also known as: incident response, site reliability engineering, SRE, incident management

Documents (22)

• AI Development Patterns Paul Duvall · 2026-10-08 source ↗ Reader 🖥️
  Organizations should adopt this catalog's AI-development patterns selectively and in an order diagnosed from their own team size, project type, and technology stack — not uniformly or in catalog order — because the coordination problems a solo developer needs to solve (rapid iteration, tool-matching) are categorically different from, and the patterns designed for them can actively mismatch, what a two-pizza team or a multi-team organization needs (parallel-agent isolation and conflict resolution, cross-team traceability, automated review of agent output at volume).
• Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 Reader 🖥️
  Durable workflow execution is fundamentally a database-checkpointing problem, so a team that already operates a relational database like Postgres at scale can implement production-grade durable execution — concurrent dispatch, idempotency, crash recovery, durable waits, and observability — as a handful of well-understood SQL primitives, without deploying a separate stateful orchestrator such as Temporal or AWS Step Functions.
• Presentation: How To Run on Three Clouds at Once, and When Not To Ross McFarlane, Kevin Holditch · 2026-09-11 source ↗ Reader 🖥️
  Form3 proves that a regulated payments platform can run truly active-active-active across three cloud providers by treating each cloud as nothing more than another availability zone and building custom Kubernetes operators to extend the platform's own primitives across cluster boundaries where they run out — but the same team's parallel US disaster-recovery build shows that this triple-active design is not a universal best practice: it is justified only when a specific market's resilience expectations, a mature platform-engineering practice, and a willingness to pay a real cost premium all line up, and even then the harder problem — deciding, mid-incident, whether to actually trigger a failover — remains an unresolved judgment call under uncertainty rather than something engineering can fully automate away.
• Operating Mode as Runtime State: A Contract for Enterprise Shreshta Shyamsundar and Chidambaram GS · 2026-09-11 source ↗ Reader 🖥️
  Because enterprise AI agents infer whether they're operating under normal, incident, or recovery conditions from prompts and accumulated context rather than receiving it as authoritative input, temporary emergency exceptions quietly harden into permanent runtime behavior ('exception drift') — the fix is to make organizational operating mode a first-class, scoped, time-bound runtime construct, injected alongside identity and permissions, whose closure the platform itself can verify rather than merely document.
• Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments Salim Adedeji · 2026-09-08 source ↗ Reader 🖥️
  In regulated, stateful payment systems running on ECS, chaos engineering only works if teams first replace the standard stateless-web playbook's three violated assumptions — clean rollback, definable blast radius, and free production experimentation — with a compliance-integrated, four-stage blast-radius graduation, because it is measured behavior under actual injected faults, not configuration values or architecture review, that reveals where these systems actually break.
• How Figma Uses AI Agents for Security Renato Losio · 2026-09-06 source ↗ Reader 🖥️
  Figma's security engineering team, as relayed by InfoQ from the company's own blog posts, holds that AI agents equipped with persistent, separated memory, tightly scoped tools, and strict human-review guardrails (draft-by-default PRs, anti-leak prompts) can absorb much of a security team's repetitive investigative and detection work -- reporting roughly 70% faster complex-alert resolution, 20% fewer on-call pages, 100+ newly found vulnerabilities, and 80% code-reviewer precision -- while insisting that the right balance between agent autonomy and human oversight remains an open, evolving question rather than a solved one.
• Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 source ↗ Reader 🖥️
  Progressive collapse — civil engineering's term for a small, localized failure cascading into a system-wide catastrophe — is a productive lens for distributed-systems resilience: mitigation requires reducing hazards, strengthening components, and reducing interconnection together, but each mitigation (redundancy, multi-region, circuit breakers) carries real cost and complexity and can itself introduce new interconnection and new failure modes, so teams must apply these techniques as conscious, business-contextualized trade-offs rather than default best practice.
• Grab Cuts Mechanical Analytics Work From 44% to 30% with AI Agents Leela Kumili · 2026-08-17 source ↗ Reader 🖥️
  Grab reduced the mechanical share of its analysts' work by deploying AI agents across a staged autonomy model, an agent-routing system, and continuously maintained certified data context, while deliberately keeping humans accountable for metric definitions, causal interpretation, and final decisions.
• AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act Steef-Jan Wiggers · 2026-07-27 source ↗ Reader 🖥️
  AWS's July 2026 billing-estimate incident -- in which a single unit-pricing bug displayed trillion-dollar phantom bills to customers for over 24 hours while AWS's own cost-anomaly alarms detected the anomaly but failed to halt the pipeline, alert engineers, or trigger any automated safeguard -- shows that billing telemetry is itself a fallible system dependency whose failure modes propagate into every automated cost control built on top of it, exposing that AWS still gives customers no way to convert an alert into a hard spending limit.
• AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering Mark Silvester · 2026-07-25 Reader 🖥️
  The hard part of AI-assisted root cause analysis has shifted from model reasoning -- which recent evidence suggests is largely solved by frontier and even many open-weight models given the same prepared context -- to context engineering: the harness work of correlating signals and assembling a compact, unmisleading context before the model is ever called.
• Stranded in the Slow Zone Tim O'Reilly · 2026-07-24 source ↗ Reader 🖥️
  Frontier AI agents now let individual builders create personal systems with the complexity and fragility of enterprise software, so the DevOps discipline that once belonged only to large organizations — documentation that outlives any one model, and tool portability rather than model-agnostic architecture or self-hosting — has become a personal survival skill, even as the same shift turns engineers into managers of AI 'bots' and raises the same craft-versus-slop question for writing that it already raised for code.
• Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation Leela Kumili · 2026-07-23 source ↗ Reader 🖥️
  Expedia built STAR, an internal AI-assisted observability tool, on the premise that a deterministic, human-validated, prompt-chained workflow over standardized infrastructure telemetry -- deliberately forgoing function calling, RAG, memory and autonomous tool use -- is the right way to accelerate production-incident root-cause investigation today, with engineers remaining the final validators of every finding.
• DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability Leela Kumili · 2026-07-20 Reader 🖥️
  DoorDash's Entity Cache shows that moving caching out of individual services and into a transparent, infrastructure-layer proxy (Envoy + Valkey) — with event-driven invalidation, staleness-tolerant failure handling, and stampede-avoidance techniques — can absorb very large request volume at extremely high availability without requiring any application code changes.
• What is “loop engineering?” The Pragmatic Engineer · 2026-07-14 Reader 🖥️
  “Loop engineering” — designing the orchestration around an AI coding agent (goals, completion checks, state persistence, fresh-context iteration) rather than hand-writing one-off prompts — is a real practice that originated as a DIY workaround (the “Ralph loop”) for small context windows, but by mid-2026 major harnesses had absorbed most of it into a native /goal primitive, so for most developers besides those building AI infrastructure, it is a largely-already-commoditized skill rather than a durable new discipline.
• OpenAI Fixes 18-Year-Old GNU libunwind Bug by Treating Crash Debugging Like Epidemiology Steef-Jan Wiggers · 2026-07-11 Reader 🖥️
  When production crashes resist every hypothesis, the fastest way to see their structure is not deeper analysis of individual cases but building a complete, labeled dataset across the whole population of failures — a population-level, epidemiological approach that revealed OpenAI had conflated two unrelated bugs (silent CPU corruption on one Azure host and an 18-year-old race condition in GNU libunwind's setcontext) into a single unsolvable mystery.
• AI Is Changing Software Engineering. The Fundamentals Still Matter. Abby Bangser · 2026-07-07 source ↗ Reader 🖥️
  AI is transforming who produces, consumes, and maintains software, but the organisations moving fastest succeed by doubling down on engineering fundamentals — right-sized boundaries, modular abstractions, clear interfaces, and operational discipline — as the foundation that lets humans and AI build together safely and sustainably.
• Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident Sudhakararao Sajja · 2026-07-07 source ↗ Reader 🖥️
  The mechanical, judgment-free first phase of incident response — SSHing in, pulling logs from multiple sources, scanning for known failure patterns, and mapping them to the Four Golden Signals — is repetitive search rather than debugging, so it can be automated into a deliberately read-only, locally-run CLI that hands the responder a structured incident brief in under a minute, reserving the actual diagnosis and any state-changing action for the human.
• 5 Things World-Class Engineers Do That You Don't Steve Huynh · 2026-07-01 source ↗ Reader 🖥️
  Elite engineers are distinguished not by raw intelligence or effort but by five deliberate practices the author claims to have observed repeatedly across podcast conversations with principal/distinguished engineers: measuring growth monthly rather than waiting for reviews, amplifying colleagues instead of maximizing solo output, absorbing complexity so customers experience simplicity, deliberately avoiding the crowded competitive path, and converting personal failures into institutional lessons.
• Your AI agent just blamed the network team. Now what? Nik Kale · 2026-06-15 source ↗ Reader 🖥️
  Deploying autonomous, cross-domain AI diagnostic agents in production incident response introduces a new organizational-political risk -- a confidently-asserted, team-implicating finding that may be contested -- and that risk is best managed not by better AI reasoning but by a deliberately staged trust-building process (investigation-scoped credentials, a read-only start, an auditable reasoning trail, and treating rollout as organizational change with pre-secured cross-team buy-in) that earns autonomy incrementally rather than being engineered away as a purely technical problem.
• How a Culture of Data-Driven Conversations Can Support Platform Engineering Ben Linders · 2026-06-04 Reader 🖥️
  To survive growing cognitive load and operate as a multi-platform team without scaling headcount, a platform org should run SRE as a service through a center of excellence, embed Federated SREs and new bridging roles (production manager, technical tribe lead) into business units, and make SLOs/SLAs a shared organisational currency so that operational ownership is pushed down into product teams rather than centralised in the platform.
• Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) J. Paul Reed · 2026-05-22 Reader 🖥️
  The ironies Bainbridge identified in 1983 — that automation makes the human operator more, not less, crucial while simultaneously eroding the skills needed to intervene — persist and intensify under AI, so engineering organizations must treat AI as an efficiency lever that demands deliberate cultivation of human thoroughness, mental models, and coordination practices rather than as a substitute for them.
• How Claude Code Is Built Gergely Orosz · 2025-09-23 source ↗ Reader 🖥️
  Claude Code is built by a small team that treats the Claude model itself — not its own surrounding code — as the real product, deliberately minimizing scaffolding, choosing an "on-distribution" tech stack the model already masters, deleting code whenever a new model release subsumes it, and restructuring code review, testing, incident response, and prototyping around AI doing the first pass; Orosz presents this as a plausible preview of how "AI-first" engineering teams will operate, while flagging that Anthropic's unlimited, cost-free access to its own frontier model and its identity as a research lab (not a product company) limit how directly the specifics transfer elsewhere.

Propositions touching this concept (10)

By publication month, oldest first (extraction month when no publication date is known).

2025-09

• explicitClaude Code is used for oncall and incident response — triaging incidents by pulling in relevant Slack contacts and Sentry logs, finding root causes, and (with permission) auto-bisecting or reverting changes — and Boris claims it is often faster at finding root causes than a human engineer.
  How Claude Code Is Built · p10

2026

• implicitAdopting Rust requires an organizational shift in where 'design effort' is spent — upstream into types and data structures rather than downstream into runtime defense and incident response.
  Why Rust is different, with Alice Ryhl · p15

2026-05

• implicitcorroboratedSoftware is a safety-critical system class, and the human-factors discipline developed for aviation, medicine, and nuclear control rooms transfers to software incident response.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) · p24

2026-07

• explicitFor teams adding LLMs to incident response, effort spent preparing context pays off more than reaching for a larger model.
  AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering · p2
• explicitAWS's expanded DevOps Agent moves AI validation earlier in the software lifecycle — from post-deployment incident response into pre-merge code review (Release Readiness Review) and pre-production test generation (Autonomous Release Testing).
  AWS Expands DevOps Agent with AI-Powered Release Management to Validate Code Before Production · p1
• explicitPeople over-estimate agent capability for incident response because they mentally model incident resolution as a simple, linear process, when in practice it involves surprises and adaptation.
  Fragments: July 21 · p12
• explicitGoogle recommends a phased rollout (Deploy: baseline controls like Workload Identity and Confidential Nodes; Operate: signed image policies and log aggregation; Govern: organization-wide guardrails and automated incident response) rather than adopting all blueprint controls at once.
  GKE Security Blueprint Joins Growing List of Cloud AI Frameworks · p5

2026-08

• explicitdevelopedMaintenance and incident response can run headless end to end: a deterministic, version-controlled detection script monitors a production metric against a rolling baseline, and a breach invokes Claude without a person in the path to diagnose and write the finding back as an intent.md that re-enters the normal pipeline, with response tiers (log, diagnose, propose) set in advance by breach severity.
  The AI-Native SDLC playbook · p11

2026-09

• explicitClaude Code's current comparative advantage is code implementation; the durable industry-wide direction is turning the rest of the software development lifecycle (review, CI/CD, incident response, feedback triage) into agent-run loops as well.
  How Anthropic Builds And How Engineering Will Change Soon | Thariq Shihipar · p14
• explicitException drift is architectural, not a failure of human incident/change-management process: enterprises already have adequate human machinery (incident management, change control, postincident review), but most agent platforms treat organizational operating state as external to the runtime rather than as an input to it.
  Operating Mode as Runtime State: A Contract for Enterprise · p3

Tensions

Open gaps (48)

• incompleteAll five anecdotes are drawn from a narrow institutional band — Meta, Microsoft, Amazon/AWS, Google, NVIDIA — large, resourced, prestige-laden organizations with internal audiences for cross-team initiatives (a company-wide code health summit), promotion committees that reward 'the story,' and executive-level design reviews (Bezos personally weighing in on S3's interface). The article never examines whether 'measure growth monthly,' 'amplify others,' 'absorb complexity,' 'chase a non-consensus bet,' or 'turn failure into shared infrastructure' are equally available, meaningful, or even advisable for engineers at small companies, understaffed teams, blameful cultures, or organizations without a promotion ladder that rewards these exact behaviors. Point 5 in particular assumes a blameless-postmortem culture (a director who responds to an 8-hour outage by asking 'how do we prevent this' rather than punishing) as a given, when that culture is itself a scarce organizational precondition, not a personal practice a reader can simply adopt.
  5 Things World-Class Engineers Do That You Don't — read next: Literature on organizational preconditions for the behaviors prescribed here — psychological safety and blameless postmortem culture (extract-human-scalability-problem, SRE incident-response literature), and accounts of career advancement outside FAANG-scale organizations — to test whether the five practices travel, or whether they are downstream of already being inside a resourced, high-visibility organization.
• incompleteNo single concrete case study of an actual deployed cross-domain diagnostic agent's finding implicating a team, and how the organizational/political fallout was actually resolved (accepted, disputed, or overridden). The article's formative anecdote is entirely pre-AI and human-only; the 'two years building and deploying' claim is asserted authority without a demonstrated instance of the exact scenario the title promises to address.
  Your AI agent just blamed the network team. Now what? — read next: Look for practitioner post-mortems or case studies (SRE/platform-engineering conference talks, vendor case studies) that document a real agent-attributed root cause, the implicated team's reaction, and how it was resolved -- to test whether 'evidence beats politics' holds under an actual contested finding rather than only in the abstract.
• incompleteNo treatment of what happens when a team disputes a technically sound, evidence-based finding -- the piece assumes the presence of a reasoning trail resolves disputes, but doesn't address disagreement about the interpretation of correctly-gathered evidence (e.g., 'yes, the config change was ours, but the alerting design that mis-routed the page to us is also broken') -- plural, contestable culpability rather than a single clean root cause.
  Your AI agent just blamed the network team. Now what? — read next: Blameless-postmortem and organizational-accountability literature distinguishing causal contribution from culpability.
• incompleteNo consideration that removing political friction and cross-domain investigative labor from human responders may erode their own cross-domain diagnostic skill and organizational relationships over time -- the same automation-complacency concern left unexamined in a companion piece about a narrower (single-domain, deterministic, read-only) incident-copilot tool, and arguably more consequential here since Phase 3 hands over actual diagnosis and remediation, not just search.
  Your AI agent just blamed the network team. Now what? — read next: Deskilling/automation-complacency literature; cross-reference extract-ai-incident-copilot-first-15-minutes#g3.
• incompleteThe article asserts that removing humans from the loop removes organisational and product learning but does not specify the mechanism, nor how to preserve that learning while still automating parts of operations.
  AI Is Changing Software Engineering. The Fundamentals Still Matter. — read next: Read on the ironies of automation, joint cognitive systems, and how SRE teams learn from incidents while automating remediation.
• incompleteNo production evidence that the tool reduces MTTR or improves incident outcomes. The entire benefit case rests on toil removed (15-45 minutes of manual search, 'thousands of engineering hours lost per year') plus the assertion that responders can then 'skip straight to actual problem-solving' — but time-to-brief is not time-to-resolution, and nothing here is a controlled before/after or a field measurement. For a rigorous adopter this is the first missing thing: a build log's plausible mechanism, not demonstrated impact.
  Building an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident — read next: Incident-response outcome measurement: DORA's failed-deployment-recovery / MTTR data, controlled before-after studies of triage-automation on resolution time, and whether pre-assembled context actually shortens the diagnosis phase or just relocates the clock.
• incompleteThe 'reasoning is basically solved' verdict rests on a single experimental scenario (one Chaos Mesh network-delay fault type, with one set of misleading signals the harness designer already anticipated). There is no report of repeated trials of the same scenario (to check consistency), no second incident type, and no test of an incident whose confounds were not anticipated at harness-design time -- exactly the case where the article itself says agent-based designs would have an edge.
  AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering — read next: Look for a multi-scenario or multi-trial replication of Coroot's benchmark, or for evaluation frameworks purpose-built to separate model-reasoning-quality from harness-quality across a broader set of incident types (e.g. AIOps/observability benchmark suites).
• incompleteThe article never explains why AWS's cost-anomaly alarms, despite firing, were not wired to any automatic halt-the-pipeline or page-an-engineer action. Was this an intentional design choice (e.g. to avoid auto-halting on what might be a false positive) or a plain configuration gap nobody had closed? Without that distinction, the reader cannot judge whether this was a one-off oversight or a predictable consequence of how AWS designed alarm response for this system.
  AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act — read next: AWS's own postmortem practices and SRE literature on the design tradeoff between auto-remediation and human-in-the-loop response for anomaly detection; how organizations decide which alarms are allowed to trigger automatic action versus which require a human gate.
• incompleteThe 4.5-hour gap between AWS's alarms firing (7:46 PM) and customer escalations reaching engineering (12:19 AM) is stated but not explained. The article doesn't distinguish 'no on-call routing existed for billing-system alarms' from 'routing existed but the page was missed or deprioritized' -- two very different failure modes with different fixes, and the article leaves the reader unable to tell which happened.
  AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act — read next: AWS's incident-response and on-call/paging design practices; general literature on alert routing and paging discipline for internal-facing systems versus customer-facing ones.
• incompleteThe article frames an earlier, similar unit-pricing bug (fixed in roughly two hours by a paged engineer) as evidence this is a 'familiar' and recurring bug class at AWS, but never asks whether AWS applied any structural fix after that earlier incident -- new end-to-end tests, unit-type validation, schema checks -- that should have caught a recurrence. If such a fix existed and still failed to catch this instance, that is a much stronger indictment of AWS's testing architecture than the article draws out; if no fix was ever applied, that is a different and equally notable finding the piece doesn't pursue.
  AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act — read next: Whether AWS billing/metering teams run and publish cross-incident lessons-learned processes; general SRE literature on incident recurrence and whether post-incident action items actually get implemented.
• incompleteNo framework is given for deciding which of the concrete numeric findings (93s failover, 2.4x DB load amplification, 14,000 stuck transactions) are specific to this author's stack (400 TPS, this JVM version, this HikariCP tuning) versus generalizable. A reader on a non-JVM stack, a different database, or Kubernetes instead of ECS cannot tell how much of the 'what surprised us' material would transfer.
  Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments — read next: Look for a comparable chaos-engineering-in-fintech account on a different runtime/orchestrator (e.g. Kubernetes-based payment platform) to see which findings recur and which are ECS/JVM-specific.
• incompleteThe compliance/change-management approval process is asserted to 'produce an audit trail that satisfies PCI DSS and SOC 2,' but no example of that documentation, approval turnaround time, or an auditor's actual response is shown. How Stage 3/4 experiments get approved in real time, and what happens when a rollback condition triggers mid-experiment under regulatory scrutiny, is left unaddressed.
  Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments
• incompleteThe financial cost/benefit of running the chaos-engineering program itself is only asserted qualitatively (six to twelve months to Stage 4; 'the regulatory cost may exceed the learning value'), with no quantification of engineering-hours invested versus incidents prevented or dollars saved — creating an asymmetry, since the cost of NOT doing chaos engineering is made vivid with a precise seven-figure SLA-penalty figure in the opening narrative, but the cost of doing it is never quantified to compare against.
  Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments
• incompleteNo discussion of which endpoints or data types are eligible for stale-serving during outages versus excluded because staleness would be unsafe (e.g., live pricing, order status, checkout inventory). The failure-handling story treats 'keep serving' as an unqualified win without explaining how DoorDash decides, per endpoint, whether serving stale data is safe.
  DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability — read next: Staleness-classification frameworks and per-endpoint consistency tiering in caching architectures; how large-scale caching platforms (Netflix EVCache, Facebook TAO/McRouter) segment cacheable data by consistency requirement.
• incompleteThe 99.99999% (seven-nines) availability figure has no defined measurement window, no statement of whether it's client-observed or server-measured, no indication of whether 'serve stale during outage' counts as a success or a degraded state within that number, and no comparison to the availability of the upstream services being proxied — so it's unclear whether the cache is actually more available than what it fronts, or the number simply describes Entity Cache's own component uptime in isolation.
  DoorDash Uses Envoy and Valkey for a 1.5M RPS Proxy Cache with 99.99999% Availability — read next: SRE literature on error budgets and availability measurement methodology (e.g., Google SRE book) — how availability figures are actually computed and what 'available' means when a system degrades rather than fails.
• uninformedEvery pattern in the catalog is presented with the author's own prescriptive worked example as its sole evidence; the catalog never cites an external case study, a controlled comparison, or independent practitioner validation for any individual pattern, despite the patterns ranging from simple (constraint-based prompting) to organizationally significant (atomic task decomposition coordinating parallel work across multiple teams).
  AI Development Patterns
• incompleteThe team-size framework (Solo / Two-Pizza / Multi Two-Pizza) treats team size as the organizing variable for pattern adoption but never addresses the transition itself — there is no migration path described from, say, a two-pizza team's Human-AI Handoff Protocol to a multi-team organization's Atomic Task Decomposition, only separate, static recommendations for each size category.
  AI Development Patterns
• incompleteNo quantitative evidence is given that STAR actually reduced TTK or TTR -- the stated objective is never followed by a measured outcome (a duration, a before/after comparison, or even a qualitative estimate). The same absence-of-outcome-metrics pattern recurs in sibling AI-incident-tooling vendor posts already in the corpus (extract-ai-incident-copilot-first-15-minutes#g1, extract-ai-agent-blame-network-team#g4) -- this is the corpus's most consistent gap in AI-for-incident-response tooling.
  Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation — read next: Look for a follow-up post or case study reporting concrete before/after TTK/TTR numbers for STAR, or for independent (non-vendor) benchmarking of LLM-assisted incident-diagnosis tools.
• uninformedNo detail is given on how the 'domain-specific prompts' are engineered, what mitigations exist against LLM hallucination or fabricated root causes in the multi-stage summarization, or what the SME-review/Langfuse evaluation loop actually measures (accuracy? usefulness? both?). The article names the ingredients (Langfuse for prompt management/tracing/evaluation, SME review, user feedback) without describing evaluation criteria or results.
  Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation — read next: Read Expedia's original Medium engineering post (linked in the article) for evaluation methodology details, or look for a follow-up on false-positive/false-negative rates in STAR's generated findings.
• incompleteNo recall or false-negative figures are given for the vulnerability-hunting or code-review agents -- only precision (80%) and detection-improvement percentages are reported, so a reader cannot tell how many real vulnerabilities or bugs the agents miss. This is exactly the blind spot the article's own 'precision before recall' framing would predict if recall improvement is being deliberately deferred.
  How Figma Uses AI Agents for Security — read next: Read Figma's full engineering blog posts (linked from the article) for any recall/false-negative disclosure; compare against extract-multi-agent-ai-production-security-operations#g2, which raises the identical unmeasured false-negative gap for a telco SOC's anomaly-gating layer.
• incompleteNo description of how the alert-triage agent's cost or inference volume is managed at scale -- e.g. whether every alert invokes the LLM agent directly, or whether a cheaper deterministic/heuristic pre-filter (comparable to the anomaly-detection gating layer in a telco SOC architecture elsewhere in the corpus) triages first.
  How Figma Uses AI Agents for Security — read next: Read the full Figma blog post for volume/cost details; compare to extract-multi-agent-ai-production-security-operations#p5's Isolation-Forest pre-filter pattern for gating expensive LLM inference behind cheap anomaly detection.
• illogicalThe US active-standby failover runbook's top decision heuristic requires an accurate estimate of how much longer the ongoing incident will last relative to the predicted failover time — but the talk's own headline anecdote (the October 2025 AWS incident) demonstrates that this exact estimate is unavailable precisely when the decision must be made ('we don't know how long this underlying incident is going to last'). The heuristic is stated as actionable guidance while the narrative that follows shows it could not actually be applied with confidence.
  Presentation: How To Run on Three Clouds at Once, and When Not To — read next: None needed — this is a structural tension worth flagging on its own terms rather than something to chase down externally.
• uninformedThe talk repeatedly asserts that triple-active multi-cloud 'is definitely more expensive' than single-cloud, and that splitting one large production environment into several smaller ones reduces cost, but gives no order-of-magnitude figure for either claim (no multiplier, no percentage, no dollar range), so a reader cannot judge whether the cost tradeoff argument in the 'when not to' section is proportionate advice or an unquantified hand-wave.
  Presentation: How To Run on Three Clouds at Once, and When Not To — read next: Look for a Form3 engineering blog post, cost breakdown, or follow-up talk quantifying the multi-cloud cost premium.
• incompleteSame gap as the Golden Bricks extract, in a different colour. The whole programme assumes a Team-Topologies / Spotify-style organisation already exists — stream-aligned squads inside tribes, a platform team adjacent to them, business decision-makers sitting at tribe level. The Federated SRE, production manager, and technical tribe lead are role inventions overlaid on this topology, not designed to produce it. The article describes role responsibilities mechanically but treats the underlying org redesign — the part that costs years and political capital in most enterprises — as already done. For a reader whose org is functionally shaped, this is the part that matters most, and it is invisible.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: Skelton & Pais, Team Topologies (esp. the 'enabling team' interaction mode, which is what Federated SREs functionally are). Cutler on operating-model design and accountability layering. Pais on cognitive-load assessment as a precondition for federation. Reinertsen on capacity allocation and queueing — the 20% tithe is a capacity-allocation decision that needs queue-theoretic defence.
• incompleteThe 'democratisation' mechanism is never opened up. The article uses 'democratised SLOs/SLAs' as the load-bearing cultural claim but never explains how democratisation happens — who publishes the SLOs, in what tool, who is allowed to negotiate them, what stops squads from quietly setting easy SLOs to escape scrutiny, what the production manager does when a squad's SLO drifts. Without this, 'data-driven conversations' is a slogan with no enforcement surface. The mechanical specification matters: democratisation without governance is just SLO theatre.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: Google SRE Workbook on error budgets and SLO policy (the governance scaffolding democratisation usually rides on). Honeycomb / Charity Majors on observability ownership. Forsgren et al. on metrics that resist Goodhart pressure.
• incompleteThe 20% mandatory squad investment is presented as a structural lever, but the article never says how it is funded. 20% of every squad is a large standing tax on feature delivery; in most orgs this is the single most contested resource decision a CTO can make. The article gives no defence of the figure (why 20% and not 10% or 30%), no description of how it shows up in planning (is it a fifth person, a fifth of everyone's week, a rotation?), no answer to what gets dropped when the rest of the squad's commitments don't shrink. This is the place where the prescription would either hold or shatter in practice.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: Reinertsen, The Principles of Product Development Flow, on capacity allocation under variability. Allen Holub on the politics of capacity-as-budget. Cutler on the operating-system level of organisational design.
• incompleteThe four-rebuilds story is reported as triumph but never costed. Destroying and rebuilding the platform architecture four times has a cost — for the consuming squads who must re-learn it, for the regulators who must re-approve it, for the institutional memory that bleeds out at each rebuild. The article reports the platform team's experience ('it worked perfectly for us') without checking the consumer side. The same testimony from the squads might read very differently.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: Ford, Parsons & Kua on evolutionary architecture (the disciplined version of what Petean is describing). Hohpe on architecture decision records and the institutional cost of irreversibility. Naur on programming-as-theory-building — what gets destroyed in a rebuild that isn't in the code.
• incompleteCost-per-change is offered as the financial counterpart to DORA but is not interrogated. Of the four reasons given for the decline (platform effect, expertise maturity, federation, massive business scaling), the fourth alone could explain the result via simple denominator inflation. Without weighting or counterfactual ('cost per change held flat through a comparable scaling period before federation'), the metric is decorative. This matters because cost-per-change is the kind of number that gets quoted as proof of platform ROI without the caveats travelling with it.
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: FinOps Foundation material on attribution and unit economics. Cutler on signal-vs-noise in metrics. The DORA / SPACE original papers on the limits of self-reported organisational improvement.
• uninformedNo engagement with the existing literature on embedded/federated SRE. Google's SRE Book has a chapter on the embedded-SRE model; the CNCF Platforms WG has written on platform-team interaction modes; Team Topologies' 'enabling team' is functionally close to the Federated SRE role described here. The article presents the three-role pattern as if it were freshly invented, which makes it harder to compare against the failure modes the prior literature documents (embedded SREs going native to squads, central guidance eroding, federation creating shadow operators).
  How a Culture of Data-Driven Conversations Can Support Platform Engineering — read next: Beyer, Jones, Petoff, Murphy (eds.), Site Reliability Engineering, esp. the embedded-SRE chapter. CNCF Platforms WG whitepapers on team topologies for platforms. Skelton & Pais, Team Topologies, on the enabling-team interaction mode.
• incompleteThe analogy between Bainbridge's control-room operators and software engineers is taken as given but never defended. Bainbridge studied operators of physical, bounded, regulated, slow-dynamics systems (nuclear plants, chemical processes, aviation cockpits) where (a) the human's role is fundamentally MONITORING-AND-OVERRIDE, (b) the system's correct behavior is specified ex ante, (c) regulators audit the human-machine boundary, and (d) the operator is licensed and trained on simulators. Software engineers are CONSTRUCTORS — they design the system itself, not just operate it; the system's correct behavior is partially undefined; no licensing or regulator polices the boundary; and the 'incident' role is only one slice of the work. Reed's mapping works best for the incident-response slice (which is what his job is) and is weakest for routine development. He never disambiguates these. The argument that 'AI ironies for engineers = Bainbridge ironies for operators' may overgeneralize from the incident-response edge case to the whole of engineering practice.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Read the original Bainbridge 1983 paper directly. Then Hollnagel & Woods on resilience engineering, and David Mindell's 'Our Robots, Ourselves' (history of human-machine collaboration). Pair with Naur's 'Programming as Theory Building' for the contrasting view that software engineering is theory construction, not operation.
• incompleteReed's prescription assumes organizations will reinvest AI-driven efficiency gains in human thoroughness, but the same talk reports that the current pattern is the opposite: 'AI's main impact on my life is more incidents because we're generating more code.' The systems-thinking question of how to design feedback loops that prevent capture of efficiency-as-throughput is left untouched. Without that loop, the 'AI does efficiency, humans do more thoroughness' division is a moral exhortation, not a mechanism.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Donella Meadows on leverage points (especially feedback loops, rules of the system, and goals of the system). John Cutler on institutional overload and on how organizations metabolize efficiency gains. Possibly Christensen on capabilities-as-process — i.e., the org's existing process consumes new capability.
• incompleteThe empirical evidence is thin and uneven. The nurses study (Woods et al., July 2025) is presented as decisive but is a single study, in a different domain (clinical prediction), with a likely small N; the developer study (METR-style) is cited from memory. Reed does not engage the contrary evidence (e.g., positive AI productivity findings from DORA, GitHub Copilot studies) or distinguish task types where AI does and does not pay off. The asymmetric-risk frame is compelling and probably right but is underdetermined by the cited data.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: DORA 2024/2025 AI findings, the Princeton/MIT studies on developer AI productivity, METR's actual published methodology, Woods's published paper (rather than the slide). Also Cal Newport / Bainbridge-skeptics on whether the operator analogy fits knowledge work.
• incompleteThe animacy paradox is introduced as a phenomenological observation but is not connected to design recommendations. If interpretation flips with situational stakes, what design pattern reliably keeps the deterministic-machine interpretation alive in high-tempo contexts? Reed gestures at attribution and explainability but doesn't develop this.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Lucy Suchman 'Plans and Situated Actions'; Dave Woods's papers on 'envisioned worlds' and on coordination breakdowns; possibly the air-traffic-control HCI literature on 'mode confusion'.
• incompleteReed touches the systems-thinking question (juniors who never built the mental model, attrition of senior engineers carrying tacit knowledge) but doesn't push it to its leverage-point: the org's HIRING and TRAINING flows may be where the irony actually lives, not in the engineer's individual relationship to AI. The dynamic he describes is a stock-and-flow problem (the stock of tacit knowledge is depleting faster than it is being replenished) and warrants stock-and-flow analysis.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Meadows again on stocks-and-flows; the National Research Council's literature on knowledge attrition in nuclear and aerospace; and possibly the apprenticeship literature (Lave and Wenger, situated learning).
• illogicalReed conflates 'AI is non-deterministic' with 'AI lacks interpredictability.' These are related but distinct. Humans are non-deterministic and have high interpredictability with their teammates. The reason AI lacks interpredictability is not non-determinism per se but lack of a SHARED MODEL of context, goals, and norms. This matters because it suggests the fix is not 'make AI more deterministic' but 'invest in shared context and norm-conformance' — a different research and product direction.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Herbert Clark on common ground in conversation; the multi-agent systems literature on shared mental models; recent work on AI alignment through context-engineering.
• incompleteThe scale-dependence of the method itself is unexamined. Epidemiological debugging worked here only because of ChatGPT-scale volume (a full year of core dumps) plus Rockset's abnormally high signal frequency, which together produced enough failures for population statistics to be meaningful. The article says nothing about the small-team or rare-bug case, where population data never accumulates and correlations would never appear.
  OpenAI Fixes 18-Year-Old GNU libunwind Bug by Treating Crash Debugging Like Epidemiology — read next: How to root-cause when failure volume is too low for population statistics — fault injection, deterministic record-and-replay, coverage-diffing single passing/failing runs, and stress-amplifying the triggering event; and how the epidemiological approach degrades at small N.
• incompleteThe headline lesson under-credits the expertise leg. The team concludes the most important step was building the dataset, 'not the clever assembly reading or deep knowledge' — but isolating the libunwind race still required exactly that deep systems knowledge (red zone, signal frames, %rsp-before-%rip ordering). The dataset localized the two populations; it did not explain either root cause. The piece never asks how much interpretation depended on scarce expertise, nor whether 'just build the dataset' generalizes to a team that cannot read the assembly.
  OpenAI Fixes 18-Year-Old GNU libunwind Bug by Treating Crash Debugging Like Epidemiology — read next: The interplay of data infrastructure and human diagnostic expertise: whether population data substitutes for, or merely presupposes, deep systems knowledge; and the systems-thinking distinction between localizing a problem and explaining it.
• incompleteSilent hardware data corruption is treated as a one-off anecdote, not a class of failure. A CPU 'quietly getting math wrong' with no machine-check exception is described as a lucky find, but the article does not address how to systematically detect silent data corruption (SDC) at fleet scale, how prevalent it is, or what monitoring would have caught the bad host earlier.
  OpenAI Fixes 18-Year-Old GNU libunwind Bug by Treating Crash Debugging Like Epidemiology — read next: Silent data corruption at datacenter scale — detection strategies, prevalence findings (e.g. Google/Meta 'cores that don't count' / CEE studies), and fleet-level guardrails and continuous verification for silently faulty hardware.
• incompleteNo quantitative comparison is given between outcomes under Claude-Code-first-pass review/testing/incident-response and Anthropic's own prior, pre-Claude-Code baseline. Every improvement cited (67% PR throughput increase, faster incident root-causing, two-week migrations) is self-reported by Anthropic insiders or by customer companies with an interest in the tool's success, with no external or controlled measurement.
  How Claude Code Is Built — read next: Read against extract-dx-measuring-ai-impact and extract-five-studies-ai-software-engineering, which raise the same self-selection and construct-measurement concerns for AI-engineering-impact claims generally.
• incompleteThe article notes Anthropic deliberately keeps feature flags minimal because 'feature flags slow the team down,' but never examines whether this is sustainable given the growth-driven monitoring load the same article describes (alerts firing 'pretty much every day' simply from rapid growth) — nor what happens to blast-radius risk as Claude Code's external user base keeps scaling well past where it is today.
  How Claude Code Is Built — read next: Read against async-code-reviews-choking-throughput's discussion of trunk-based development and feature flags as an alternative risk-mitigation mechanism the article doesn't engage with directly.
• incompleteThe author treats unrestricted, cost-free access to the most expensive Claude model as a structural advantage unique to Anthropic, but never asks what Claude Code's own internal practices (TDD renaissance, 60-100 releases/day, 5+ prototypes/day) would look like, or whether they would remain viable, under the token-cost and usage-quota constraints that every other company — including Claude Code's own external users, per a reader comment quoted in the discussion thread — actually operates under.
  How Claude Code Is Built — read next: Compare against practitioner reports from non-Anthropic teams operating under real token budgets, to see which of these practices survive a hard cost constraint.
• incompleteThe 67% PR-throughput increase coincides with, but the article never disentangles from, the fact that new joiners during Anthropic's hiring surge were likely disproportionately experienced senior engineers already fluent with Claude Code's own on-distribution stack (TypeScript/React) — leaving open whether the throughput gain is attributable to Claude Code itself or to who Anthropic was able to hire.
  How Claude Code Is Built — read next: Look for engineering-productivity research that isolates tool effect from hiring-composition effect during headcount-doubling periods.
• incompleteThe organisational shift the article requires but does not theorise. Bricks-not-paths only works in an organisation that has already pushed correctness, security review, cost ownership, and incident response down into stream-aligned teams. In an org where the platform team is also the compliance team and the on-call team, bricks-without-a-path means each stream team must reinvent guardrails. The piece treats this organisational redesign as free — but it is the actual hard part.
  How Platform Engineering Using Golden Bricks Can Enable Fast and Smooth Delivery — read next: Cutler's TBM essays on institutional overload and accountability. Allen Holub's #NoAccountability work. Conway's Law re-readings (Skelton/Pais). Andrew Clay Shafer / DevOps adoption literature on the politics of platform ownership.
• incompleteThe article's evaluative back half (sections 5–7: cost/'tokenmaxxing' disappointment stories, the 'was it just a hack' argument, and the context-engineering conclusion) is paywalled; this extraction only had the one-line table-of-contents previews for that material, not the supporting evidence or full argument.
  What is “loop engineering?” — read next: Get subscriber access to read the full piece, especially the cost-tradeoff stories in section 5 and the closing argument in section 7 for why context engineering may matter more than loop engineering — directly relevant to the persona's 'rigorous adopter, not hype-follower' stance on AI-agent practice.
• incompleteThe article doesn't examine whether Gene's specific recovery mechanism — a smart model having already, autonomously, written documentation as a side effect of its normal work, later discovered by luck — actually generalizes to other AI-built systems, or whether it depended on quirks specific to his project (a documentation-happy model, a single owner who knew where to look).
  Stranded in the Slow Zone — read next: Read alongside production-engineering literature on documentation-driven resilience and disaster recovery, or a follow-up account of whether this pattern held for a second builder in a second incident.
• illogicalThe article states that SKIP LOCKED gives "exactly-once processing guarantees," but the mechanism it actually describes only guarantees exactly-once *claim* — a row is assigned to exactly one worker at a time. Exactly-once *effect* (the property practitioners usually mean by that phrase) depends on the separate idempotent-checkpoint mechanism introduced two sections later. Stating the stronger claim in the SKIP LOCKED section conflates dispatch-level and effect-level guarantees.
  Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Kleppmann's 'Designing Data-Intensive Applications' on effectively-once/exactly-once semantics, or Temporal's own documentation explaining why it separates at-least-once activity delivery from exactly-once effects via idempotency keys — the same distinction this article's own idempotency section implicitly relies on but doesn't name when making the SKIP LOCKED claim.
• uninformedThe article credits MySQL 8.0+, MariaDB 10.6+, Oracle, Db2, and SQL Server with SKIP LOCKED-equivalent mechanisms in a single aside, implying the pattern ports cleanly across databases, but does not note that these engines differ meaningfully in isolation-level defaults and MVCC/locking implementation (e.g., MySQL's gap-locking behavior under REPEATABLE READ) — real portability risk for a reader who lifts the exact SQL onto a different database.
  Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Each database's own locking-read documentation, cross-compared for isolation-level defaults, before assuming this pattern is a drop-in replacement across engines.
• underdevelopedThe blockchain C2 technique is mentioned but its broader implications are not explored. If attackers routinely move C2 to smart contracts, what does this mean for the entire domain-takedown model of incident response?
  Attacker Bought 30 WordPress Plugins on Flippa and Backdoored All of Them