Cascading Failure & System Resilience Patterns
Architectural and organizational patterns — bulkheads, circuit breakers, load shedding, redundancy, multi-site strategies — for containing a small localized failure before it cascades into a system-wide collapse, plus the resilience-engineering view (accept failure, mitigate impact) that supersedes single-root-cause safety thinking; includes the paradox that added resilience mechanisms can themselves introduce new interconnection and failure modes.
Also known as: progressive collapse, cascading failure, bulkhead, bulkheads, circuit breaker, circuit breakers, load shedding, resilience engineering, traditional safety management, multi-region failover, pilot light, warm standby, active-active
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 | Progressive collapse | A situation where a small failure, innocuous in isolation, cascades into a significantly larger collapse of the wider system — borrowed directly from civil engineering (Ronan Point) and applied to distributed digital systems. |
| Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 | Resilience engineering | A paradigm that accepts failure is inevitable and focuses on mitigating impact and enabling recovery, contrasted explicitly with 'traditional safety management.' |
| Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 | Bulkhead | A shipping-derived metaphor for compartmentalizing failure so it cannot spread — in software, isolating resources (e.g., separate connection pools per downstream dependency) so one failing dependency cannot exhaust shared capacity. |
Documents (5)
- Article: Implementing Durable Workflows on Postgres Without an External Orchestrator Raman Varma · 2026-09-14 Reader 🖥️
Durable workflow execution is fundamentally a database-checkpointing problem, so a team that already operates a relational database like Postgres at scale can implement production-grade durable execution — concurrent dispatch, idempotency, crash recovery, durable waits, and observability — as a handful of well-understood SQL primitives, without deploying a separate stateful orchestrator such as Temporal or AWS Step Functions. - Presentation: How To Run on Three Clouds at Once, and When Not To Ross McFarlane, Kevin Holditch · 2026-09-11 source ↗ Reader 🖥️
Form3 proves that a regulated payments platform can run truly active-active-active across three cloud providers by treating each cloud as nothing more than another availability zone and building custom Kubernetes operators to extend the platform's own primitives across cluster boundaries where they run out — but the same team's parallel US disaster-recovery build shows that this triple-active design is not a universal best practice: it is justified only when a specific market's resilience expectations, a mature platform-engineering practice, and a willingness to pay a real cost premium all line up, and even then the harder problem — deciding, mid-incident, whether to actually trigger a failover — remains an unresolved judgment call under uncertainty rather than something engineering can fully automate away. - Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments Salim Adedeji · 2026-09-08 source ↗ Reader 🖥️
In regulated, stateful payment systems running on ECS, chaos engineering only works if teams first replace the standard stateless-web playbook's three violated assumptions — clean rollback, definable blast radius, and free production experimentation — with a compliance-integrated, four-stage blast-radius graduation, because it is measured behavior under actual injected faults, not configuration values or architecture review, that reveals where these systems actually break. - Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure Sam Newman · 2026-08-19 source ↗ Reader 🖥️
Progressive collapse — civil engineering's term for a small, localized failure cascading into a system-wide catastrophe — is a productive lens for distributed-systems resilience: mitigation requires reducing hazards, strengthening components, and reducing interconnection together, but each mitigation (redundancy, multi-region, circuit breakers) carries real cost and complexity and can itself introduce new interconnection and new failure modes, so teams must apply these techniques as conscious, business-contextualized trade-offs rather than default best practice. - Lenny Rachitsky's Post Lenny Rachitsky · 2026-04-24 source ↗ Reader 🖥️
Anthropic's Claude Code team ships at compressed (week-or-day-scale) timelines by merging the PM/engineer/designer roles around product taste, deliberately building and shipping products before the underlying model is fully capable so the gap can be tested the instant a better model ships, building its own internal tools with Claude Code rather than buying SaaS, and hiring people who respond to escalating production chaos with enthusiasm rather than stress.
Propositions touching this concept (14)
By publication month, oldest first (extraction month when no publication date is known).
2026-04
- explicitcorroboratedEffective fleet management requires layered feedback loops operating at different timescales: months (supply shaping), minutes (autoscaling), and seconds (load shedding).
- implicitEnd-to-end control over traffic — from DNS steering through to in-service load shedding — is a prerequisite for achieving the described efficiency-reliability balance.
2026-07
- explicitcorroboratedReview effort should be matched to the cost of being wrong — tier by risk not by author, fast-fail the expensive tail with a cheap 'circuit breaker', refuse to review changes that arrive without evidence (intent, a readable diff, run test output), keep PRs deliberately small, and read the test-change diffs more carefully than the code, because the agent's characteristic failure is rewriting a failing assertion to match new broken behavior.
- explicitThe decision of what to multi-region (only payroll, active-active across two US regions) and what not to is itself a deliberate budget and risk trade-off, not an oversight -- most components tolerate single-region operation, and over-replication would cost more than it saves.
- explicitMaking imperfect models economically viable in closed workflows requires separating probabilistic reasoning ('user space') from deterministic execution authority ('kernel space') via three governance layers: syntactic (schema/authority checks), semantic (evidence-based intent validation), and temporal (decision telemetry and circuit breakers).
2026-08
- explicitA progressive collapse is a small, seemingly innocent initial failure that cascades to cause a significantly larger collapse in the wider system.
- explicitMitigating progressive collapse requires three complementary categories of technique — reducing hazards, strengthening components, and reducing interconnection — applied together where possible.
- explicitcorroboratedMechanisms added specifically to strengthen or protect a system (redundant instances, circuit breakers, multi-cloud) can themselves introduce new interconnection and new sources of failure, sometimes causing the very failure mode they were meant to prevent.
- explicitcorroboratedCircuit breakers, despite being a standard resilience pattern, can worsen partial failure (an error in part of a service's functionality can take down the whole circuit) and cause boom-bust retry storms if poorly configured.
- explicitcorroboratedWhether to invest in a given mitigation (redundancy, multi-region failover, fixing a failing component) should be a deliberate, business-contextualized cost/benefit decision, not a default or rote application of an industry pattern.
- explicitMulti-region/multi-site strategies form a continuum (backup-and-restore, pilot light, warm standby, active-active) that trades reduced downtime and data loss against increasing cost and complexity.
- explicitPublishing detailed incident reports (AWS's, Cloudflare's) is a genuine mechanism of industry-wide learning about progressive collapse, even though such reports also function as public relations.
- implicitThe industry over-applies established resilience patterns (circuit breakers, Kubernetes-based redundancy, multi-cloud) by convention or imitation rather than evaluating whether they fit the specific context, which itself increases system complexity and interconnection without commensurate benefit.
2026-09
- explicitCircuit breakers that have never been triggered in practice behave unpredictably the first time a chaos experiment opens them — traffic fails over to a backup route that has not been maintained and has its own unresolved dependencies.
Open gaps (16)
- incompleteNo framework is given for deciding which of the concrete numeric findings (93s failover, 2.4x DB load amplification, 14,000 stuck transactions) are specific to this author's stack (400 TPS, this JVM version, this HikariCP tuning) versus generalizable. A reader on a non-JVM stack, a different database, or Kubernetes instead of ECS cannot tell how much of the 'what surprised us' material would transfer.Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments — read next: Look for a comparable chaos-engineering-in-fintech account on a different runtime/orchestrator (e.g. Kubernetes-based payment platform) to see which findings recur and which are ECS/JVM-specific.
- incompleteThe compliance/change-management approval process is asserted to 'produce an audit trail that satisfies PCI DSS and SOC 2,' but no example of that documentation, approval turnaround time, or an auditor's actual response is shown. How Stage 3/4 experiments get approved in real time, and what happens when a rollback condition triggers mid-experiment under regulatory scrutiny, is left unaddressed.
- incompleteThe financial cost/benefit of running the chaos-engineering program itself is only asserted qualitatively (six to twelve months to Stage 4; 'the regulatory cost may exceed the learning value'), with no quantification of engineering-hours invested versus incidents prevented or dollars saved — creating an asymmetry, since the cost of NOT doing chaos engineering is made vivid with a precise seven-figure SLA-penalty figure in the opening narrative, but the cost of doing it is never quantified to compare against.
- incompleteThe 'virtual workforce' framing never addresses runtime failure modes for a remote A2A specialist (unavailable, slow, or behaving maliciously/incorrectly), despite the paper's own MCP best-practices section explicitly warning against trusting unverified public servers — the same caution is absent from the A2A discovery/registry sections.Agent Tools & Interoperability — read next: Look for A2A-specific resilience patterns (circuit breakers, fallback orchestration, registry trust scoring) in follow-on protocol documentation.
- uninformedNo engagement with the Resilience Engineering academic tradition (Hollnagel, Woods, Dekker), which has spent 20+ years arguing resilience is the adaptive capacity of a sociotechnical system — not something that can be 'owned' by a single accountable executive. The article reads as if this body of thought does not exist.The Case for Hiring a Chief Resilience Officer — read next: Hollnagel, Woods & Leveson, 'Resilience Engineering: Concepts and Precepts'; David Woods' lectures on graceful extensibility.
- incompleteNo discussion anywhere of correlated or shared-upstream failure modes across the three nominally independent cloud providers (root DNS, BGP routing, major CDNs, certificate authorities, common package registries) that could plausibly take out more than one cloud at once. This matters concretely here: the nine-pod CockroachDB Raft cluster only tolerates the loss of one full cloud (needs 5 of 9 alive); a correlated two-cloud event, never modeled in the talk, would be a full database outage.Presentation: How To Run on Three Clouds at Once, and When Not To — read next: Look for Form3 engineering follow-ups or independent analysis addressing correlated multi-cloud failure risk; this is the same gap already flagged against Sam Newman's multi-cloud claims in extract-understanding-progressive-collapse.
- uninformedThe talk repeatedly asserts that triple-active multi-cloud 'is definitely more expensive' than single-cloud, and that splitting one large production environment into several smaller ones reduces cost, but gives no order-of-magnitude figure for either claim (no multiplier, no percentage, no dollar range), so a reader cannot judge whether the cost tradeoff argument in the 'when not to' section is proportionate advice or an unquantified hand-wave.Presentation: How To Run on Three Clouds at Once, and When Not To — read next: Look for a Form3 engineering blog post, cost breakdown, or follow-up talk quantifying the multi-cloud cost premium.
- incompleteThe analogy between Bainbridge's control-room operators and software engineers is taken as given but never defended. Bainbridge studied operators of physical, bounded, regulated, slow-dynamics systems (nuclear plants, chemical processes, aviation cockpits) where (a) the human's role is fundamentally MONITORING-AND-OVERRIDE, (b) the system's correct behavior is specified ex ante, (c) regulators audit the human-machine boundary, and (d) the operator is licensed and trained on simulators. Software engineers are CONSTRUCTORS — they design the system itself, not just operate it; the system's correct behavior is partially undefined; no licensing or regulator polices the boundary; and the 'incident' role is only one slice of the work. Reed's mapping works best for the incident-response slice (which is what his job is) and is weakest for routine development. He never disambiguates these. The argument that 'AI ironies for engineers = Bainbridge ironies for operators' may overgeneralize from the incident-response edge case to the whole of engineering practice.Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Read the original Bainbridge 1983 paper directly. Then Hollnagel & Woods on resilience engineering, and David Mindell's 'Our Robots, Ourselves' (history of human-machine collaboration). Pair with Naur's 'Programming as Theory Building' for the contrasting view that software engineering is theory construction, not operation.
- incompleteThe escalating-severity incident example (P0 -> P00 -> P000 within about a day) is used to justify a hiring-for-attitude solution, but the post never asks why severity is escalating that quickly in the first place, or whether structural/process fixes (bulkheads, circuit breakers, staged rollouts) could reduce the frequency or speed of such cascades rather than only selecting people who can emotionally absorb them.Lenny Rachitsky's Post — read next: Look at cascading-failure-resilience content in this corpus for architectural/organizational patterns (bulkheads, circuit breakers, load shedding) that address the frequency of severity escalation directly, as a complement to -- not substitute for -- hiring for resilience.
- uninformedThis is Lenny's secondhand, seven-bullet summary of a podcast interview whose full content is already extracted elsewhere in this corpus (extract-catherine-wu-claude-code-anthropic, 19+ propositions). Points 1, 2, 3, and 5 here restate claims already present in that primary extraction; points 4, 6, and 7 do not appear among that extraction's top propositions. It is unclear whether that reflects genuinely new material from the full conversation or simply a difference in what Lenny chose to highlight versus what the original extraction's author prioritized.Lenny Rachitsky's Post — read next: See extract-catherine-wu-claude-code-anthropic for the fuller primary-source extraction this post summarizes.
- incompletePoint 6 (abandoning Notion, Linear, and Figma in favor of custom Claude-Code-built tools plus Cowork and Slack) is presented as a pure productivity win, with no discussion of the switching costs, the maintenance burden of many bespoke internal tools built by non-engineers, or what is lost specifically in design handoff, accessibility review, or historical design-rationale tracking without a dedicated design tool like Figma.
- incompleteThe talk never offers a framework or heuristic for deciding, in advance and quantitatively, when the interconnection/complexity cost a resilience mechanism adds (redundancy, circuit breakers, multi-region) outweighs its benefit — the only guidance given is qualitative ('have a conversation as a team,' negotiate cost with the business owner). There is no attempt to measure or bound the complexity a given mitigation introduces before deploying it.Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure — read next: Resilience-engineering and safety-II literature (Hollnagel, Woods) on quantifying trade-offs between robustness and complexity; chaos-engineering practice for empirically testing whether an added mitigation actually reduces or relocates risk before it ships.
- incompleteNewman explicitly leaves the AI-agent guardrails question open in the Q&A, but the talk never develops what a structural, Ronan-Point-style progressive collapse specific to AI-agent-produced software would actually look like — e.g., whether a single flawed architectural pattern an agent replicates across many services or many repositories is the software analogue of Ronan Point's shared, flawed prefab construction system, and if so, what 'reduce interconnection' would mean for a fleet of AI-generated services sharing the same generative pattern or the same underlying model.Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure — read next: Corpus material on agentic engineering and code-quality risk from AI-generated code at scale; look for post-incident analyses of correlated failures traced to a shared AI-generated pattern or shared prompt/skill.
- incompleteNewman's claim that multi-cloud hedges against 'vendor failure' as opposed to 'site failure' doesn't engage with the shared upstream dependencies that nominally independent cloud vendors still have in common (root DNS, major CDNs, BGP routing, common upstream registries like npm/PyPI, common certificate authorities) — his own 'shared code is a form of interconnection' argument implies these shared substrates could still produce correlated multi-cloud failure, but the talk doesn't extend the analysis that far.Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure — read next: Internet infrastructure dependency-mapping research (e.g., studies of correlated cloud/CDN/DNS outages) that quantify how much 'independence' multi-cloud actually buys once shared internet substrate is accounted for.
- illogicalThe article states that SKIP LOCKED gives "exactly-once processing guarantees," but the mechanism it actually describes only guarantees exactly-once *claim* — a row is assigned to exactly one worker at a time. Exactly-once *effect* (the property practitioners usually mean by that phrase) depends on the separate idempotent-checkpoint mechanism introduced two sections later. Stating the stronger claim in the SKIP LOCKED section conflates dispatch-level and effect-level guarantees.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Kleppmann's 'Designing Data-Intensive Applications' on effectively-once/exactly-once semantics, or Temporal's own documentation explaining why it separates at-least-once activity delivery from exactly-once effects via idempotency keys — the same distinction this article's own idempotency section implicitly relies on but doesn't name when making the SKIP LOCKED claim.
- uninformedThe article credits MySQL 8.0+, MariaDB 10.6+, Oracle, Db2, and SQL Server with SKIP LOCKED-equivalent mechanisms in a single aside, implying the pattern ports cleanly across databases, but does not note that these engines differ meaningfully in isolation-level defaults and MVCC/locking implementation (e.g., MySQL's gap-locking behavior under REPEATABLE READ) — real portability risk for a reader who lifts the exact SQL onto a different database.Article: Implementing Durable Workflows on Postgres Without an External Orchestrator — read next: Each database's own locking-read documentation, cross-compared for isolation-level defaults, before assuming this pattern is a drop-in replacement across engines.