Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure

Sam Newman · rss · 2026-08-19 source ↗ Reader 🖥️

Cascading Failure & System Resilience Patterns Ironies of Automation Modularity & Coupling SRE & Incident Response

Progressive collapse — civil engineering's term for a small, localized failure cascading into a system-wide catastrophe — is a productive lens for distributed-systems resilience: mitigation requires reducing hazards, strengthening components, and reducing interconnection together, but each mitigation (redundancy, multi-region, circuit breakers) carries real cost and complexity and can itself introduce new interconnection and new failure modes, so teams must apply these techniques as conscious, business-contextualized trade-offs rather than default best practice.

practical · mixed — Primarily practical/argumentative: Newman argues software teams should adopt a three-part mitigation framework, applied consciously and case-by-case rather than by rote. But the talk is delivered mostly through narrative case studies (Ronan Point, the AWS us-east-1 outage, his own 'Sauron' incident) that carry the argument, plus an expository segment translating a civil-engineering NIST report into a software framework.

Structure

Part	Role
The Ronan Point tower collapse (1968) narrative	Founding case study that defines 'progressive collapse' and introduces the talk's central metaphor (dominoes, load-bearing structure).
The AWS us-east-1 outage narrative (DynamoDB DNS enactor race condition)	Second case study, showing the same pattern in a digital system and setting up the 'redundancy paradox' argument developed later.
The 'Sauron' used-car-site outage narrative (Newman's own incident)	Third case study, a first-person, ground-level example used throughout the talk to illustrate each mitigation category concretely.
Critique of single-'root cause' thinking and the traditional-safety-management vs. resilience-engineering framing	Theoretical pivot that motivates why three complementary mitigation categories are needed instead of eliminating one trigger.
The three-part mitigation framework (reduce hazards / strengthen components / reduce interconnection), sourced from a NIST report on building progressive collapse and mapped onto all three case studies in turn	Main argumentative body — evidence and worked examples for the practical claim.
The redundancy/complexity paradox (AWS's own DNS-enactor redundancy caused its race condition; circuit breakers can worsen partial failure and cause retry storms)	Counter-argument / complication that undercuts a naive reading of 'strengthen components' and 'reduce interconnection' as unconditionally good, forcing the conscious-trade-off conclusion.
Closing appeal to publishing and reading incident reports (Ronan Point inquiry, AWS's and Cloudflare's postmortems) as the mechanism of industry-wide learning	Conclusion / call to action, independent of the three-part framework.
Q&A: prioritization framing (business-context cost/benefit) and the AI-agent/junior-developer parallel to Ronan Point's low-skilled, fast-build labor	Extends the talk's argument into two new applications the main talk didn't cover: how to prioritize mitigation work, and whether AI-agent-built software risks a Ronan-Point-style structural flaw.

Key terms

Progressive collapse Cascading Failure & System Resilience Patterns
A situation where a small failure, innocuous in isolation, cascades into a significantly larger collapse of the wider system — borrowed directly from civil engineering (Ronan Point) and applied to distributed digital systems.
Root cause
A misleadingly singular framing Newman argues against; he insists on 'root causes' (plural) and treats the search for one blameable trigger as an oversimplification that leaves systemic cascade-susceptibility unaddressed.
Resilience engineering Cascading Failure & System Resilience Patterns
A paradigm that accepts failure is inevitable and focuses on mitigating impact and enabling recovery, contrasted explicitly with 'traditional safety management.'
Bulkhead Cascading Failure & System Resilience Patterns
A shipping-derived metaphor for compartmentalizing failure so it cannot spread — in software, isolating resources (e.g., separate connection pools per downstream dependency) so one failing dependency cannot exhaust shared capacity.
Reduce interconnection
One of Newman's three mitigation categories: minimizing the pathways through which a local failure can propagate — includes bulkheads, circuit breakers, and (his broader claim) treating identical deployed code across sites/vendors as itself a form of interconnection and correlated-failure risk.
Interconnection (as failure vector)
Extended beyond physical network topology to include shared code, shared infrastructure, and shared vendor dependencies — 'a bug that takes down your system on AWS could just as likely take down your system on Azure' if the code is identical.

Propositions

explicitA progressive collapse is a small, seemingly innocent initial failure that cascades to cause a significantly larger collapse in the wider system.conf: high
explicitMitigating progressive collapse requires three complementary categories of technique — reducing hazards, strengthening components, and reducing interconnection — applied together where possible.conf: high
supportsThe Hard-Stop Rule: From 3 HCM Monoliths to 120 Domain Microservices · p6 — The HCM migration's per-domain-subscription blast-radius isolation is a concrete production instance of Newman's 'reduce interconnection' mitigation category.
explicitFixating on a single 'root cause' is deeply problematic because it encourages overly simplistic thinking about failure in large, complex systems where not all triggers can be eliminated.conf: high
explicitcorroboratedMechanisms added specifically to strengthen or protect a system (redundant instances, circuit breakers, multi-cloud) can themselves introduce new interconnection and new sources of failure, sometimes causing the very failure mode they were meant to prevent.conf: high
supportsPresentation: How Netflix Shapes our Fleet for Efficiency and Reliability · p6 — Both show that a uniform capacity/reliability strategy (same DNS enactor design everywhere / same newest hardware generation fleet-wide) creates new systemic risk rather than simply improving reliability.
supports ←Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments · p8 — A concrete case of a mechanism meant to preserve capacity (ECS task replacement/rebalancing) becoming the interconnection that produces a worse failure — a start-stop loop that took the service down by a third.
explicitIdentical code or infrastructure deployed across otherwise-independent sites or vendors is itself a form of interconnection, because a shared bug or shared dependency can cause correlated failures across those sites.conf: high
extendsHow to Build More Resilient Local-First Applications With AT Protocol Infrastructure · p2 — Extends the single-point-of-failure critique of a centralized backend to any shared dependency — identical code, shared vendor infrastructure — as a general interconnection-based blast-radius risk.
explicitcorroboratedCircuit breakers, despite being a standard resilience pattern, can worsen partial failure (an error in part of a service's functionality can take down the whole circuit) and cause boom-bust retry storms if poorly configured.conf: high
supports ←Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments · p10 — Confirms that untested circuit breakers behave unpredictably in practice: the breaker opens and traffic fails over to a backup route with its own unresolved dependencies.
explicitcorroboratedWhether to invest in a given mitigation (redundancy, multi-region failover, fixing a failing component) should be a deliberate, business-contextualized cost/benefit decision, not a default or rote application of an industry pattern.conf: high
supports ←Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments · p15 — A concrete instance of treating a resilience/mitigation investment as a deliberate cost-benefit call — the team abandoned a cost optimization (Spot) entirely rather than engineer around an ambiguous-financial-state failure mode.
explicitMulti-region/multi-site strategies form a continuum (backup-and-restore, pilot light, warm standby, active-active) that trades reduced downtime and data loss against increasing cost and complexity.conf: high
explicitPublishing detailed incident reports (AWS's, Cloudflare's) is a genuine mechanism of industry-wide learning about progressive collapse, even though such reports also function as public relations.conf: high
implicitThe industry over-applies established resilience patterns (circuit breakers, Kubernetes-based redundancy, multi-cloud) by convention or imitation rather than evaluating whether they fit the specific context, which itself increases system complexity and interconnection without commensurate benefit.conf: medium
implicitThe proximate technical trigger of a structural failure (a gas explosion, a race condition) is frequently less important than the underlying organizational or resource decisions that made the system fragile in the first place (using low-skilled labor and improvised materials at Ronan Point; the implied parallel to AI agents replacing junior developers).conf: medium
implicitThe three-part mitigation framework (hazard reduction, component strengthening, interconnection reduction), though derived from a civil-engineering report about physical buildings, transfers to software and organizational systems with only light translation.conf: medium

Arguments

Teams should not treat elimination of a single root cause as sufficient resilience work; they must also reduce a system's capacity to cascade once some trigger occurs. strength: high

Explicit premises:

Systems are getting bigger and more complex.
Not all triggers can be eliminated (lightning strikes; gas explosions; race conditions).
Removing one trigger (e.g., banning open fires) is useful but not by itself sufficient to prevent catastrophic outcomes.

Implicit premises:

The triggering event and the system's structural susceptibility to cascade are separable causal factors that can be addressed independently.
Reducing cascade-susceptibility yields resilience benefits that persist even against unanticipated or unremovable future triggers.

Inferences:

Organizations should audit systems for cascade-prone structure proactively, independent of any specific past incident, rather than only reacting to the trigger a given incident revealed.

Resilience mechanisms are not unconditionally beneficial; adding them can introduce new interconnection and new failure modes, sometimes causing exactly the failure category they were meant to prevent. strength: high