Chaos Engineering
The practice of deliberately injecting failure into a running system — under a defined steady-state hypothesis, scoped blast radius, and rollback condition — to discover failure modes empirically rather than through review or architecture diagrams alone.
Also known as: chaos engineering, fault injection, game day, game days, Chaos Monkey, AWS Fault Injection Simulator, FIS, chaos experiments, blast radius scoping, steady-state hypothesis
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Stranded in the Slow Zone Tim O'Reilly · 2026-07-24 | chaos monkey (applied to AI model dependency) | Deliberately pulling your own most capable model to test whether a lesser one can keep the system running — a deliberate resilience-testing practice Gene applies to model choice specifically. |
Documents (2)
- Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments Salim Adedeji · 2026-09-08 source ↗ Reader 🖥️
In regulated, stateful payment systems running on ECS, chaos engineering only works if teams first replace the standard stateless-web playbook's three violated assumptions — clean rollback, definable blast radius, and free production experimentation — with a compliance-integrated, four-stage blast-radius graduation, because it is measured behavior under actual injected faults, not configuration values or architecture review, that reveals where these systems actually break. - Stranded in the Slow Zone Tim O'Reilly · 2026-07-24 source ↗ Reader 🖥️
Frontier AI agents now let individual builders create personal systems with the complexity and fragility of enterprise software, so the DevOps discipline that once belonged only to large organizations — documentation that outlives any one model, and tool portability rather than model-agnostic architecture or self-hosting — has become a personal survival skill, even as the same shift turns engineers into managers of AI 'bots' and raises the same craft-versus-slop question for writing that it already raised for code.
Propositions touching this concept (4)
By publication month, oldest first (extraction month when no publication date is known).
2026-09
- explicitPCI DSS, SOC 2, and most banking regulations require change-management approval for anything that intentionally degrades production systems, so running chaos experiments without an approval chain creates audit findings.
- explicitThe safe path to running chaos experiments on primary payment services runs through four stages of increasing risk exposure — staging with production-traffic shadows, non-transaction-path production services, secondary transaction-path services during low-traffic windows, and finally primary services with full rollback automation — with most teams reaching Stage 2 within a month and Stage 4 within a year.
- explicitChaos engineering should not be adopted before a team has observability (distributed tracing, structured logging, real-time dashboards), an incident-response process (on-call, runbooks, escalation), architectural stability, and basic testing (load, integration, deployment automation) already in place.
- implicitThe organizational process of running compliant chaos experiments — writing steady-state definitions, filing change requests, scoping blast radius by transaction role, writing postmortems — is harder to execute correctly, and more valuable, than the technical fault-injection mechanics themselves.
Open gaps (3)
- incompleteNo framework is given for deciding which of the concrete numeric findings (93s failover, 2.4x DB load amplification, 14,000 stuck transactions) are specific to this author's stack (400 TPS, this JVM version, this HikariCP tuning) versus generalizable. A reader on a non-JVM stack, a different database, or Kubernetes instead of ECS cannot tell how much of the 'what surprised us' material would transfer.Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments — read next: Look for a comparable chaos-engineering-in-fintech account on a different runtime/orchestrator (e.g. Kubernetes-based payment platform) to see which findings recur and which are ECS/JVM-specific.
- incompleteThe compliance/change-management approval process is asserted to 'produce an audit trail that satisfies PCI DSS and SOC 2,' but no example of that documentation, approval turnaround time, or an auditor's actual response is shown. How Stage 3/4 experiments get approved in real time, and what happens when a rollback condition triggers mid-experiment under regulatory scrutiny, is left unaddressed.
- incompleteThe financial cost/benefit of running the chaos-engineering program itself is only asserted qualitatively (six to twelve months to Stage 4; 'the regulatory cost may exceed the learning value'), with no quantification of engineering-hours invested versus incidents prevented or dollars saved — creating an asymmetry, since the cost of NOT doing chaos engineering is made vivid with a precise seven-figure SLA-penalty figure in the opening narrative, but the cost of doing it is never quantified to compare against.