Presentation: How To Run on Three Clouds at Once, and When Not To

Ross McFarlane, Kevin Holditch · rss · 2026-09-11 source ↗ Reader 🖥️

practical · conference talk transcript (InfoQ presentation writeup, with audience Q&A) — First-person practitioner account by Form3's VP of Engineering (Kevin Holditch) and a US-market architect (Ross McFarlane) of building and operating a real production triple-active multi-cloud payments platform, plus a separate active-standby disaster-recovery build for the US market. Structured as a chronological engineering narrative (v1 -> v2 -> three hard problems -> does it work? -> US resilience story -> lessons -> when not to -> Q&A) rather than an argued thesis paper.

Structure

Key terms

• Active-active-active (treating a cloud provider like an availability zone)
  Extending the failure-tolerance paradigm Form3 had already mastered at the availability-zone level up one level of abstraction, so that any one of three simultaneously-serving cloud providers can fail entirely with no visible customer impact.
• X-PDB (cross-cluster pod disruption budget) Custom Infrastructure Operators (Extending Platform Primitives)
  A custom, open-sourced Kubernetes operator that spans a single pod-disruption-budget invariant across n independent Kubernetes clusters, so that node rollouts in one cloud respect how many pods are already down in the others.
• CLO (Cluster Lifecycle Operator) Custom Infrastructure Operators (Extending Platform Primitives)
  An in-house operator that manages node-pool rollout policy (how many pools can roll at once, at what pace) at the level of 'what image should the whole platform run', collapsing per-node-pool, per-cloud, per-environment PRs into one PR per platform.
• Non-zero RTO/RPO
  A deliberately accepted disaster-recovery posture (some recovery time, some possible data loss) that Form3's US customers already expected and were comfortable with, used as license to build a cheaper active-standby architecture instead of replicating the UK's triple-active design.
• Asteroids of the mind
  Ross's phrase, initially used dismissively, for the far-fetched multi-site-disaster scenarios his sales team worried US customers were imagining — before he concludes those customers' concerns were not far-fetched but simply expressed in a different (geographic) resilience vocabulary.

Propositions

• explicitA UK banking regulator's 2021 statement about cloud-concentration risk, combined with a major bank customer's explicit demand for a multi-cloud strategy, was the actual trigger for Form3's multi-cloud investment — not a proactive architectural decision.conf: high
• explicitTreating a cloud provider as if it were just another availability zone, and reapplying the AZ-failure-tolerance techniques Form3's v1 platform had already mastered, is the design principle that made a triple-active multi-cloud platform tractable.conf: high
• explicitChoosing cloud-agnostic technologies (Kubernetes, Go, NATS JetStream, CockroachDB) over cloud-native ones was necessary to avoid maintaining separate versions of the payments software per cloud provider, which the team explicitly framed as an unacceptable 'maintenance nightmare.'conf: high
  extendsPresentation: Clean Architecture for Serverless: Business Logic You Can Take Anywhere · p4 — Both pursue cloud-vendor-portable architecture to avoid maintaining per-provider software versions, but Form3's case extends the idea from deploy-time portability (same FaaS code deployable to two providers) to simultaneous, always-on runtime portability (one logical platform actively serving from three providers at once).
• explicitA single logical NATS JetStream cluster and a single logical CockroachDB cluster can run across three independent Kubernetes clusters in three different clouds, letting application services remain unaware they are operating in a multi-cloud environment.conf: high
• explicitCross-cloud pod-to-pod DNS addressing was solved by extending Kubernetes' built-in headless-service DNS primitive with a custom cloud-name suffix convention, CoreDNS forwarding rules, and a per-cloud network load balancer on a static IP, rather than by replacing Kubernetes' native networking.conf: high
• explicitRunning three separate, independent Kubernetes clusters (instead of one logical cluster spanning three clouds) was a deliberate fault-tolerance tradeoff, made in full awareness that it would sacrifice built-in Kubernetes primitives' ability to reason across cluster boundaries.conf: high
• explicitKubernetes' built-in Pod Disruption Budget primitive is cluster-scoped and so cannot express a genuinely cross-cluster invariant: a naive per-cloud max-unavailable-of-one PDB applied to a nine-pod, Raft-based CockroachDB cluster (which needs 5 of 9 pods alive) could legally leave the cluster with only four pods running — a full database outage.conf: high
  supportsPresentation: Parting the Clouds: The Rise of Disaggregated Systems · p8 — Demirbas's abstract point that Raft-style replication needs a quorum majority (2f+1-style reasoning) to avoid split-brain is directly instantiated by Form3's concrete case: a nine-pod Raft cluster needing 5 of 9 alive, and the cross-cluster PDB gap that could have violated that quorum.
• explicitForm3 built and open-sourced a custom Kubernetes operator (the cross-cluster pod disruption budget, or 'X-PDB') that spans a single disruption-budget invariant across n independent clusters, closing the quorum-safety gap the built-in primitive left open.conf: high
• explicitModeling infrastructure-as-code at too fine a grain (separate node pools per technology, multiplied across clouds, environments, geographies, and customer product sets) is what turned a routine task — rolling an OS image for a CVE — into what was effectively a full-time job of generating and merging hundreds of pull requests.conf: high
• explicitcorroboratedBuilding a custom operator (the Cluster Lifecycle Operator, CLO) that manages node pools at a higher level of abstraction — letting an engineer specify one target image for an entire platform — collapsed what had been hundreds of pull requests into one PR per platform.conf: high
  supports ←Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs · p9 — Same general pattern in a different domain: extending/wrapping a platform's native primitive with custom control-plane engineering (Form3's Cluster Lifecycle Operator; Netflix's Spring Boot/Temporal control plane) rather than adopting the primitive as-is.
• explicitThe 'treat a cloud as an availability zone' design was validated empirically by a real, unplanned event: during a multi-hour Google Cloud outage, Form3's UK platform showed only a low-level alert (some pods crash-looping in GCP) while all payments continued to flow through the other two clouds.conf: high
• explicitA resilience architecture's credibility with a given market's customers depends on matching that market's specific, conventional resilience narrative and vocabulary, not on the architecture's objective technical merit.conf: high
• explicitLatency, not lack of technology, is the specific reason Form3's triple-active multi-cloud design could not simply be lifted and shifted to span US geography: every write requires cross-cloud quorum, and continental transit distances would burn through payment-processing SLAs.conf: high
• explicitBecause US customers already treated a non-zero RTO/RPO as the market-normal expectation for geographic disaster recovery, Form3 could deliberately choose a cheaper, simpler active-standby architecture (AWS East primary, GCP West standby) for the US market instead of replicating the UK's triple-active design.conf: high
• explicitA payments platform's real disaster-recovery bottleneck after a regional failover is data restoration, not compute or service redeployment, because each of the three datastores (CockroachDB, OpenSearch, Postgres) needs its own replication/backup mechanism, and hosted versions of two of them remove the operator's control over software version and installable plugins.conf: high
• explicitA deterministic-but-lossy UUID-to-scheme-ID mapping, if lost in a disaster because it existed only on the failed side, would permanently sever Form3's ability to tell customers what happened to in-flight payments; the chosen fix (asking customers to resend the IDs of their recently submitted payments) shifts the recovery burden onto the customer rather than eliminating the underlying data-loss risk.conf: medium
• explicitA critical piece of infrastructure's officially designed failover process (the payment scheme's manual, four-business-hour, email-a-Word-document IP-change process) can be effectively bypassed and automated (via BGP route publication/withdrawal) without needing the scheme's cooperation, process change, or knowledge.conf: high
• explicitThe decision to actually trigger a disaster-recovery failover during a live incident is not a purely technical calculation but a race against an unknowable quantity (how long the underlying outage will last), where both failing over and not failing over carry a real risk of making the outcome worse for the customer.conf: high
• explicitIn the actual October 2025 AWS incident, Form3 chose not to fail over, and in hindsight this was the right call because AWS recovered before a failover-plus-eventual-failback cycle would have finished — but this could not have been known by the team at decision time.conf: high
• implicitThe felt difficulty of the October 2025 non-failover decision — not advance planning — is what drove Form3's subsequent investment in shrinking RTO/RPO via asynchronous cross-cloud logical replication of CockroachDB and the NATS event stream.conf: medium
• explicitIntroducing asynchronous, bidirectional cross-cloud data replication for disaster recovery creates new correctness hazards for patterns that assume single-writer semantics: the outbox pattern's event-and-marker-row can be replicated and processed on the wrong cloud (silently producing no event), and long-running timers must be re-scoped to fire only where the owning customer is currently configured — both required deliberate redesign to preserve resource replication while suppressing side-effect replication.conf: high
• explicitA regulatory requirement that customers demonstrate disaster-recovery capability annually, in production, on a shared multi-tenant platform, forces DR design toward per-customer failover granularity (splitting NATS subjects and gateway configuration by customer identity) rather than whole-platform failover, so one tenant's DR test cannot disrupt others.conf: high
• explicitRunning a triple-active multi-cloud platform is genuinely more expensive than running on a single cloud, and the primary justification the speakers offer for that cost is market/customer demand, not a technical or risk-quantified cost-benefit argument.conf: high
  refinesPresentation: Clean Architecture for Serverless: Business Logic You Can Take Anywhere · p14 — extract-clean-architecture-for-serverless#p14 asserts multi-cloud deployability is 'itself worth' its architectural overhead; Form3's talk refines that claim with a more skeptical, ROI-driven framing — the overhead is worth it specifically because and only because a paying market demanded it, not as an intrinsic architectural good.
• explicitSplitting one large production environment into several smaller production environments can reduce total infrastructure cost, because pre-production environments must be scaled to at least match their corresponding production environment (plus projected growth), and matching several smaller productions in aggregate is cheaper than matching one very large one.conf: medium
• implicitThe regulatory framing that motivated the whole project ('banks must be able to exit a cloud') and the engineering solution actually delivered (tolerate any one cloud's temporary unavailability without customer-visible impact) address two different risks — vendor-portability/exit capability versus availability/uptime — and the talk never distinguishes them or demonstrates the former.conf: medium

Arguments

Narratives

• Kevin is on a train, opens a football club's usually high-production-value YouTube tactics-analysis channel, finds it glitching and degraded; reading the comments reveals a major Google Cloud outage is underway. He checks his work laptop expecting a crisis and instead finds only a low-level alert about pods crash-looping in GCP, with all payments still flowing via the other two clouds.
  → Provides the talk's only empirical (not just theoretical) validation that the 'treat a cloud as an availability zone' design actually works under a real, unplanned, multi-hour outage — answering the skeptical 'does it work?' question the speaker poses to the audience directly.
• In October 2025, two weeks after the US active-standby DR service went live, AWS suffers a major incident; Form3's VPN to the payment scheme goes down and payments stop flowing. The incident team, despite being experienced and biased to action, is stuck deciding whether to fail over to GCP without knowing how long the AWS outage will last. They choose not to fail over; AWS recovers; in hindsight this was the right call, but it 'didn't feel like it' in the moment.
  → Illustrates that the hardest part of disaster recovery is not the engineering that enables failover but the judgment call of whether and when to actually trigger it — a decision made under genuine uncertainty that can't be fully de-risked by architecture, and which motivated the team's subsequent RTO/RPO-reduction work.
• To build the failover decision heuristic ('is it likely AWS will take significantly longer to recover than our predicted failover time?'), Ross says he and 'my friend ChatGPT' went through the complete history of AWS incidents, reviewing outage duration and affected services, and concluded that in only one historical incident would failing over have actually helped.
  → Illustrates how the team tried to replace gut-feel with an empirical base rate for a decision that otherwise has almost no historical data points to learn from — but the casual framing ('my friend ChatGPT and I') gives no visibility into methodology, making it hard to assess how rigorous this historical analysis actually was.

Questions addressed

• ⭕ How do you satisfy a bank regulator's demand that banks 'must be able to exit a cloud' because of concentration risk?
  The talk answers a different, related question instead: how to keep a payments platform running without customer-visible impact if one cloud provider becomes unavailable. It never shows or claims an actual capability to permanently migrate off / exit a cloud vendor.
• ✅ How do you keep a strongly-consistent, Raft-based distributed database's replicas safely coordinated when they are spread across independent Kubernetes clusters that share no control plane and no built-in cross-cluster primitives?
  Solved via a custom cross-cloud DNS addressing convention (CoreDNS forwarding + static-IP load balancers) plus a custom cross-cluster pod disruption budget operator (X-PDB) that enforces the Raft quorum invariant across all three clusters.
• ✅ How do you keep infrastructure-as-code maintainable at scale once it's replicated across clouds, environments, and geographies?
  Solved by building a custom operator (CLO) that raises the unit of infrastructure change from 'per node pool' to 'per platform image', collapsing hundreds of PRs into one.
• ✅ How do you sell and architect disaster recovery for a market (the US) whose regulatory, latency, and mental-model constraints differ from the market the original architecture was designed for?
  Solved by deliberately building a different, simpler active-standby architecture matched to US customer expectations, accepting a non-zero RTO/RPO instead of replicating the triple-active design.
• ⭕ When, exactly, should an incident team actually pull the trigger on a disaster-recovery failover during a live, ongoing incident?
  A heuristic is offered (compare predicted failover time to the likely remaining duration of the incident), but the talk's own real-world example shows the team could not actually apply it with confidence in the moment, and the question is left as a fundamentally hard judgment call rather than something engineering resolves.
• ⭕ What happens if two of the three 'independent' cloud providers fail at the same time, whether by coincidence or through a shared upstream dependency (e.g. common DNS roots, BGP routing, certificate authorities)?

Gaps

• incompleteThe stated regulatory motivation is that 'banks must be able to exit a cloud' — implying vendor lock-in / migration / bargaining-power risk — but the delivered solution (simultaneous multi-cloud operation with automatic failover) addresses a different risk entirely (temporary unavailability of one vendor). The talk never distinguishes these two risks or demonstrates an actual cloud-exit capability (e.g., migrating a running workload permanently off one of the three providers).
  Read next: Read the UK regulator's original 2021 statement/policy on cloud concentration risk to check whether 'exit a cloud' was meant as continuity-during-outage or genuine long-term portability, and whether Form3's design would actually satisfy a literal cloud-exit requirement if a provider relationship had to be terminated entirely.
• incompleteNo discussion anywhere of correlated or shared-upstream failure modes across the three nominally independent cloud providers (root DNS, BGP routing, major CDNs, certificate authorities, common package registries) that could plausibly take out more than one cloud at once. This matters concretely here: the nine-pod CockroachDB Raft cluster only tolerates the loss of one full cloud (needs 5 of 9 alive); a correlated two-cloud event, never modeled in the talk, would be a full database outage.
  Read next: Look for Form3 engineering follow-ups or independent analysis addressing correlated multi-cloud failure risk; this is the same gap already flagged against Sam Newman's multi-cloud claims in extract-understanding-progressive-collapse.
• illogicalThe US active-standby failover runbook's top decision heuristic requires an accurate estimate of how much longer the ongoing incident will last relative to the predicted failover time — but the talk's own headline anecdote (the October 2025 AWS incident) demonstrates that this exact estimate is unavailable precisely when the decision must be made ('we don't know how long this underlying incident is going to last'). The heuristic is stated as actionable guidance while the narrative that follows shows it could not actually be applied with confidence.
  Read next: None needed — this is a structural tension worth flagging on its own terms rather than something to chase down externally.
• uninformedThe talk repeatedly asserts that triple-active multi-cloud 'is definitely more expensive' than single-cloud, and that splitting one large production environment into several smaller ones reduces cost, but gives no order-of-magnitude figure for either claim (no multiplier, no percentage, no dollar range), so a reader cannot judge whether the cost tradeoff argument in the 'when not to' section is proportionate advice or an unquantified hand-wave.
  Read next: Look for a Form3 engineering blog post, cost breakdown, or follow-up talk quantifying the multi-cloud cost premium.
• incompleteThe closing warning — don't attempt this 'unless you've got a really robust platform engineering practice and a really capable and trusted platform team' — is asserted without any concrete description of what that practice costs: headcount, time invested building and maintaining the two custom operators (X-PDB, CLO), or the ongoing on-call burden of running bespoke cross-cluster infrastructure. As an InfoQ writeup of a conference talk rather than the speakers' own detailed engineering post, the format itself likely selected/compressed the Q&A and technical depth available.
  Read next: The transcript references a companion talk by colleague 'Moritz' specifically on the X-PDB operator (linked via YouTube in the original talk) — that talk likely has the team-size/build-cost detail this one omits.

Closes gaps elsewhere

• Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments · g4 — That extraction flagged that every finding was confined to single-region ECS behavior and that multi-region/multi-cloud disaster-recovery failure scenarios for enterprise payment processors were entirely unaddressed. This document is exactly that missing case: a concrete, real-production multi-cloud (triple-active) and multi-region (active-standby) disaster-recovery design for a regulated payments processor, including an actual live-incident failover decision and its aftermath.

Related documents

• Presentation: Parting the Clouds: The Rise of Disaggregated Systems
• Presentation: Clean Architecture for Serverless: Business Logic You Can Take Anywhere
• AI Is Changing Software Engineering. The Fundamentals Still Matter.
  Platform Engineering SRE & Incident Response
• Article: Implementing Chaos Engineering in Financial Payment Systems: Lessons from Enterprise ECS Deployments
  Cascading Failure & System Resilience Patterns SRE & Incident Response
• Article: Modular Edge Computing at Multi-Tenant SaaS Scale on Cloudflare Workers
  Cloud Vendor Portability & Lock-in Avoidance Platform Engineering
• Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents
  Custom Infrastructure Operators (Extending Platform Primitives) Platform Engineering
• How a Culture of Data-Driven Conversations Can Support Platform Engineering
  Platform Engineering SRE & Incident Response
• Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs
  Custom Infrastructure Operators (Extending Platform Primitives) Platform Engineering
• Presentation: Understanding Progressive Collapse: How To Avoid A Cascading Failure
  Cascading Failure & System Resilience Patterns SRE & Incident Response
• Article: Implementing Durable Workflows on Postgres Without an External Orchestrator
  Cascading Failure & System Resilience Patterns SRE & Incident Response