Custom Infrastructure Operators (Extending Platform Primitives)
Building bespoke operators/controllers that extend a platform's native primitives (Kubernetes PDBs, node-pool management) to solve problems the platform doesn't natively support — e.g. spanning an invariant across cluster boundaries, or collapsing per-node-pool config into a single higher-level abstraction.
Also known as: custom kubernetes operator, cluster lifecycle operator, clo, x-pdb, cross-cluster pod disruption budget, extending built-in primitives
Part of: Platform Engineering
How authors use it
Oldest first (extraction date when no publication date is known).
| Author | Their term | Their meaning |
|---|---|---|
| Presentation: How To Run on Three Clouds at Once, and When Not To Ross McFarlane, Kevin Holditch · 2026-09-11 | X-PDB (cross-cluster pod disruption budget) | A custom, open-sourced Kubernetes operator that spans a single pod-disruption-budget invariant across n independent Kubernetes clusters, so that node rollouts in one cloud respect how many pods are already down in the others. |
| Presentation: How To Run on Three Clouds at Once, and When Not To Ross McFarlane, Kevin Holditch · 2026-09-11 | CLO (Cluster Lifecycle Operator) | An in-house operator that manages node-pool rollout policy (how many pools can roll at once, at what pace) at the level of 'what image should the whole platform run', collapsing per-node-pool, per-cloud, per-environment PRs into one PR per platform. |
Documents (4)
- Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents Olimpiu Pop · 2026-09-22 source ↗ Reader 🖥️
Google's AX is a Kubernetes-style, open-source control plane that treats AI agents as stateful, checkpointable actors on a purpose-built runtime (Agent Substrate) rather than as microservices or batch jobs, solving the idle-compute and cold-start economics of long-running agentic workloads at the price of inheriting full Kubernetes operational complexity — a trade-off the technical community is already visibly split on rather than one the article itself adjudicates. - Presentation: How To Run on Three Clouds at Once, and When Not To Ross McFarlane, Kevin Holditch · 2026-09-11 source ↗ Reader 🖥️
Form3 proves that a regulated payments platform can run truly active-active-active across three cloud providers by treating each cloud as nothing more than another availability zone and building custom Kubernetes operators to extend the platform's own primitives across cluster boundaries where they run out — but the same team's parallel US disaster-recovery build shows that this triple-active design is not a universal best practice: it is justified only when a specific market's resilience expectations, a mature platform-engineering practice, and a willingness to pay a real cost premium all line up, and even then the harder problem — deciding, mid-incident, whether to actually trigger a failover — remains an unresolved judgment call under uncertainty rather than something engineering can fully automate away. - Presentation: Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server Felipe Huici · 2026-09-09 source ↗ Reader 🖥️
The historical trade-off between isolation strength and speed/density in cloud infrastructure was never intrinsic to the hypervisor boundary itself — it was an engineering artifact distributed across the whole platform chain (proxy, controller, VMM, snapshot storage, host-OS resource limits) — and once each link is rebuilt for millisecond latency, snapshot-based stateful scale-to-zero turns the (still gold-standard) VM isolation boundary into the highest-density primitive available for bursty, mostly-idle agent and sandbox workloads. - Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs Leela Kumili · 2026-09-07 source ↗ Reader 🖥️
Reasoning about autoscaling at the level of individual dataflow operators (via true processing rate estimated from throughput and busy time) rather than the whole cluster is a strictly better model for scaling complex, heterogeneous, stateful streaming pipelines — good enough that Netflix, despite years of investment in its own cluster-level autoscaler, is migrating 30,000+ jobs to the open-source Apache Flink Autoscaler and building new internal control-plane infrastructure around it rather than continuing to extend its bespoke system.
Propositions touching this concept (2)
By publication month, oldest first (extraction month when no publication date is known).
2026-09
- explicitForm3 built and open-sourced a custom Kubernetes operator (the cross-cluster pod disruption budget, or 'X-PDB') that spans a single disruption-budget invariant across n independent clusters, closing the quorum-safety gap the built-in primitive left open.
- explicitcorroboratedBuilding a custom operator (the Cluster Lifecycle Operator, CLO) that manages node pools at a higher level of abstraction — letting an engineer specify one target image for an entire platform — collapsed what had been hundreds of pull requests into one PR per platform.
Open gaps (5)
- illogicalThe virtual kubelet always reports pod status as 'running' regardless of actual suspend state, on the premise that sub-perception latency makes the lie harmless. But Kubernetes's own HA and self-healing guarantees — which the presenter explicitly cites as reasons people value Kubernetes — depend on accurate liveness/readiness signals; a control plane fed a false world-model cannot correctly diagnose a genuinely stuck or crashed VM versus one that is merely asleep, undermining exactly the properties (self-healing, HA) the talk claims to preserve.Presentation: Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server — read next: Consider how this tension is handled (or not) in Unikraft's own documentation or in AWS/Azure's competing persistent-agent-runtime products (see extract-multi-agent-collaboration-bedrock-agentcore), which appear to expose real state rather than fabricate it.
- illogicalThe article implicitly treats 'the technical community is divided' as itself validating AX's significance, when a distinct and more useful move would be to disentangle two separate questions it conflates by reporting them together: whether the underlying actor-substrate genuinely solves a real cost problem (an architecture question), versus whether Google's 'ergonomic workflows' marketing claim is accurate (a rhetoric question). A platform can be architecturally sound and still be overmarketed as easy to adopt — the piece never separates these.
- uninformedThe talk repeatedly asserts that triple-active multi-cloud 'is definitely more expensive' than single-cloud, and that splitting one large production environment into several smaller ones reduces cost, but gives no order-of-magnitude figure for either claim (no multiplier, no percentage, no dollar range), so a reader cannot judge whether the cost tradeoff argument in the 'when not to' section is proportionate advice or an unquantified hand-wave.Presentation: How To Run on Three Clouds at Once, and When Not To — read next: Look for a Form3 engineering blog post, cost breakdown, or follow-up talk quantifying the multi-cloud cost premium.
- incompleteNo discussion of the actual engineering time/cost required to build the Spring Boot/Temporal control plane and the JobManager modifications, so a reader can't weigh the reported savings against the cost of achieving them, or judge how replicable this is without Netflix-scale platform-engineering resources.Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs — read next: Netflix Tech Blog engineering-effort retrospectives, or a case study from a company with fewer platform-engineering resources attempting a similar migration.
- incompleteThe article names two known unresolved issues (FORWARD-connection redistribution, the FLINK-38538 busy-operator ratio problem) but doesn't say how often they manifest in Netflix's own fleet, or what operational workaround Netflix uses beyond general subgraph preservation.Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs — read next: Apache Flink JIRA/dev-list threads tracking FLINK-38538, and any future Netflix follow-up on operational incidents from the new autoscaler.