Beyond Kubernetes at Modal: How to Scale 1 Million Concurrent Sandboxes in Seconds
Sergio De Simone · rss · 2026-09-23 Reader 🖥️
Modal replaced Kubernetes-style centralized, strongly-consistent orchestration with a decentralized architecture — per-worker sources of truth, a parallel fleet of scheduling servers acting like load balancers, and a single Redis stream as the only shared bottleneck — to run over a million concurrent AI sandboxes, demonstrating that extreme-scale, millisecond-latency AI infrastructure requires walking away from, rather than extending, Kubernetes' coordination model.
theoretical · mixed — Primarily expository/explanatory: an InfoQ news piece reporting, in paraphrase and block quotes, how Modal's staff engineers diagnosed Kubernetes' scaling limits and redesigned around them. It carries practical design principles (horizontal-scalability-by-default, worker-as-source-of-truth) a reader could directly apply, and layers on argumentative commentary from two outside voices (Dowling, Jones) drawing broader interpretive conclusions the original Modal post doesn't itself make.
Structure
| Part | Role |
|---|---|
| Framing: Modal's staff engineers describe rebuilding sandbox infrastructure to support millions of concurrent sandboxes and tens of thousands of creations per second. | context |
| Diagnosis: traditional orchestration (Kubernetes) struggles at this scale because of centralized coordination and strongly consistent state — etcd load and scheduler load both grow with nodes/pods, and etcd is not natively shardable. | evidence / problem statement |
| Design principle: everything O(sandboxes) or O(nodes) must be horizontally scalable by default; sandbox creation must be simple; everything else is secondary. | thesis statement of Modal's approach |
| Mechanism: workers become their own source of truth; a fleet of parallel scheduling servers replaces one serialized scheduler; placement is negotiated via RPC accept/reject rather than centrally locked. | evidence — how the principle is implemented |
| Residual bottleneck: all workers publish state to a single Redis stream, load-tested viable past 100,000 workers. | qualification / limit |
| Benchmark: 1 million sandboxes created in under a minute, median startup-to-code time under 0.5 seconds. | evidence — proof of the claim |
| External commentary: Dowling on iterative difficulty at each order of magnitude; Jones on the architectural significance and a predicted split between an execution plane and a coordination plane. | interpretation / broader conclusion |
| Positioning: Modal is grouped with Unikraft, Google Substrate, and Overdrive as part of a broader 'rebuild the cloud' trend. | context / generalization |
Key terms
- sandbox Sandboxing
An isolated, fast-starting execution environment for AI/agent-triggered code, engineered to be created and torn down at the scale of millions of concurrent instances and tens of thousands of creations per second. - horizontally scalable by default Post-Kubernetes / Decentralized Orchestration at Scale
Modal's governing design constraint: any operation whose load is O(sandboxes) or O(nodes) must be able to scale out across independent servers rather than route through a single coordinating component. - source of truth (per worker)
Each compute worker is authoritative for its own state, rather than a central durable store (like etcd) being authoritative for cluster-wide state. - scheduling as load balancing
Replacing a single, serialized, globally-consistent scheduling decision with parallel scheduling servers that each independently pick a worker and negotiate placement via direct RPC, with the worker free to reject if it lacks capacity. - execution plane vs. coordination plane
Alex Jones's distinction (quoted, not Modal's own term): the execution plane is fast, millisecond-latency, isolated code execution (what Modal built); the coordination plane is shared memory and overlapping security boundaries for multi-agent workflows (what Kubernetes-shaped systems remain suited for).
Propositions
- explicitModal rebuilt its sandbox infrastructure from the ground up to support millions of concurrent sandboxes and tens of thousands of sandbox creations per second.conf: high
- explicitTraditional container orchestration systems like Kubernetes struggle to operate at million-sandbox scale because they rely heavily on centralized coordination and strongly consistent state.conf: highextendsPresentation: Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server · p6 — Identifies a complementary bottleneck at the same order of magnitude: where Unikraft hit host-kernel-level limits (TAP devices, bridge ports) on a single box, Modal hit cluster-coordination-level limits (etcd, centralized scheduling) across many boxes — together mapping two distinct bottleneck layers of million-instance sandbox scale.
- explicitIn Kubernetes, both the scheduling algorithm's load and etcd's load grow with the number of nodes and pods, and etcd is not natively shardable within a keyspace, so high pod-creation or pod-churn rates create serious issues.conf: high
- explicitOvercoming Kubernetes' centralized-coordination scaling limits is feasible in principle but would require "serious work" — rewriting or replacing etcd and parallelizing the scheduling algorithm.conf: high
- explicitModal's core design principle was that everything with O(sandboxes) or O(nodes) load must be horizontally scalable by default, the sandbox-creation path should be as simple as possible, and everything else is secondary.conf: high
- explicitModal replaced global coordination with a load-balancing-style model: each worker is its own source of truth rather than a central datastore, and a fleet of parallel scheduling servers, rather than one serialized scheduler, makes placement decisions.conf: high
- explicitA Modal scheduling server, once it picks a worker, contacts that worker directly by RPC to request sandbox creation; the worker accepts if it has free resources or rejects otherwise, so placement is negotiated optimistically rather than centrally locked.conf: high
- explicitThe resulting architecture's only remaining bottleneck is that all workers publish their state to a single Redis stream, but Modal's load testing suggests this remains viable to well over 100,000 workers.conf: high
- explicitIn Modal's benchmark, the system created 1 million sandboxes in under a minute, with median startup-to-running-code time under 0.5 seconds.conf: highextendsPresentation: Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server · p7 — Where Unikraft/Fly's headline '1 million' figure turned out to be idle, scaled-to-zero VMs on a single box, Modal's reported 1M-sandbox benchmark claims genuine concurrent creation with sub-second startup across tens of thousands of nodes — a different, distributed-horizontal path to the same headline number, and the same scrutiny about what 'concurrent' actually measures should apply.extendsGoogle Announces GKE Agent Sandbox and Hypercluster at Next '26 · p4 — Corroborates, in Modal's sandbox-scheduling context, the same finding GKE's hypercluster piece makes about chip-scale control planes: that a single centralized control plane becomes the blast-radius/change-management risk at million-scale AI infrastructure.
- implicitGenuine horizontal scalability at this order of magnitude requires giving up strong, globally-agreed cluster-state consistency in favor of local/eventual consistency plus optimistic, reject-capable placement.conf: medium
- implicitAI-agent infrastructure's coordination requirements are splitting into two structurally different problems — a low-latency execution plane (isolated sandbox creation) and a stateful coordination plane (shared memory/security boundaries for multi-agent workflows) — that likely need different, decoupled systems rather than one platform serving both.conf: medium
- implicitcontestedKubernetes' architecture is not merely under-optimized but structurally mismatched to GenAI infrastructure's requirements at this scale — which is why Modal built a new system instead of extending Kubernetes, even though extending it was described as feasible.conf: mediumcontradictsGoogle Announces GKE Agent Sandbox and Hypercluster at Next '26 · p1 — Modal's engineers' framing — that reaching this scale on Kubernetes would require 'serious work' rewriting core parts like etcd, and that they ultimately abandoned Kubernetes rather than extend it — pushes back on the more confident claim that Kubernetes has simply 'become the operating system for the AI era,' at least at the extreme end of AI-agent-sandbox scale.
Arguments
Modal achieved horizontal scaling to millions of concurrent sandboxes because it replaced Kubernetes' centralized, strongly-consistent coordination model with a decentralized, load-balancing-style scheduling architecture. strength: high
Explicit premises:
- Kubernetes' scheduling and etcd load both grow with node/pod count, and etcd is not natively shardable.
- Modal made every O(sandboxes)/O(nodes) operation horizontally scalable by default.
- Each worker holds its own state as source of truth instead of a central datastore.
- A parallel fleet of scheduling servers replaces a single serialized scheduler.
- Placement is negotiated via RPC accept/reject rather than centrally committed.
Implicit premises:
- Centralized, strongly-consistent state stores (like etcd) are the fundamental scaling bottleneck of orchestration systems, not merely an implementation detail that could be tuned away.
- A system where a placement request can be rejected and retried is an acceptable trade-off against the certainty of centrally-committed scheduling, at this scale.
- The single remaining bottleneck — the Redis state-publishing stream — is qualitatively different from Kubernetes' etcd bottleneck, presumably because it sits in the observability/state-propagation path rather than the decision path, so contention there doesn't block progress the way etcd writes do.
Inferences:
- Any orchestration system that keeps a single strongly-consistent store in the decision (not just observation) path will hit the same wall Modal diagnosed in Kubernetes, regardless of how well the store itself is tuned.
- The architecture trades a small rate of failed/rejected placements (and retries) for horizontal scalability — implying a latency tail behind the reported median that the article never discloses.
The industry is heading toward decoupling the 'execution plane' (fast, isolated sandbox creation, which Modal built) from the 'coordination plane' (shared memory/security boundaries for multi-agent workflows, which remains Kubernetes-shaped systems' turf). strength: low
Explicit premises:
- Modal succeeded by walking away from Kubernetes rather than trying to extend it.
- Modal's achievement is described as 'the first credible signal that Kubernetes isn't adapting fast enough' for GenAI infrastructure needs.
Implicit premises:
- Execution (isolation boundaries appearing in milliseconds) and coordination (shared state/security boundaries across multiple agents) are separable concerns that need not be solved by the same platform.
- Kubernetes' architecture remains well-suited to the coordination plane's needs even though it is unsuited to the execution plane's needs — i.e., its limitations here are scale/latency-specific, not general.
Inferences:
- If this decoupling holds, future AI infrastructure stacks will likely combine a Modal-like execution layer with a Kubernetes-like coordination layer via some integration boundary, rather than one platform replacing the other outright — a prediction the article surfaces but never spells out or tests.
Narratives
- The single Redis stream that all workers publish state to is the system's one remaining bottleneck, empirically load-tested to remain viable past 100,000 workers.
→ Functions as a 'how far did they get' case example supporting the horizontal-scaling claim — it shows the architecture wasn't bottleneck-free, but that the residual bottleneck was pushed far past the point where Kubernetes' etcd-driven wall bites, lending credibility without overclaiming perfection. - The benchmark of creating 1 million sandboxes in under a minute with sub-0.5-second median startup time.
→ Serves as the article's headline proof-narrative, making the abstract architectural redesign concrete and quotable — the 'showing' that substitutes for a rigorous, independently-verified performance study.
Questions addressed
- ✅ Why does Kubernetes-style container orchestration break down at million-sandbox / tens-of-thousands-of-creations-per-second scale?
Centralized coordination and strongly consistent state (etcd) create load proportional to node/pod count, and etcd is not natively shardable, so high creation/churn rates hit a wall. - ⭕ How can a scheduling system remain correct and reliable without a centrally consistent view of cluster state?
- ⭕ What are the failure modes and cost of the accept/reject RPC scheduling protocol — retry rates, latency tail, wasted scheduling-server attempts — at scale?
Gaps
- incompleteThe article reports a median startup-to-code time under 0.5 seconds for the 1M-sandbox benchmark but gives no tail latency (p95/p99), no data on RPC rejection/retry rates under the fleet-of-schedulers model, and no cost or resource-utilization figures — so the headline numbers can't be weighed against the failure-mode costs implied by the optimistic accept/reject protocol.
Read next: Read Modal's original engineering blog post (linked in the article: modal.com/blog/scaling-to-1-million-concurrent-sandboxes-in-seconds) for full methodology, or look for an independent benchmark/reproduction. - incompleteNeither the article nor the quoted commentary explains how the architecture avoids classic decentralized-scheduling failure modes — e.g. multiple scheduling servers racing to place sandboxes on the same worker before state propagates through the Redis stream, or a worker's state going stale between the moment a scheduling server queries it and the RPC placement request.
Read next: Look for a systems paper or engineering post detailing the consistency/staleness model; the corpus's Unikraft microVM-scaling extraction addresses comparable per-VM resource-limit questions at similar scale and is a useful point of comparison. - incompleteThe article surfaces Alex Jones's prediction of an execution-plane/coordination-plane split in AI infrastructure but doesn't examine what integration mechanism would let a Modal-like execution layer interoperate with a Kubernetes-like coordination layer — a concrete architecture question other corpus pieces (GKE Agent Sandbox, virtual-kubelet-based integrations) only partially address.
Read next: Read the GKE Agent Sandbox hypercluster extraction and the Unikraft/1M-sandbox extraction's virtual-kubelet section for how others are gluing decentralized execution back into Kubernetes' coordination plane.
Closes gaps elsewhere
- Google Announces GKE Agent Sandbox and Hypercluster at Next '26 · g6 — Jones's execution-plane/coordination-plane framing (quoted in this article) directly addresses whether Kubernetes-native sandbox primitives (Sandbox, SandboxTemplate, SandboxClaim) are the right abstraction — arguing no: execution should decouple from Kubernetes-shaped coordination entirely rather than being modeled as another Kubernetes primitive. This doesn't settle the question but supplies the outside argument the GKE piece's gap was missing.