Netflix Moves Toward Open Source Flink Autoscaler for 30,000+ Streaming Jobs
Leela Kumili · rss · 2026-09-07 source ↗ Reader 🖥️
Compute & Energy InfrastructureCustom Infrastructure Operators (Extending Platform Primitives)Infrastructure EfficiencyPlatform EngineeringVendor Self-Reported Metrics
Reasoning about autoscaling at the level of individual dataflow operators (via true processing rate estimated from throughput and busy time) rather than the whole cluster is a strictly better model for scaling complex, heterogeneous, stateful streaming pipelines — good enough that Netflix, despite years of investment in its own cluster-level autoscaler, is migrating 30,000+ jobs to the open-source Apache Flink Autoscaler and building new internal control-plane infrastructure around it rather than continuing to extend its bespoke system.
practical · expository — A short technical news report that, while descriptive in form (recounting a historical progression from 2019 to now), functions practically: it implicitly argues that operator-level, true-processing-rate autoscaling is architecturally superior to cluster-level autoscaling for heterogeneous stateful streaming jobs, and that this superiority is now strong enough to justify migrating a mature, home-grown system to an open-source alternative.
Structure
| Part | Role |
|---|---|
| History and limitation of Netflix's original (2019) cluster-level autoscaler built on Mantis/Atlas telemetry | context establishing the problem |
| Description of the community Apache Flink Autoscaler's operator-level mechanism (true processing rate, per-vertex parallelism, FLIP-271) and its DS2/True-Processing-Rate research lineage | evidence for why the new approach works |
| How Netflix integrated the OSS autoscaler into its own control plane (Spring Boot + Temporal) and the scale-specific modifications required (3,000-subtask metric collection, server-side filtering, forward-subgraph preservation, sink-backpressure handling) | evidence of production maturity and the real cost of adoption |
| Known unresolved issues (FORWARD-connection redistribution, FLINK-38538 busy-operator ratio problem) | caveat / limits of the approach |
| Reported results (58% cost reduction, ~$1.1M/year for one team) and stated future plans (full migration, Flink 2 disaggregated state investigation) | conclusion / payoff and forward-looking open question |
Key terms
- True Processing Rate
An operator's actual sustainable processing capacity, estimated from measured throughput and busy time, used as a simpler alternative to critical-path analysis for computing the parallelism an operator needs. - cluster-level vs. operator-level autoscaling
Cluster-level: one scaling decision applied to the total TaskManager count for an entire job. Operator-level: an independent parallelism decision computed per vertex in the job graph. - forward connected subgraph
A chain of Flink operators linked by a FORWARD connection (no shuffle between them), which Netflix's implementation deliberately keeps together during rescaling to avoid unnecessary redistribution. - utilization target
The configured busy-time threshold (Netflix: 0.45) at which the autoscaler decides an operator needs more/less parallelism — a knob that trades off resource efficiency against how often (and how disruptively) a job gets rescaled. - control plane Platform Engineering
Netflix's own orchestration layer (a Spring Boot service using Temporal workflows) that wraps the OSS autoscaler and isolates per-job scaling decisions, as opposed to deploying the autoscaler directly via the Flink Kubernetes Operator.
Propositions
- explicitNetflix's original (circa 2019) cluster-level Flink autoscaler, which scaled the total TaskManager count using cluster-wide Atlas telemetry, reduced resource usage 25-45% across thousands of pipelines but became less suitable as stateful pipelines grew more heterogeneous (branches, joins, terabytes of state).conf: high
- explicitThe Apache Flink Autoscaler estimates each operator's true processing rate from throughput and busy time and computes required parallelism per vertex by walking the job graph, rather than making one scaling decision for the whole cluster.conf: high
- explicitOne Netflix team reduced annualized Flink compute expenditure by 58% (approximately $1.1 million/year) after adopting the operator-level autoscaler.conf: medium
- explicitNetflix set its utilization target at 0.45, below Flink's community default of 0.7, specifically to reduce aggressive rescaling of large stateful jobs.conf: high
- explicitNetflix integrated the open-source autoscaler with its own internal control plane — a Spring Boot service using Temporal workflows to isolate per-job autoscaling decisions — rather than deploying it directly through the Flink Kubernetes Operator.conf: high
- explicitMaking the OSS autoscaler work at Netflix's scale required modifying JobManager metric collection to support jobs with up to 3,000 subtasks, adding server-side metric filtering, preserving forward-connected subgraphs during scaling, and adding sink-backpressure handling.conf: high
- explicitKnown limitations remain unresolved in the operator-level approach: changing parallelism across FORWARD-connected operators can still require redistribution in general Flink usage, and an open issue (FLINK-38538) shows output-ratio-based scaling decisions can misjudge busy operators.conf: high
- explicitNetflix plans to migrate its remaining internal autoscaling use cases to the open-source implementation, and is separately investigating Flink 2's disaggregated state architecture to address the cost of state recovery during rescaling.conf: high
- implicitAdopting an 'industry-standard' open-source autoscaler is not a low-cost path for an organization at Netflix's scale: it required comparable integration engineering to what a bespoke system would have needed, just redirected toward extension and control-plane wrapping rather than from-scratch design.conf: mediumsupportsPresentation: How To Run on Three Clouds at Once, and When Not To · p10 — Same general pattern in a different domain: extending/wrapping a platform's native primitive with custom control-plane engineering (Form3's Cluster Lifecycle Operator; Netflix's Spring Boot/Temporal control plane) rather than adopting the primitive as-is.
- implicitNetflix judges that the long-term maintenance cost of a bespoke autoscaler that has fallen behind the state of the art (community research, ongoing FLIP-271 work, future Flink 2 features) outweighs the sunk cost of its 2019 system and the switching cost of migration.conf: medium
- implicitNetflix's choice of a lower utilization target (0.45 vs. the 0.7 default) is a specific instance of a general reliability-vs-efficiency tradeoff: deliberately running below the 'efficient' threshold to buy stability against expensive, disruptive rescale events.conf: highsupportsPresentation: How Netflix Shapes our Fleet for Efficiency and Reliability · p3 — Netflix's 0.45-vs-0.7 utilization target is a concrete instance of fleet-shaping's general claim that low utilization is a deliberate reliability tradeoff, not waste.
- implicitThis operator-level autoscaler functions as a minute-timescale layer of automated feedback control within Netflix's broader capacity-management practice, distinct from slower (monthly) capacity planning and faster (seconds-scale) load-shedding mechanisms used elsewhere in Netflix's infrastructure.conf: mediumextendsPresentation: How Netflix Shapes our Fleet for Efficiency and Reliability · p9 — Names a specific real system (the Flink operator-level autoscaler) occupying the 'minutes' timescale layer of the layered feedback-loop model fleet-shaping describes abstractly.
Arguments
Operator-level autoscaling (using true processing rate) is architecturally better than cluster-level autoscaling for complex, heterogeneous, stateful streaming pipelines. strength: medium
Explicit premises:
- Cluster-level autoscaling forces all operators in a job to share the same scaling decision.
- Stateful pipelines with branches, joins, and terabytes of state have operators with genuinely different processing requirements.
- The operator-level autoscaler computes per-vertex required parallelism from each operator's measured true processing rate.
Implicit premises:
- A scaling decision that matches actual per-operator resource needs wastes less compute than one that must average across heterogeneous operators.
- Reduced compute waste translates into lower cost without a compensating loss in throughput, latency, or state-recovery guarantees.
Inferences:
- If operator-level autoscaling generalizes as reported, cluster-level autoscalers are likely the wrong default for any stateful pipeline with meaningfully heterogeneous operators, not just Netflix's specific workloads.
Migrating to, and building custom control-plane infrastructure around, an open-source community-maintained autoscaler is the right long-term choice for Netflix, even though it required substantial internal engineering to adapt. strength: medium
Explicit premises:
- Netflix modified JobManager metric collection, added server-side metric filtering, preserved forward-connected subgraphs, and added sink-backpressure handling to make the OSS autoscaler work at its scale.
- Netflix plans to migrate its remaining internal autoscaling use cases to the open-source implementation.
Implicit premises:
- A community-maintained approach that keeps improving (ongoing FLIP-271 work, Flink 2 disaggregated state) will outpace a bespoke internal system's rate of improvement over time.
- The cost of integration engineering is a largely one-time or amortizing cost, whereas maintaining a fully bespoke autoscaler indefinitely is an ongoing cost that diverges further from best practice each year.
Inferences:
- Netflix's original 2019 autoscaler and its Mantis/Atlas-based telemetry pipeline may eventually be fully retired, devaluing that accumulated institutional investment.
Narratives
- The history of Netflix's own autoscaler — built around 2019 on Mantis/Atlas telemetry, reducing resource usage 25-45% — followed by the discovery of its scaling-unit limitation as pipelines grew more heterogeneous, followed by adoption of the newer operator-level approach.
→ Implicitly argues that even a successful, long-running internal system can become the wrong architecture as workload complexity grows, and that recognizing this — rather than continuing to invest in extending the original design — is the right response. - Vasiliki Kalavri's account of the DS2 project initially exploring critical-path analysis before adopting True Processing Rate as a simpler baseline, which 'worked really well' and subsequently became part of Flink's autoscaling work.
→ Implicitly supports the value of empirically testing a simple approach against a more complex one before committing to complexity — a preference for simplicity that turned out to generalize into production use, and a case of academic streaming-systems research (an OSDI paper) flowing directly into widely-deployed infrastructure.
Questions addressed
- ✅ How can a company autoscale complex, stateful streaming pipelines with heterogeneous per-operator resource needs, when cluster-level autoscaling forces one scaling decision for the whole job?
Scale at the operator level using true processing rate estimated from throughput and busy time, walking the job graph to compute per-vertex parallelism (FLIP-271). - ✅ How do you integrate a general-purpose open-source autoscaler into an existing internal control-plane architecture and scale it to jobs with thousands of subtasks?
Build a dedicated control-plane service (Spring Boot + Temporal) to isolate per-job decisions, and modify JobManager metric collection and filtering to handle up to 3,000 subtasks. - ⭕ How do you avoid destructive over-rescaling of large stateful jobs, where each rescale event is itself expensive due to state recovery?
Partially addressed by lowering the utilization target (0.45 vs. 0.7 default) and preserving forward-connected subgraphs to reduce unnecessary redistribution; the deeper problem of state-recovery cost during rescaling is explicitly left as future work (Flink 2 disaggregated state). - ⭕ Does the reported 58% / $1.1M-per-year saving generalize beyond the one team it is reported for, and under what baseline or methodology was it measured?
Gaps
- incompleteThe 58% cost reduction and $1.1M/year savings figure is reported for a single, unnamed team with no baseline period, no measurement methodology, and no indication of whether it is representative of the other ~30,000 jobs or a best case.
Read next: The underlying Netflix Tech Blog post ('A Tale of Two Flink Autoscalers') for methodology detail, and any follow-up reporting on fleet-wide savings once the full migration completes. - incompleteNo discussion of the actual engineering time/cost required to build the Spring Boot/Temporal control plane and the JobManager modifications, so a reader can't weigh the reported savings against the cost of achieving them, or judge how replicable this is without Netflix-scale platform-engineering resources.
Read next: Netflix Tech Blog engineering-effort retrospectives, or a case study from a company with fewer platform-engineering resources attempting a similar migration. - incompleteThe article names two known unresolved issues (FORWARD-connection redistribution, the FLINK-38538 busy-operator ratio problem) but doesn't say how often they manifest in Netflix's own fleet, or what operational workaround Netflix uses beyond general subgraph preservation.
Read next: Apache Flink JIRA/dev-list threads tracking FLINK-38538, and any future Netflix follow-up on operational incidents from the new autoscaler. - uninformedNo engagement with or comparison to operator-level/graph-aware autoscaling approaches outside the Flink ecosystem (e.g. how Spark Structured Streaming, Kafka Streams, or managed cloud stream processors handle the same heterogeneous-operator scaling problem), so a reader can't judge how distinctive the Flink/DS2 approach actually is.
Read next: Comparative surveys of stream-processing autoscaling approaches across engines.
Related documents
- Presentation: Fixing the AI Infra Scale Problem by Stuffing 1M Sandboxes in a Single Server
- Presentation: From Fab To Token - The State Of The Market
- Presentation: How Netflix Shapes our Fleet for Efficiency and Reliability
- The Tokens You Can't Wait For
- Why AI agents need a Context Lake
- Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents
- The Hidden Storage Tax on Every AI Conversation
- Presentation: How To Run on Three Clouds at Once, and When Not To
- Meta Deploys Unified AI Agents to Automate Performance Optimization at Hyperscale