The Pulse: a new trend, smart model routing
The Pragmatic Engineer (Gergely Orosz) · email · 2026-07-03 Reader 🖥️
A new category of 'intelligent' model-routing tools is emerging to let engineering organizations cut AI spend by automatically directing each task to the cheapest model capable of handling it, and — because cheaper open models are now considered sufficient for a large share (~60%) of coding work — this routing capability is on track to become table stakes across all AI vendors.
practical · mixed — Despite the 'The Pulse' framing, this document is NOT a multi-item newsletter — it is a free bonus excerpt covering exactly one of four topics from a prior paid Pulse issue. It is a survey (expository) of the smart-model-routing vendor landscape, wrapped in an argumentative forecast that routing will become table stakes. Classed practical because the whole framing is instrumental: helping engineering leaders act on AI-cost control by adopting a routing layer.
Structure
| Part | Role |
|---|---|
| Problem framing: a head of engineering wishes for an 'intelligent router', motivated by 10-20x token-price variance between cheap and state-of-the-art models | context / motivation |
| Survey of dedicated router vendors (Factory Router, Not Diamond, Prism/Augment, Morph Model Router, Weave) | evidence (existence proof) |
| Survey of AI gateways with routing built in (OpenRouter, Kilo, Requestly.ai, LiteLLM, Envoy AI Gateway) | evidence / taxonomy |
| IDE built-in 'Auto' modes (Cursor, GitHub Copilot) with critical caveats | evidence + qualifying nuance |
| Expert testimony from Matan Grinberg (CEO, Factory AI) on off-the-charts enterprise demand and open-model sufficiency (~60% of coding work) | supporting testimony |
| Forecast: intelligent routing will become table stakes; all vendors will build it and new vendors will enter | conclusion |
Key terms
- intelligent router / smart model routing Model Routing
A layer or tool that automatically selects the right (typically cheapest-capable) LLM for each task, prompt, or session in order to optimize cost while preserving output quality - AI gateway Model Routing
A proxy layer through which a workplace funnels its LLM API calls; increasingly ships with routing that can pick models by cost, latency, and availability - open models / hosted open models Open-Weight Models
Open-weight models (as opposed to proprietary frontier models), claimed to be sufficient for roughly 60% of coding work by token spend and to be strictly rising in usage - table stakes
A baseline feature every vendor will be expected to have; the forecast endpoint for routing
Propositions
- explicitcorroboratedToken prices vary greatly across models — there can easily be a 10-20x difference between a cheap, average model and a state-of-the-art one.conf: highsupports ←Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months · p2 — The ~9x US-over-Chinese premium is a concrete instance of the documented 10-20x cross-model price spread.
- explicitdevelopedMultiple dedicated vendors now offer automatic model routing (Factory, Not Diamond, Prism, Morph, Weave), with vendor-claimed cost savings of roughly 20-30%.conf: highrefines ←Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS · p4 — Where the Pragmatic Engineer piece reports vendor-claimed savings of 20-30% in general terms, this document supplies a concrete, sourced number (RouteLLM's published 85% savings / 95% quality at 26% escalation) — but immediately refines it by showing the figure is benchmark-pair-specific and non-transferable, sharpening the earlier piece's implicit warning about vendor-sourced figures.extends ←Portal by Spotify cut my Claude Code token usage by 90% · p4 — Reports a much higher (~90%) savings figure than the 20-30% vendor-claimed general routing savings cited there, but for a narrower, precisely-defined delegation category (I/O-heavy bulk reads and boilerplate) rather than general-purpose per-request model routing.
- explicitRouting is increasingly bundled into AI gateways (OpenRouter, Kilo, Requestly, LiteLLM, Envoy) that select models by cost, latency, and availability.conf: high
- explicitCursor's Auto mode captures any routing savings for Cursor rather than passing them to customers, via a fixed-price model.conf: high
- explicitestablishedOpen models are often sufficient, and hosted open models are estimated to be adequate for around 60% of coding-related work by token spend, with open-model usage strictly increasing over the last six months.conf: mediumrefines ←AI Costs Are Surging and the Cheap Model Fix Might Not Last · p2 — Complicates the prior piece's economic engine (open models sufficient for ~60% of coding work, underwriting routing savings) by arguing the underlying token-cost pressure is a frontier-lab provisioning/compute problem, not an absence-of-cheap-alternatives problem — so losing the cheap-model option doesn't remove the savings opportunity's true cause, just one blunt tool for chasing it.supports ←Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months · p6 — Open models sufficient for ~60% of coding work with usage strictly increasing corroborates the open-weight adoption surge.supports ←GitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · p12 — Routing background/default work to a local or open model presupposes open models suffice for a large share of coding work — the Pulse '~60% by token spend' estimate.supports ←Agentics / Tech Things: Tokenmaxxing is dead, long live tokenmaxxing · p8 — The cost-per-iteration case for running cheap open models in loops reinforces the finding that open models are sufficient for a large (~60%) and growing share of coding work by token spend.
- explicitcorroboratedIntelligent routing will become table stakes, with essentially all AI vendors building a version and many new vendors entering.conf: mediumextends ←AI Costs Are Surging and the Cheap Model Fix Might Not Last · p7 — Extends the prior piece's 'routing becomes table stakes' forecast with a second dimension: routers will also select models by regulatory/risk exposure, not just cost — table stakes comes to include governance, not only cost optimization.supports ←GitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · p10 — ccr is a concrete open-source instance of exactly the routing layer the Pulse piece predicts will become table stakes.
- implicitdevelopedThe value proposition of routing is framed almost entirely as cost reduction, not quality improvement — the goal is to spend less while keeping quality merely 'good enough'.conf: highrefines ←GitHub Copilot's Project HydraFusion Promises Frontier Level Performance Through Multi-Model Routing · p6 — The earlier routing-vendor coverage found the category's value proposition framed almost entirely as cost reduction with quality merely 'good enough'; HydraFusion is a counter-example within the same category, explicitly marketed on quality parity-or-better ('frontier level performance') alongside cost cuts, so the generalization needs qualifying rather than being false.
- implicitcorroboratedFrontier (state-of-the-art) models are over-used relative to what most tasks actually require; the majority of coding work does not need the best model.conf: highsupports ←Portal by Spotify cut my Claude Code token usage by 90% · p13 — Independently arrives at the same diagnosis — that frontier models are routinely used on work that doesn't require them — from one practitioner's own Claude Code usage rather than from an industry-wide survey.
- implicitA router built by a party that profits from model usage (e.g., an IDE reselling tokens) has an incentive conflict — it may optimize for its own margin rather than the customer's savings.conf: medium
- implicitcorroborated'Which model to use' is shifting from a deliberate engineering decision to a commoditized, automatable one.conf: mediumsupports ←Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months · p5 — Capability parity moves 'which model to use' off benchmarks and toward a commoditized, cost-driven choice.
Arguments
Intelligent model routing will become table stakes across the AI tooling market. strength: medium
Explicit premises:
- Enterprise demand is 'off the charts' (Grinberg has met practically every bank CEO).
- Token-price variance is large (10-20x), so the cost incentive is strong.
- Many vendors and gateways already ship routing.
- Open models are sufficient for ~60% of coding work, so most spend can be shifted cheaply.
Implicit premises:
- Features that deliver obvious cost savings and are strongly demanded tend to commoditize into baseline expectations.
- A capability many vendors can build converges to a table-stakes feature rather than a durable differentiator.
- Routing to the cheapest capable model preserves output quality — i.e., 'good enough' is reliably detectable per task (the load-bearing hidden assumption).
Inferences:
- If routing commoditizes and open models cover 60% of spend, frontier-model providers face volume/margin pressure on the low-complexity tier.
- Differentiation shifts from raw model quality to routing quality and evaluation.
- Standalone routing vendors risk being squeezed once IDEs and gateways all build routing in-house.
Enterprises can cut AI spend substantially without sacrificing code quality by adopting intelligent routing. strength: medium
Explicit premises:
- Vendors claim 20-30% cost savings.
- Open models are estimated sufficient for ~60% of coding work.
- Banks want a layer to control spend 'while still generating high-quality code'.
Implicit premises:
- Routers can reliably classify task difficulty and match it to model capability.
- The 'sufficient' threshold is measurable and trustworthy.
- Vendor-sourced savings figures are representative of real deployments.
Inferences:
- The real bottleneck becomes trustworthy evaluation of 'is this model good enough for this task'.
- Without good evals, routing risks silent quality regressions that are invisible until they cause harm.
Narratives
- A head of engineering at a larger company told Gergely they wished there was an 'intelligent' router that picks the right model for the right task.
→ Origin anecdote establishing genuine demand-side pull rather than vendor push - Matan Grinberg (Factory AI CEO) says he has met 'practically every bank CEO' since launching the offering, because they want a layer to control spend while still generating high-quality code.
→ Testimony illustrating the scale and profile of enterprise demand, and supplying the ~60% open-model-sufficiency estimate - Cursor's Auto mode uses a fixed-price model where the savings accrue to Cursor, not the customer.
→ Cautionary counter-example illustrating incentive misalignment in vendor-built routers
Questions addressed
- ✅ Do intelligent model routers that pick the right model for the right task actually exist yet?
Yes — the piece catalogs dedicated routers, gateways with routing, and IDE 'Auto' modes. - ✅ How much money can routing save, and for how much of the work can cheaper/open models substitute?
Partially — vendors claim 20-30% savings and ~60% of coding work can use open models, but these figures are vendor-sourced and not independently verified. - ✅ Will intelligent routing become a standard feature?
The author forecasts yes — routing will become table stakes. - ⭕ How do you verify that a router actually picked a 'good enough' model — i.e., how do you guard against silent quality regressions without a reliable evaluation harness?
- ⭕ What are the data-egress, IP, and compliance implications of routing enterprise (e.g. bank) code and prompts to cheaper, open, or third-party-hosted models?
Gaps
- incompleteEvery quantitative claim (20-30% savings, 60% of work sufficient for open models) is sourced from vendors or a vendor CEO with a direct commercial interest; there is no independent benchmark of either the cost savings or, crucially, the routing quality.
Read next: Independent routing benchmarks and methodology — e.g., the RouteLLM paper, Not Diamond's / LMSYS routing evaluations, and cost-vs-quality studies that measure whether the 'cheapest capable' choice actually holds up on hard tasks. - incompleteThe load-bearing assumption — that 'good enough' can be reliably detected per task so that downgrading to a cheaper model preserves quality — is never examined. The hard part of any routing system is the evaluation of task difficulty and output adequacy, which the piece treats as solved.
Read next: LLM-eval literature and task-difficulty classification; how routers estimate confidence/complexity; the adjacent argument (e.g. Elena Verna's 'AI Confidence Theater') that the real work is the ongoing eval/iteration, not the first prompt. - incompleteBanks are named as the marquee buyers, yet the governance/security dimension is entirely absent: routing sensitive enterprise code and prompts to cheaper/open/third-party-hosted models raises data-residency, IP-leakage, and compliance questions that go unmentioned — a genuine blind spot given the audience.
Read next: Enterprise AI governance and data-egress controls; self-hosting open-weight models behind the firewall; regulatory constraints on where model inference may run (especially relevant to Nordic/EU public-sector and critical-infrastructure contexts). - incompleteThe incentive-conflict problem is flagged only for Cursor but never generalized: any router operated by a party that profits from token usage or from steering to a preferred model is a principal-agent problem, and the reader is left to notice this alone.
Read next: Frame routing as a two-sided market / principal-agent design problem; evaluate router neutrality (bring-your-own-keys, router-only services like Kilo) versus vendor-locked routers. - incompleteThe '60% sufficient by token spend' figure conflates volume with value: 60% of token spend is not 60% of business value, and the hardest 40% may be exactly where frontier-model quality matters most, so the savings story may understate the risk concentrated in high-stakes tasks.
Read next: Distribution of task criticality and the marginal value of frontier models on the hardest tasks; portfolio thinking about where to spend the expensive tokens rather than average-cost reasoning.
Related documents
- AI Costs Are Surging and the Cheap Model Fix Might Not Last
- GitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic.
- Downstream #1: Where Do Our Tokens Go?
- Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS
- Guest post: AI Inference Is Breaking Unit Economics
- AlloyDB Ships Proxy Models That Replace LLM Calls with Local Inference Inside the Database
- Baseten CEO Tuhin Srivastava on the AI Inference Crunch, Custom Models
- Travel Through the Lens of AI with Booking.com CEO Glenn Fogel
- Howard Yu's Post — American AI models went from 70% of usage to 30% in 12 months
- Hugging Face's Clem Delangue: Stop Comparing Engines to Cars