Microsoft Three-Layer LLM Routing Architecture for AI Agents on AKS

Claudio Masolo · rss · 2026-07-29 Reader 🖥️

AI Unit Economics Capability Provisioning (right-sizing to the task)Compute & Energy Infrastructure Inference Optimization Model Routing

Because agentic workloads generate large volumes of small, heterogeneous-difficulty LLM calls that a single router or load balancer cannot handle well, production routing on Kubernetes should be decomposed into three independent, composable layers — semantic model selection (RouteLLM), request governance (agentgateway), and live GPU-state-aware placement (the Gateway API Inference Extension's Endpoint Picker) — each solving exactly one of the three distinct signals (which model, how the call is managed, which replica) rather than one component trying to do all three.

practical · expository — A reporting/summary piece (InfoQ news format) describing Microsoft's reference architecture. It is practical because the whole point is prescriptive — how teams should structure production routing infrastructure — but the text itself is expository (describing an existing design) rather than argumentative in its own voice; the argumentative content (why decompose into three layers) is inherited from the source blog post being summarized.

Structure

Part	Role
Problem framing: agent loops fire hundreds of calls per task, many trivial, and naive round-robin balancing is blind to both semantic difficulty and live GPU state	context / motivation
The three-layer architecture: RouteLLM (semantic routing) + agentgateway (AI proxy / policy) + Gateway API Inference Extension's Endpoint Picker (GPU-aware placement), unified behind one OpenAI-compatible endpoint	core exposition of the design
Wiring detail: KAITO provisions GPU node pools and runs vLLM; strong path goes to Azure OpenAI, weak path to KAITO pods via a service backend with inferenceRouting/passthrough; Prometheus/Grafana observe both routing and GPU metrics	evidence (implementation detail)
RouteLLM's benchmark numbers (95% of GPT-4's MT-Bench quality at ~26% escalation, up to 85% cost savings) and Microsoft's caveat that this escalation threshold is specific to the trained model pair and must be recalibrated for other pairs; plus the note that prompt caching complicates the true cost of a 'strong' call	qualifying nuance
Author's explicit warning that the open-source pieces (Inference Extension, agentgateway) are young and pre-v1, with CRD fields still moving	caveat
Managed-vs-self-hosted split: Azure Foundry model router is a managed version of RouteLLM's semantic layer, but there is no managed equivalent for the Endpoint Picker's GPU-aware placement — it must run in-cluster regardless of gateway choice	conclusion (about the shape of the market)
Adoption-in-slices guidance: pure governance needs only agentgateway; self-hosting one model class needs KAITO + Inference Extension without semantic routing; RouteLLM is worth adding only once there's a real strong/weak price gap and enough easy traffic	conclusion / recommendation

Key terms

the three-choice decomposition (which model / how the call is managed / which GPU replica)
The organizing frame of the whole architecture: routing for agents is not one problem but three orthogonal ones — semantic model selection, request policy/governance, and physical placement onto a GPU replica — each requiring a different signal and a different component to solve well
escalation threshold
The calibration knob in RouteLLM that determines what share of calls get promoted from the weak/cheap model to the strong/expensive one; explicitly not a portable constant — it must be re-tuned per model pair and per team's real traffic mix
Endpoint Picker
The component of the Kubernetes Gateway API Inference Extension that chooses which specific GPU replica of an already-selected model should serve a request, based on live signals (vLLM's KV-cache occupancy and queue depth) rather than static load-balancing rules

Propositions

explicitcorroboratedAgentic workloads fire far more LLM calls per task than chat, in a plan-act-observe loop, and a large share of those calls (tool-argument fills, yes/no gates, summaries) do not need a frontier model.conf: high
supports ←Enterprise Analytics Beyond Dashboards: Intelligent Data Orchestration with LLMs · p13 — Khapekar's practice of using a small model for intent classification/decomposition and reserving the large model for synthesis is a concrete instance of the broader claim that most LLM calls in an agentic loop don't need frontier-model quality.
explicitRouting for agentic workloads should be decomposed into three independent layers — semantic model selection, request policy/management, and GPU-replica placement — rather than solved by one router or load balancer.conf: high
extendsGitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · p10 — claude-code-router optimizes cost/capability along a single axis (route each task to the cheapest capable model); this architecture extends that to two more independent axes — request governance/policy and live GPU-state-aware placement — that a pure model-router never addresses.
explicitA naive round-robin load balancer is blind to live GPU state and can queue a small completion behind a huge prefill on a busy pod while an idle pod sits unused nearby.conf: high
explicitOn the model pair RouteLLM was benchmarked on, its matrix-factorization router reached about 95% of GPT-4's MT-Bench quality while sending only ~26% of calls to GPT-4, yielding up to 85% cost savings versus routing every call to the strong model.conf: high
refinesThe Pulse: a new trend, smart model routing · p2 — Where the Pragmatic Engineer piece reports vendor-claimed savings of 20-30% in general terms, this document supplies a concrete, sourced number (RouteLLM's published 85% savings / 95% quality at 26% escalation) — but immediately refines it by showing the figure is benchmark-pair-specific and non-transferable, sharpening the earlier piece's implicit warning about vendor-sourced figures.
explicitRouteLLM's escalation-threshold benchmark is specific to the model pair it was trained on and does not transfer automatically to a different pair (e.g. phi-4-mini/GPT-5.1); teams must calibrate the threshold against their own traffic instead of trusting the published figure.conf: high
explicitPrompt caching complicates true cost comparisons between a 'strong' and 'weak' call, because cached input tokens are discounted and switching models invalidates ('cools') both caches.conf: high
explicitThere is no managed equivalent for the GPU-aware placement layer (the Endpoint Picker) — unlike the semantic-routing layer (managed as Azure Foundry model router), it must run in-cluster no matter which gateway is used.conf: high
explicitThe three layers are independently adoptable: pure governance needs only agentgateway; self-hosting a single model class needs KAITO plus the Inference Extension without semantic routing; RouteLLM is only worth adding once there is a real price gap between a strong and weak model and a meaningful volume of easy traffic.conf: high
implicitTreating 'smart routing' as a single solved problem conflates three independent engineering concerns — semantic model choice, request governance, and physical GPU placement — that most vendor and product narratives compress into one undifferentiated 'router'.conf: high
implicitcorroboratedA vendor- or project-published routing benchmark (like RouteLLM's headline 85%-savings, 95%-quality figures) functions more as a marketing anchor for the general approach than as an operational guarantee, since the actual number is contingent on a specific model pair and must be re-earned per deployment.conf: medium
supports ←Introducing System One Models & Jev · p5 — Reinforces the corpus finding that a vendor- or project-published performance benchmark functions as a marketing anchor for the general approach rather than an operational guarantee — here the 'ground truth' being agreement with two chosen reference LLMs rather than verified correctness.
implicitcorroboratedBecause the open-source pieces are pre-v1 and their CRDs/field names are still shifting, adopting this architecture in production imposes an early-adopter tax (tracking upstream renames/restructuring) that the 'adopt in slices' recommendation does not itself address.conf: medium
supports ←Google Open-Sources AX a Kubernetes Style Orchestrator for Autonomous AI Agents · p5 — AX's primitives are also versioned v1alpha1 — the same pre-v1, still-shifting-CRD early-adopter tax this Azure/AKS agent-routing architecture piece flags for teams adopting Kubernetes-native agent infrastructure early.

Arguments

Production agent routing on Kubernetes should be built as three independently composable layers rather than as one monolithic router or load balancer. strength: medium

Explicit premises:

Agentic loops generate very high call volume with wide variance in per-call complexity.
A single load balancer or router cannot simultaneously reason about semantic difficulty, request governance/policy, and live GPU state.
Purpose-built open-source components already exist for each concern: RouteLLM (semantic), agentgateway (policy/proxy), Gateway API Inference Extension (GPU-aware placement).

Implicit premises:

These three concerns are separable enough in practice that solving them independently does not lose performance to cross-layer interaction effects.
The composition of three different OSS/managed pieces behind one OpenAI-compatible endpoint will remain operationally coherent despite each piece evolving on its own release cadence.

Inferences:

Teams can adopt a subset of the stack and still benefit — the architecture is a menu of independently useful layers, not an all-or-nothing prescription.
Because GPU-aware placement has no managed equivalent, it becomes the one layer every self-hosting team must own regardless of which vendors they use elsewhere — a durable operational responsibility, not a commodity.

Narratives

A round-robin load balancer queues a 200-token completion behind a 100k-token prefill on a busy GPU pod while another pod sits idle nearby.
→ Illustrates concretely why naive load balancing fails agentic traffic and motivates the need for a GPU-state-aware placement layer (the Endpoint Picker)
The author's own block-quoted warning that several pieces are young, field names drift, and the whole setup was validated end-to-end only against specific pinned versions (Inference Extension v1.0.0, agentgateway v1.3.1) in mid-2026.
→ Functions as a narrative caveat about lived operational experience with pre-v1 infrastructure — signals that the decomposition is stable even though the implementation details are not

Questions addressed

✅ How should LLM calls be routed for agentic (as opposed to chat) workloads running on Kubernetes?
By decomposing routing into three independent layers — semantic model selection, request governance, and GPU-aware placement — rather than solving it with one component.
⭕ Does routing to a cheaper model actually preserve output quality?
Only demonstrated for the specific model pair RouteLLM was benchmarked on (95% of GPT-4's MT-Bench quality at ~26% escalation); the author explicitly states this does not generalize and must be recalibrated per deployment.
✅ Is there a managed (non-self-hosted) option for every layer of the architecture?
No — the semantic-routing layer has a managed equivalent (Azure Foundry model router), but the GPU-aware placement layer (Endpoint Picker) must always run in-cluster.
⭕ What is the operational cost of building production infrastructure on pre-v1, actively-changing Kubernetes CRDs (Inference Extension, agentgateway)?

Gaps

incompleteRouteLLM's headline numbers (95% MT-Bench quality at 26% escalation, up to 85% savings) come from one benchmark on one specific model pair. The piece states plainly that the figure doesn't transfer to other pairs, but gives no methodology or tooling for how a team should actually validate or calibrate the threshold against its own traffic — the 'how do you calibrate this' step is asserted as necessary but never shown.
Read next: Practical guides or tooling for online calibration of routing thresholds against live traffic and quality signals (e.g. shadow-routing, canary evaluation, human-preference feedback loops) — this is the same unresolved 'how do you actually verify good-enough' gap flagged in the Pragmatic Engineer smart-routing piece, now recurring in a concrete production architecture.
incompleteNo cost or performance numbers are given for the assembled three-layer system itself — Endpoint Picker's own latency/overhead, agentgateway's proxy tax, or the inference cost of running RouteLLM's classifier. Every efficiency figure quoted (95%/26%/85%) belongs to a sub-component in isolation, not the end-to-end stack a team would actually operate.
Read next: Independent, end-to-end benchmarks of assembled LLM-routing stacks (not individual components) — comparable to the missing quantitative data flagged in the Netflix Triton+vLLM serving-platform piece.
incompleteThe article flags that the Inference Extension and agentgateway are 'young' and that CRD fields are still moving pre-v1, but never discusses the operational cost of tracking that churn in production — upgrade cadence, breaking-change history, or how a team without Microsoft's own platform expertise should pin and test across releases.
Read next: Version-pinning and upgrade-management practices for pre-1.0 Kubernetes CRDs / operators in production; postmortems or case studies of teams that adopted similarly young infra and hit breaking API churn.
incompleteThe architecture routes some calls to Azure OpenAI (managed, off-cluster) and others to self-hosted KAITO pods (in-cluster, open models) purely on a cost/quality basis — the piece never asks whether that split should also be constrained by data sensitivity or compliance, i.e. whether some agent traffic should be forced onto the self-hosted path regardless of RouteLLM's cost-driven verdict.
Read next: Enterprise AI governance and data-residency-aware routing — pairs with the same governance blind spot already flagged in the Pragmatic Engineer smart-routing piece and in claude-code-router's README.

Closes gaps elsewhere

GitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · g2 — claude-code-router's gap was the absence of any quality-aware routing mechanism — its routing is by static scenario/token-count, never by measured task difficulty or output adequacy. RouteLLM, as described here, is a concrete counter-example: a matrix-factorization router trained on human-preference data that predicts whether a cheaper model matches a stronger model's answer quality before routing. This partially closes the gap by showing quality-aware routing exists and has a published benchmark — but only partially, since the benchmark is general MT-Bench chat quality, not a coding session, and Microsoft explicitly warns it does not transfer to new model pairs without recalibration.