Expedia Uses AI Driven Service Telemetry Analyzer to Accelerate Incident Investigation

Leela Kumili · rss · 2026-07-23 source ↗ Reader 🖥️

Agentic AI Human-in-the-Loop LLM as Navigator (Not Oracle)SRE & Incident Response AI Workflow Patterns

Expedia built STAR, an internal AI-assisted observability tool, on the premise that a deterministic, human-validated, prompt-chained workflow over standardized infrastructure telemetry -- deliberately forgoing function calling, RAG, memory and autonomous tool use -- is the right way to accelerate production-incident root-cause investigation today, with engineers remaining the final validators of every finding.

practical · expository — InfoQ news reporting on an Expedia engineering blog post. It narrates a design decision -- a deterministic, prompt-chained workflow instead of an autonomous agent -- without the source text actually arguing for that choice over alternatives; the 'argument' extracted below is a reconstruction of the reasoning implied by the choice, not something the text defends explicitly.

Structure

Part	Role
Motivation: the stated objective to minimize Time to Know (TTK) and Time to Recover (TTR)	context
Core architecture decision: deterministic, prompt-chained workflow instead of an autonomous agent (no function calling, RAG, memory, or autonomous tool use)	the central claim the rest of the article elaborates
Scope of telemetry: standardized Kubernetes and JVM infrastructure metrics, chosen for consistency across languages/frameworks	evidence/context for why infra-level signals were chosen as the diagnostic substrate
Evolution of the implementation: FastAPI background tasks replaced by an async, Redis-backed architecture	implementation detail explaining how the deterministic workflow scales under I/O-bound load and external rate limits
Production use and mandatory human review of generated findings before action	evidence that the tool is deployed and that the human-in-the-loop guardrail is real, not aspirational
Roadmap (MCP-based tool integration, dependency data, conversational interfaces, chaos-engineering use) and evaluation via Langfuse plus SME review	conclusion/future direction; also introduces an unacknowledged tension with the deterministic-workflow rationale

Key terms

Time to Know (TTK)
The elapsed time before responders understand what is wrong with a service and why -- presented as one of the two objective functions STAR was built to minimize.
Time to Recover (TTR)
Time from incident onset to service restoration, used identically to the standard SRE sense but abbreviated without 'Mean.'
deterministic workflow (as opposed to an autonomous AI agent)
A fixed, predefined sequence of steps -- collect telemetry, analyze via domain-specific prompts, consolidate intermediate findings, summarize into a report -- that the LLM does not deviate from by planning its own actions, calling functions, retrieving from a vector store, or invoking tools autonomously.
prompt chaining
Sequencing multiple specialized LLM calls so each stage's output feeds the next stage, producing a consolidated diagnosis from several narrower analyses.

Propositions

explicitExpedia states that STAR's objective was to minimize Time to Know (TTK) and Time to Recover (TTR) for production incidents.conf: high
explicitSTAR deliberately does not use function calling, retrieval-augmented generation, memory, or autonomous tool use; it instead relies on a predefined, deterministic workflow to produce consistent analyses.conf: high
supportsGenerative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production · p10 — Reganti's principle -- 'If a task can be solved with a workflow (with human audit points), do not make it autonomous' -- is exactly the design choice STAR embodies in production: a fixed telemetry-to-prompt-chain-to-report workflow with mandatory human review before any action, deliberately declining function calling, RAG, memory and autonomous tool use. STAR is a concrete, named, production instance of that abstract principle.
extendsBuilding an AI Incident Copilot: How I Automated the First 15 Minutes of Every Production Incident · p6 — The sibling tool ships only deterministic pattern-matching today and names LLM summarization as a future roadmap item ('the AI incident copilot framing is forward-looking'). STAR is a further step down that same trajectory already in production: a deterministic, non-agentic workflow that already incorporates real LLM analysis (domain-specific prompts, multi-stage summarization) rather than pattern-matching alone.
implicitA deterministic, predefined-workflow architecture -- not an autonomous agentic one -- is the appropriate design choice for a production incident-diagnosis tool at this stage.conf: medium
explicitStandardized infrastructure-level Kubernetes and JVM telemetry gives a more consistent diagnostic view across services than would service-specific business metrics, because it is uniform across programming languages and frameworks.conf: high
explicitEngineers review STAR's generated findings before acting on its recommendations, making it an assistive tool rather than an autonomous operational system.conf: high
supportsYour AI agent just blamed the network team. Now what? · p11 — The sibling piece prescribes a three-phase autonomy-expansion model for diagnostic AI (shadow mode -> read-only human-in-the-loop -> limited autonomous remediation) and argues organizations should not rush past phase 2. STAR's current deployment -- engineers review every generated finding before acting, no autonomous remediation -- is a real-world instance sitting exactly at that read-only, human-in-the-loop phase, corroborating the framework with a named production example.
explicitBecause most STAR processing is I/O-bound (telemetry retrieval and LLM requests), an asynchronous, Redis-backed execution model lets it process multiple analyses concurrently while respecting external rate limits from Datadog and the internal AI gateway.conf: high

Arguments

A deterministic, prompt-chained workflow -- not an autonomous agent -- is the right architecture for this production incident-diagnosis tool. strength: low

Explicit premises:

The current implementation does not use function calling, RAG, memory, or autonomous tool use.
It instead relies on predefined workflows to generate consistent analyses.

Implicit premises:

Consistency and predictability of output matter more, for production incident diagnosis, than the flexibility autonomous tool-use would offer.
Autonomous agentic techniques would produce less consistent or less easily auditable output than a fixed prompt-chain.
Engineers can trust and act on a report more readily when they can predict and audit the fixed steps that produced it.

Inferences:

This design trades exploratory flexibility for reproducibility and auditability -- a trade-off the article never states or defends explicitly.
Because the roadmap proposes adding 'MCP based tool integrations' and 'conversational interfaces,' the deterministic architecture is implicitly a stage in an evolving system rather than a permanent ceiling -- raising the unaddressed question of whether those additions reintroduce the very autonomy the initial design avoided.

Narratives

A direct quote from the Expedia engineering team stating their objective: "Our objective with this service was to minimize the time to know (TTK) and time to recover (TTR)."
→ Grounds the abstract architecture description in a stated organizational goal, implicitly framing every subsequent design choice (deterministic workflow, infra-metric scope, async architecture) as instrumental to that TTK/TTR objective -- even though no data in the article ties any of those choices to a measured outcome.

Questions addressed

✅ How can engineers reduce the time needed to identify the root cause of a production incident?
By building STAR: an internal tool combining infrastructure telemetry with LLM analysis via a deterministic, prompt-chained workflow, producing a structured root-cause report for engineers to validate.
✅ Should AI-driven observability/incident tooling be built as an autonomous agent or as a constrained, deterministic workflow?
Expedia chose the deterministic, prompt-chaining route explicitly, over autonomous agents with function calling, RAG, memory, or tool use.
⭕ Does adding the planned future capabilities (MCP-based tool integration, conversational interfaces) undermine the deterministic-workflow rationale the current design relies on?

Gaps

incompleteNo quantitative evidence is given that STAR actually reduced TTK or TTR -- the stated objective is never followed by a measured outcome (a duration, a before/after comparison, or even a qualitative estimate). The same absence-of-outcome-metrics pattern recurs in sibling AI-incident-tooling vendor posts already in the corpus (extract-ai-incident-copilot-first-15-minutes#g1, extract-ai-agent-blame-network-team#g4) -- this is the corpus's most consistent gap in AI-for-incident-response tooling.
Read next: Look for a follow-up post or case study reporting concrete before/after TTK/TTR numbers for STAR, or for independent (non-vendor) benchmarking of LLM-assisted incident-diagnosis tools.
illogicalThe article presents avoiding function calling, RAG, memory, and autonomous tool use as a deliberate design principle for consistency and reliability, then lists 'Model Context Protocol (MCP) based tool integrations' and 'conversational interfaces' as near-term planned enhancements -- without acknowledging that MCP-based tool use is itself the kind of dynamic, agent-directed tool access the current architecture was built to avoid. 'Deterministic today' and 'more autonomous tomorrow' are treated as compatible without revisiting the rationale that justified staying deterministic in the first place.
Read next: Watch for Expedia's own follow-up post once MCP integration ships, to see whether they re-justify the autonomy trade-off or quietly drop the 'no autonomous tool use' framing.
uninformedNo detail is given on how the 'domain-specific prompts' are engineered, what mitigations exist against LLM hallucination or fabricated root causes in the multi-stage summarization, or what the SME-review/Langfuse evaluation loop actually measures (accuracy? usefulness? both?). The article names the ingredients (Langfuse for prompt management/tracing/evaluation, SME review, user feedback) without describing evaluation criteria or results.
Read next: Read Expedia's original Medium engineering post (linked in the article) for evaluation methodology details, or look for a follow-up on false-positive/false-negative rates in STAR's generated findings.