Beyond the 200 OK: Architecting Observability for AI
Adriano Mota · rss · 2026-09-11 source ↗ Reader 🖥️
Agent ObservabilityEvalsGuardrailsMCP (Model Context Protocol)Non-Deterministic SoftwareRetrieval & RAG
Because an AI system can fail silently behind a healthy 200 OK response, traditional infrastructure-focused APM is structurally blind to AI failure and must be supplemented with purpose-built observability that traces cost, quality, and decision-making across the LLM/RAG/MCP/agent stack and feeds a continuous evaluation loop.
practical · vendor-adjacent trade-blog overview with heavy internal cross-linking (13 links to other Nordic APIs posts) and an appended AI-generated summary — A taxonomy-and-checklist piece for API architects and platform engineers, not a research report or case study: it names no tools, no companies, no figures, and no worked example. Its value is in the layered vocabulary (computational/semantic/agentic observability) and the metric catalog it assembles, not in evidence that any of it works.
Structure
| Part | Role |
|---|---|
| Framing: the 200 OK inversion | problem statement |
| Defining AI observability (computational / semantic / agentic layers) | definition and taxonomy |
| The gaps when observability isn't applied (silent failures, retrieval blind spots, runaway agents, uncontrolled costs) | motivating case via failure modes |
| Core metrics by component (LLMs, RAG, MCP servers, AI agents) | prescriptive checklist |
| Implementing observability patterns (tracing, evaluation-driven development, production sampling, feedback loop, guardrails) | prescriptive process |
| Governing the non-deterministic enterprise | conclusion / thesis restatement |
Key terms
- determinism gap
the structural difference between deterministic software (same input, same output, traceable via infra metrics) and AI systems (probabilistic, can fail while returning a technically successful response) that traditional APM was never built to see - computational observability
the layer of AI observability covering unit economics and operational metrics: latency, token throughput, cost per request - semantic observability
the layer evaluating actual output quality and safety — hallucination, relevance, data leakage — i.e. whether the content is right, not just whether the request succeeded - agentic observability
tracing an autonomous agent's decision-making and reasoning path to understand why specific functions/tools were selected - glass box
the end-state AI observability is meant to produce: a probabilistic system rendered measurable and auditable, contrasted with the 'black box' of untraced probabilistic reasoning - evaluation-driven development
treating evals as unit tests: a curated input/expected-output dataset that isolates and tests individual pipeline components and gates deployment on quality thresholds
Propositions
- explicitAI systems can fail while returning a technically successful (200 OK) response — producing hallucinated, biased, or irrelevant output with no infrastructure-level signal that anything went wrong.conf: high
- explicitTraditional APM, built to monitor deterministic infrastructure health (CPU, latency, HTTP error rates), cannot detect semantic failures because it has no mechanism for judging content quality.conf: highextendsThe Meter Was Always Running · p1 — Meter's claim that billing/logs show a run was expensive or completed but not why is a cost-attribution version of the same blindness; this doc generalizes the diagnosis from cost-opacity to semantic-quality-opacity — APM/billing telemetry can't see content correctness any more than it can see cost causation.
- explicitAI observability must operate across three distinct layers — computational (cost/latency/throughput), semantic (quality/safety), and agentic (decision/reasoning tracing) — because a single infra-style metric surface cannot capture all three.conf: high
- explicitWithout observability, autonomous agents are a black box whose 'thought process' can get stuck in expensive infinite reasoning loops or repeatedly call the wrong functions with no detection mechanism.conf: highsupportsThe Meter Was Always Running · p2 — Both name runaway/uncontrolled agent behavior as a distinct failure type requiring a dedicated observability layer, not just request-level monitoring.
- explicitLLM API costs can spiral out of control because without granular, per-step/per-agent/per-function token visibility, inefficient workflows are undetectable and therefore unoptimizable.conf: highsupportsThe Meter Was Always Running · p2 — Uncontrolled LLM costs is the same failure category (cost spikes) that Meter identifies as one of three symptoms of a missing control-plane layer.
- explicitRAG failures frequently occur upstream of generation — wrong documents retrieved, missing context, outdated sources — so evaluating only the final LLM output misses the actual point of failure.conf: high
- explicitAn AI trace must capture the complete execution tree of an interaction (prompt assembly, retrieval, function execution, generation), not just a linear log, because failure can originate at any node in that tree.conf: highsupportsThe Meter Was Always Running · p3 — Both argue tracing must capture the full multi-step execution structure of an interaction rather than a flat/linear log, though Meter specifies the turn as the concrete unit while this doc stays at the level of 'execution tree.'
- explicitEvaluation metrics should be integrated into deployment pipelines as release gates, the same way unit tests block a bad build, so that increased hallucination rates block shipping.conf: high
- explicitcorroboratedA durable improvement loop requires converting every flagged production failure into a permanent regression-test case, so failures accumulate into growing test coverage rather than being fixed once and forgotten.conf: mediumsupports ←Meta's Recipe for Building Agents as "Organizational Second Brains" · p6 — Same mechanism — folding every fix into a permanent regression suite — applied here to expert-knowledge corrections rather than production incident failures.
- implicitObservability and safety are the same pipeline, not separate concerns: real-time guardrails against toxic content, data leakage, and prompt injection should be wired into the same telemetry path that measures quality and cost.conf: medium
- implicitRigorous AI observability is what legitimizes autonomous AI systems for mission-critical enterprise roles — without it, an AI system remains an 'experiment' rather than 'enterprise-grade architecture' regardless of its raw capability.conf: medium
Arguments
Engineering teams must adopt AI-specific observability (computational + semantic + agentic) rather than extending existing APM. strength: medium
Explicit premises:
- A 200 OK response and correct output are not the same thing for AI systems.
- APM was engineered to monitor deterministic infrastructure and has no concept of output quality.
- AI observability is defined precisely to close this gap via computational, semantic, and agentic layers.
Implicit premises:
- Content quality and decision correctness can be reliably measured by automated instrumentation in the first place — the argument assumes semantic and agentic observability are achievable, not just infra observability.
- The cost of NOT having this instrumentation (silent failures reaching users) exceeds the cost of building and running it.
Inferences:
- APM-insufficiency (premise 1-2) is treated as sufficient justification for a wholesale new discipline (conclusion), without weighing whether narrower/cheaper additions to existing APM (e.g. output-length or retry-count alerts) could close part of the gap.
Evaluation should be run continuously as a development-cycle discipline (evaluation-driven development, production sampling, regression-test accumulation), not as a one-time pre-launch check. strength: low
Explicit premises:
- A curated dataset of inputs/expected outputs can be built and used like unit tests.
- Production traffic can be sampled and scored automatically.
- Flagged failures can be extracted and added to the curated dataset.
Implicit premises:
- There exists a reliable, low-noise automated scorer for hallucination, relevance, faithfulness, and toxicity — the entire pipeline (gate releases, score production traffic, grow the regression set) presupposes an evaluator that is itself accurate and stable, but no such evaluator, nor its own error rate, is named.
- The judge/evaluator used to score outputs does not itself introduce the same class of non-deterministic failure the pipeline exists to catch.
Inferences:
- The unit-test analogy is carried over wholesale (curate, gate, regress) without acknowledging that unit-test assertions are deterministic boolean checks, while evals are threshold decisions on a noisy, distributional score — a different epistemic situation that the analogy quietly smooths over.
Questions addressed
- ✅ Why can't existing APM tools be extended to monitor AI systems?
APM measures infrastructure health; it has no mechanism for evaluating whether content is correct, relevant, or safe, which is where AI systems actually fail. - ✅ What should be measured, concretely, for each component of an AI stack (LLM, RAG, MCP server, agent)?
The article supplies a per-component metric list (latency/tokens/hallucination-rate for LLMs; context precision/recall/faithfulness/answer-relevance for RAG; invocation counts/success-rates/context-window usage for MCP servers; execution-flow/function-accuracy/cost-per-task for agents). - ⭕ How do you actually measure hallucination rate, faithfulness, or answer relevance in production, and how reliable is that measurement?
- ⭕ Who or what evaluates the evaluator — i.e., what is the accuracy/failure rate of the automated scoring pipeline itself?
- ⭕ What does an AI observability pipeline cost to build and run, and does that cost undermine the cost-discipline goal it's meant to serve?
Gaps
- incompleteThe entire evaluation-driven-development prescription (gate releases on hallucination rate, score production traffic automatically, treat flagged failures as regression tests) presupposes a reliable automated judge for hallucination, faithfulness, relevance, and toxicity. The article never names what performs this scoring, what its own error/false-positive rate is, or how you'd know if the evaluator itself degrades — the load-bearing implicit premise across the whole implementation section is that semantic quality is reliably scorable by a non-human judge that is itself trustworthy and stable.
Read next: LLM-as-judge reliability and calibration literature; work on evaluator drift and adversarial gaming of automated graders; the corpus's own evals concept for prior material on grading criteria and rubric reliability.closed byPresentation: From AI Agent Demo to Production: Automated Testing and Evaluation — That document's evaluation-driven-development prescription presupposes a reliable automated judge for hallucination/faithfulness/relevance without saying what makes such a judge reliable. This talk gives a concrete recipe: ground the judge in a specific simulated scenario with a known, verifiable goal so task completion can be checked by rule-based verification against real backend state, and calibrate softer per-turn LLM-as-judge metrics (helpfulness, brand voice) with curated hard-negative and hard-positive examples. - incompleteThe article recommends full-stack tracing, continuous production sampling, and layered dashboards as the cure for uncontrolled AI costs, but never addresses the token/infrastructure cost of the observability pipeline itself (tracing overhead, LLM-as-judge scoring calls, dashboard compute). It treats observability as a pure cost-reduction move without costing the new cost center it introduces.
Read next: Cost-accounting treatments of observability/tracing overhead in production ML systems; the corpus's ai-unit-economics material on unaccounted infrastructure costs. - illogicalThe piece imports the unit-test analogy for evaluation wholesale ('evaluation metrics should be treated like software unit tests') without acknowledging that unit tests are deterministic boolean assertions on deterministic code, while evals are threshold decisions on a noisy, distributional score from a non-deterministic system. Blocking a release on 'increased hallucination rate' is a statistically different decision — subject to sample-size and variance questions — than a unit test failing, and the article gives it the same epistemic confidence.
Read next: Statistical treatments of eval-threshold decisions under sampling noise; comparisons of CI gating semantics for deterministic vs. probabilistic test suites. - incompleteNo worked example, named tool, benchmark figure, or case study appears anywhere in the piece — every claim (about silent failures, retrieval blind spots, cost spirals, or the value of tracing) is asserted at the level of general architecture, with no evidence that any organization has implemented this stack or what it cost/saved them.
Read next: Practitioner case studies of AI observability rollouts with before/after incident or cost data (the corpus's extract-the-meter-was-always-running and extract-downstream-1-where-do-our-tokens-go are closer to this evidentiary bar).
Related documents
- Presentation: Engineering AI for Creativity and Curiosity on Mobile
- Presentation: From Thousands to One: Building LLM-Powered Selection Systems
- Session Traces and Cost Controls Help Diagnose AI Agent Failures
- Article: Agentic Fitness Functions: Extending Evolutionary Architecture Beyond Deterministic Rules
- Your AI agent just blamed the network team. Now what?
- Presentation: From AI Agent Demo to Production: Automated Testing and Evaluation
- The Anatomy of an Agent Harness
- The Problem Is Prompt Debt
- Cloudflare WriteGuard Brings Fine-Grained Security Controls for MCP Servers
- DoorDash Used Copilot to Convert Its XCTest-Based iOS Test Suite to Swift Testing