The Meter Was Always Running

Bennie Haelen · rss · 2026-07-23 source ↗ Reader 🖥️

practical · argumentative — Argues for a specific engineering action (build a loop-aware, below-the-agent observability substrate) using a mix of an opening narrative hook, a theoretical reframing (the loop, not the request, is the unit of observation), and a concrete design specification (signal table, SQL query, pilot description).

Structure

Key terms

• control plane Platform Engineering
  The platform layer above individual agents where an organization centralizes observability and enforces policy, access, budget, routing, and execution constraints -- distinguished from the evidence/observability layer beneath it that the control plane reads from.
• observability substrate
  The fleet-queryable, loop-aware store of turn-level signals (tokens, tools, guardrail decisions, identity/authority) that the control plane reads from -- evidence, not a visualization layer.
• instrumentation points
  The runtime chokepoints an agent's execution must pass through and cannot bypass -- model gateways, tool proxies, API gateways, execution sandboxes, runtime harnesses, policy engines -- and therefore the only valid place to capture governance evidence.
• the loop / the turn
  The unit of agent work -- one reason/act/observe cycle -- proposed as the natural unit that traces should represent, replacing the request as the atomic thing being observed.
• metadata by default, content capture by exception
  A data-minimization design rule: capture structural/quantitative evidence (ids, counts, sizes, policy decisions) by default, and raw prompt or tool-result content only when specifically justified -- enough evidence to reconstruct why a run behaved as it did without an uncontrolled archive of sensitive payloads.
• loop bound / payload cap
  Deterministic circuit-breaker thresholds (max turns, max tool-result size) whose firing is itself a governance signal -- a run that stops because it hit a bound is categorically different from one that stops because it completed.

Propositions

• explicitdevelopedProvider billing and application logs can show that an agent run was expensive or that it completed, but neither can explain which design choice inside the platform caused the cost or how the agent's path led there -- visibility is not the same as attributability.conf: high
  extends ←AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act · p6 — The Meter piece's central distinction is that the invoice is 'honest but incomplete' -- visible but not attributable, implicitly assuming the billing signal a control plane reads is at least correct even if under-explained. This incident extends that claim by showing the billing/estimate telemetry layer (not the invoice itself, which stayed correct here) can fail at an even more basic level than incompleteness: the displayed cost signal can simply be wrong, platform-wide, for over 24 hours -- a more fundamental threat to any system that treats provider cost telemetry as ground truth to observe from.
  extends ←Beyond the 200 OK: Architecting Observability for AI · p2 — Meter's claim that billing/logs show a run was expensive or completed but not why is a cost-attribution version of the same blindness; this doc generalizes the diagnosis from cost-opacity to semantic-quality-opacity — APM/billing telemetry can't see content correctness any more than it can see cost causation.
  extends ←The tokenmaxxing party is over, and Revenium is mopping up · p3 — Extends the 'visibility is not the same as attributability' gap beyond an agent's own token/tool-call trace to costs incurred in third-party downstream systems the agent triggers but never appears on — a harder, cross-system version of the same attribution problem.
• explicitcorroboratedCost spikes, unsafe delegation, and runaway actions are three distinct failure types that all expose the same missing layer: a control plane cannot govern a loop it cannot independently observe.conf: high
  extends ←AWS Billing Bug Shows Customers Trillion-Dollar Estimates While Its Own Cost Alarms Fail to Act · p1 — The Meter piece argues a control plane can only govern a loop it can independently observe, and treats agent self-reporting/opaque telemetry as the core failure mode to fix. This incident supplies a case where observation itself worked (AWS's alarms did detect the anomaly) yet the failure still occurred, because detection was never wired to an enforcement action -- extending the framing from 'you need observability' to 'observability without a bound action is not governance either.'
  supports ←Beyond the 200 OK: Architecting Observability for AI · p4 — Both name runaway/uncontrolled agent behavior as a distinct failure type requiring a dedicated observability layer, not just request-level monitoring.
  supports ←Beyond the 200 OK: Architecting Observability for AI · p5 — Uncontrolled LLM costs is the same failure category (cost spikes) that Meter identifies as one of three symptoms of a missing control-plane layer.
• explicitestablishedAgent observability must be architected around the turn as the unit of observation, not the request, because a single user intent fans out into many model and tool calls whose relationships -- not any one call's timing -- carry the signal that matters.conf: high
  supports ←Beyond the 200 OK: Architecting Observability for AI · p7 — Both argue tracing must capture the full multi-step execution structure of an interaction rather than a flat/linear log, though Meter specifies the turn as the concrete unit while this doc stays at the level of 'execution tree.'
  supports ←Presentation: From Retrieval to Reasoning: Building Production-Ready Agentic AI Systems with Knowledge Graphs · p11 — Both treat the turn — not the request or the final answer — as the unit that visibility/observability must be built around: Shum's harness counts turns per task for cost/efficiency reporting, matching the claim that agent observability must be architected around the turn as the unit of observation.
  supports ←Session Traces and Cost Controls Help Diagnose AI Agent Failures · p2 — StackGen's Langfuse nested spans are a concrete implementation of the Meter piece's claim that observability must be architected around the turn (not the request) because the signal lives in the relationships between calls -- nesting child spans beneath parents is precisely how those relationships get captured.
• explicitThe agent's context is accumulating state carried forward turn by turn (prior messages, tool descriptions, retrieved files, intermediate results), and this growth -- not a fixed request payload -- is where much of a run's cost and risk originate.conf: high
  supportsThe Production Gap: 5 Patterns for Building Long-Running AI Agents · p20 — Both treat agent state as an accumulating stock that drives behavior, cost, and risk rather than a fixed request payload -- Turing Post frames it as an architecture requirement (checkpoint/resume, distinct memory stores), Haelen as an observability requirement (watch the state grow turn by turn).
• explicitA tool call is a first-class governance decision -- which tool, what parameters, what result size, whether policy constrained it -- not an implementation detail invisible to observability.conf: high
  extendsPrompt Injection to Data Exfil in 3 Hops · p6 — That piece argues you cannot answer a probabilistic attack with a probabilistic defense -- guardrails must be deterministic. Haelen's guardrail-decision-event signal (policy_id, policy_decision: allow/deny/rewrite/escalate) assumes and operationalizes exactly this deterministic-guardrail requirement without itself arguing for it.
• explicitdevelopedBecause every run can trace a different path through the same prompt, fixed call graphs and clean service maps assume a regularity agent behavior doesn't have, so request-level monitoring can show ten thousand successful calls and never notice the one loop that ran far longer than it should have.conf: high
  extends ←Session Traces and Cost Controls Help Diagnose AI Agent Failures · p5 — Rolling-average cost-baseline comparison is a concrete method for catching exactly the blind spot the Meter piece names: request-level monitoring can show ten thousand successful calls and never notice the one loop that ran far longer than it should have.
• explicitdevelopedA useful observability substrate needs a minimum concrete signal set -- run/turn structure, token and model accounting, tool-call events, guardrail decision events, identity and authority context, and outcome/bound metadata -- landed in a fleet-queryable store, not only a per-run trace viewer.conf: high
  extendsWhy Token Handling Must Evolve for AI Agents · E3 — Token-handling argues authorization for AI agents must be evaluated at runtime, per call, rather than at session start; Haelen's identity-and-authority-context signal (principal_id, delegated_scope, service_account, data_scope captured per turn) is the audit trail that would let you verify that runtime, per-call authorization actually happened.
  extendsBuilding AI Agents For The Enterprise · p14 — That piece asserts auditability is 'available via OpenTelemetry-compatible observability tools' in general terms; Haelen supplies the concrete schema and granularity (turn-level signal categories, example fields) that claim needs to be actionable rather than aspirational.
  extendsPresentation: The Multi-Agent Approach: Building Reliable and Controllable Software Development Automation · p7 — That piece asserts governance, observability, and controllability (including per-agent least-privilege credentials) must be a first-class architectural layer; Haelen's signal table is a concrete design for exactly that layer.
  extendsAI Model Context Protocol Adds Centralised Auth for Enterprise · p10 — That piece notes connection-level authorization (MCP's Enterprise-Managed Authorisation) is insufficient governance and organizations must still supply their own runtime, per-action controls; Haelen's tool-call-event and guardrail-decision-event signals are a description of what that missing runtime, per-action evidence layer actually captures.
  refines ←Session Traces and Cost Controls Help Diagnose AI Agent Failures · p8 — Adds an operational nuance the Meter piece's signal-set list doesn't specify: which telemetry type (trace/log vs. metric) each kind of signal should live in, and why (metric-label cardinality limits).
• explicitThe substrate should capture metadata by default and raw prompt/tool content only by exception, so the record reconstructs why a run behaved as it did without becoming an uncontrolled archive of everything the agent saw.conf: high
• implicitcorroboratedAn expensive run's cost only becomes legible once its turn-level structure is visible: a pattern of retrying the same tool call with slightly different parameters across several turns is a self-reinforcing cost spiral that keeps compounding until a deterministic control -- a loop bound or payload cap -- rather than clean completion, stops it.conf: medium
  supports ←Session Traces and Cost Controls Help Diagnose AI Agent Failures · p4 — Hard iteration caps and pre-execution blocking of identical consecutive tool calls are exactly the deterministic control (loop bound / payload cap) the Meter piece argues is what actually stops a self-reinforcing cost spiral, rather than clean completion.
• explicitThe same observability substrate answers three governance problems organizations usually treat as separate -- cost, delegation, and runaway actions -- because all three require the same kind of turn-level evidence of the same loop.conf: high
  supportsAI Costs Are Surging and the Cheap Model Fix Might Not Last · p10 — Structural parallel only, not the same substantive claim: that piece describes cost-optimization routing and compliance/sovereignty routing converging into a single infrastructure control plane; Haelen describes cost, delegation, and runaway-action governance converging onto a single observability substrate. Different concerns, same consolidation pattern -- previously separate governance problems turning out to share one underlying layer.
• explicitAn agent cannot be the system of record for its own authority, cost, or refusals: self-reported scope and blocked-action claims are exactly what fails an auditor and what a sufficiently clever prompt can talk its way around, so evidence must be captured at instrumentation points the agent cannot bypass or opt out of.conf: high
  refinesYour AI agent just blamed the network team. Now what? · p7 — Not a contradiction: Haelen explicitly allows the agent to 'emit useful breadcrumbs' and only denies it as the system of record for authority, cost, and refusals. Blame-network's p7/p8 treat the agent's own reasoning trail as an auditable observability layer for explaining a diagnostic finding to a human responder -- exactly the breadcrumb/explanation case Haelen permits. This extraction splits the term the two pieces both use ('observability'/'auditable record') into two senses: agent-produced narration as explanation (fine, per both) versus independently captured evidence as audit record (must be below the agent, per Haelen) -- a term-collision worth surfacing even though the claims aren't actually opposed.
• explicitEnforcement and observation are two faces of the same layer: policy can only be enforced on evidence that was independently observed, so putting both anywhere the agent can edit yields neither real enforcement nor real observability.conf: high
  supportsThe Production Gap: 5 Patterns for Building Long-Running AI Agents · p11 — Near-identical claim in different words: Turing Post says policy decisions belong in the governance layer, not in agent code or prompts; Haelen says the control plane is where policy decisions live and enforcement/observation are two faces of the same layer that must sit outside anything the agent can edit.
• explicitExisting application tracing, agent run viewers, and billing dashboards show that something happened, but not whether the underlying evidence is complete, independent of the agent, and queryable across the whole fleet -- so having a trace viewer is not the same as having agent governance.conf: high
• implicitWithout this substrate, organizations discover the same underlying governance gap repeatedly and in escalating form -- first as a surprising invoice, later as an audit finding, eventually as an incident -- because no signal closes the loop early enough to act before the failure compounds.conf: medium

Arguments

Narratives

• A team opens its first agent invoice, sorts runs by cost, and finds one that cost 40x the median -- and the provider meter, the application logs, and the trace viewer are each individually clean, none explaining why this run wandered while its neighbors finished cleanly.
  → Dramatizes the core distinction the whole piece rests on -- visibility vs. attributability -- and motivates the need for a different kind of evidence than any existing tool provides.
• Revisited later with a loop-aware trace: turn 3 retrieved 80,000 rows, turn 4 carried that result forward, turn 5 selected the expensive model, turns 6 through 11 retried the same tool call with slightly different parameters, and the run stopped because it hit a loop bound rather than completing cleanly.
  → Turns the abstract signal table into a concrete causal story, showing what the proposed substrate would have revealed that the old trace couldn't -- the payoff example for the whole argument, and support for the reinforcing/balancing-loop reading of proposition 9.
• A pilot for a large healthcare organization built this observability layer first, on OpenTelemetry, Cloud Trace, and a usage-log table in the warehouse, with the author noting the particular stack matters less than the shape.
  → Grounds the abstract substrate in a real, in-progress deployment and signals that the recipe is meant to generalize beyond the specific tools named.

Questions addressed

• ✅ Why does an honest, correct bill still feel unexplained?
  Because billing and logs are request/aggregate-level; attribution requires turn-level, causal evidence of what happened inside the loop that produced the cost.
• ✅ What is the minimum concrete signal set an organization needs to instrument agent loops?
  Run/turn structure, token and model accounting, tool-call events, guardrail decision events, identity and authority context, and outcome/bound metadata, stored in a fleet-queryable table.
• ✅ Can the agent be trusted to report its own cost, authority, and refusals?
  No -- self-reporting is exactly what fails an auditor and what a clever prompt can talk its way around; evidence must come from instrumentation points below the agent that it cannot bypass.
• ✅ Doesn't existing tracing/observability tooling already solve this?
  No -- existing tools show one run's path through a UI; the gap is completeness, agent-independence, and fleet-wide queryability, not visualization.
• ⭕ Given the metadata-by-default, content-capture-by-exception rule, how does the substrate ever explain the semantic 'why' behind a model's choice to retry, escalate, or pick an expensive path -- as opposed to merely the 'that' and 'how much'?

Gaps

• illogicalThe piece promises evidence 'to reconstruct why a run behaved the way it did' while simultaneously prescribing metadata-by-default, content-capture-by-exception as the data-minimization rule. But in an LLM loop the 'why' is largely semantic -- it lives in the prompt and tool-result content the design says not to store by default. Turn counts, byte sizes, retries, and policy decisions can show that a run wandered (which turn, how much, how many retries) but not what the model was reasoning toward when it chose to retry or picked an expensive model. The recipe answers 'that' and 'how much' more convincingly than the 'why' its own framing promises.
  Read next: extract-fowler-bockeler-static-sensors-iii draws exactly this computational-vs-inferential-sensor distinction for code modularity review and argues explicitly that deterministic (computational) sensors alone are insufficient where semantic interpretation is required -- the same lesson applied to a different loop.
• incompleteThe instrumentation points -- model gateways, tool proxies, execution sandboxes -- are assumed to be chokepoints the agent 'can't bypass.' That holds when every tool and model call must cross a mediated API. It's less obvious for an agent that writes and executes its own code inside a sandbox, which can perform internal retries, sub-calls, or state changes that never cross an instrumented boundary until a final result is returned. The piece doesn't address what 'below the agent' means once part of the loop is the agent's own generated code running inside the sandbox rather than a call the harness mediates.
  Read next: Sandboxed code-execution agent architectures and network-egress-level containment (the fallback chokepoint when application-layer instrumentation can be walked around from inside a sandbox) -- see extract-prompt-injection-to-data-exfil-in-3-hops for the analogous move of pushing the deterministic control down to a boundary the agent's own code can't route around.
• incompleteThe piece assumes that once evidence lands in a fleet-queryable store, 'a control plane uses what happened to change what happens next' -- treating technical visibility as sufficient for governance action to actually follow. It doesn't address organizations where the real blocker on acting on inconvenient evidence (a team's own cost pattern, a flagged delegation) is political rather than technical: whether a finding gets acted on depends on who owns the metric, who is implicated by it, and whether disclosure is met with stewardship.
  Read next: extract-tbm-426-trouble-with-mirrors argues that visualizing how a system works and 'embracing transparency' cannot by themselves improve it, because organizational truth is positional and negotiated among parties who each have something at stake in what gets seen -- automating the mirror (a policy engine's deterministic deny) narrows but doesn't eliminate the human, political layer that decides what the mirror is allowed to show and who has to answer for it.

Closes gaps elsewhere

• AWS MCP Server Reaches GA with Full API Coverage and IAM-Based Governance · g5 — That gap flags that cost-runaway from sandboxed loops or chatty API calls is a real failure mode IAM cannot stop, and that the piece it critiques glides past 'cost-as-safety.' Haelen's substrate directly supplies the missing detection and attribution layer: turn/token accounting that identifies which turn, tool, or model choice drove the cost, plus loop_bound_hit and payload-cap signals that make a runaway loop a queryable, alertable outcome rather than a postmortem story. It supplies detection and attribution, not a budget-enforcement mechanism itself -- the piece doesn't propose spending caps, only the evidence that would let a policy engine enforce one.

Related documents

• AI Model Context Protocol Adds Centralised Auth for Enterprise
• Building AI Agents For The Enterprise
• Why Token Handling Must Evolve for AI Agents
• The Production Gap: 5 Patterns for Building Long-Running AI Agents
• Governing AI in the Cloud: A Practical Guide for Architects
  Agent Governance (Internal) Guardrails Platform Engineering Policy-as-Code
• Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation
  Agent Observability AI Unit Economics Feedback Loops Guardrails
• Your AI agent just blamed the network team. Now what?
  Agent Governance (Internal) Agent Observability Guardrails
• Presentation: Autonomous Data Products for the Autonomous Era: Rethinking Data Architecture for GenAI
  Agent Governance (Internal) Platform Engineering Policy-as-Code
• Achieving Compliance as a Platform Engineering Team by Helping Developers
  Guardrails Platform Engineering Policy-as-Code
• Presentation: Platform Engineering in the Age of AI
  Agent Governance (Internal) Guardrails Platform Engineering