Agent Observability

Capturing and visualizing telemetry and traces from agent coding sessions (e.g. via OpenTelemetry) for usage insight, subject to user-level data-privacy constraints distinct from those of traditional application observability.

Also known as: agent traces, view of agent traces, agent session telemetry, coding session insights, session telemetry

Part of: Agentic Engineering

Documents (11)

• Cloudflare Introduces the Agent Development Lifecycle to Replace Traditional SDLC Olimpiu Pop · 2026-09-21 source ↗ Reader 🖥️
  Because AI models have made code generation fast, the bottleneck in software delivery has shifted downstream to testing, deployment, and maintenance, so Cloudflare argues the entire SDLC must be rebuilt as an event-driven 'Agent Development Lifecycle' — with Workflows as the orchestration primitive, OpenTelemetry-style observability into agent reasoning, and machine-speed security enforcement (short-lived scoped credentials plus a dynamically-narrowing 'Trust Ratchet') replacing human-paced review gates.
• Presentation: From Retrieval to Reasoning: Building Production-Ready Agentic AI Systems with Knowledge Graphs Cassie Shum · 2026-09-12 source ↗ Reader 🖥️
  A knowledge graph earns its place in agentic software engineering less as a retrieval-accuracy mechanism (an edge that has largely eroded as models got better at parsing raw context) than as an organizational substrate for four practices: assembling task-specific context bundles, tracing decision provenance from idea to production, treating code rather than spec as the ground truth of system behavior, and making agent activity visible enough to audit and manage.
• Beyond the 200 OK: Architecting Observability for AI Adriano Mota · 2026-09-11 source ↗ Reader 🖥️
  Because an AI system can fail silently behind a healthy 200 OK response, traditional infrastructure-focused APM is structurally blind to AI failure and must be supplemented with purpose-built observability that traces cost, quality, and decision-making across the LLM/RAG/MCP/agent stack and feeds a continuous evaluation loop.
• Session Traces and Cost Controls Help Diagnose AI Agent Failures Mark Silvester · 2026-09-11 source ↗ Reader 🖥️
  Diagnosing and controlling AI agent failures in production requires two complementary instrumentation layers beyond standard application monitoring -- nested session traces that reconstruct the full multi-agent delegation chain for after-the-fact debugging, and pre-execution cost/iteration limits paired with rolling-average anomaly detection for before-the-fact and slow-onset failure prevention -- because uptime/response monitoring can only show that a service answered, not why an autonomous workflow looped, misused a tool, or silently skipped work.
• Presentation: From Thousands to One: Building LLM-Powered Selection Systems Jendrik Jördening · 2026-08-17 source ↗ Reader 🖥️
  When you need an LLM to select one item from among many database-backed options (an ID, a station, a record), you must strip the model down to only the irreducibly textual/semantic step it is actually good at — schema-restricted, range-bounded, and independently validated by a second model call — while pushing every deterministic operation (filtering, comparison, control flow) into ordinary code, recovering a classic model-view-controller separation that a single freeform LLM call destroys.
• Downstream #1: Where Do Our Tokens Go? Johannes Gehrke · 2026-08-01 source ↗ Reader 🖥️
  The cost of agentic coding is set less by which model you use than by how the harness manages context, exploration, and stopping — so engineering leaders should invest in harness discipline, per-workflow model evaluation, and cost visibility rather than wait for cheaper tokens to fix agent economics.
• The Self-Building Agent: A LangChain4j Experiment Kevin Dubois, Mario Fusco · 2026-07-24 source ↗ Reader 🖥️
  In a controlled experiment where a code assistant designed and rebuilt its own multi-agent coding system, replacing an autonomous LLM-driven supervisor architecture with a deterministic, pre-declared workflow architecture cut end-to-end task time roughly threefold by eliminating LLM-induced coordination overhead, establishing a concrete speed-versus-autonomy trade-off between the two patterns.
• The Meter Was Always Running Bennie Haelen · 2026-07-23 source ↗ Reader 🖥️
  Because a control plane can only govern a loop it can independently observe, organizations running tool-using agents must build one below-the-agent, turn-level, fleet-queryable observability substrate -- not per-run trace viewers, billing dashboards, or agent self-reporting -- and that single substrate, not three separate systems, is what answers cost, delegation, and runaway-action governance at once.
• Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation Leela Kumili · 2026-07-10 Reader 🖥️
  Agentic testing — AI agents that execute end-to-end tests from a high-level intent rather than fixed scripts, adapting around superficial UI or service changes instead of failing — reduces brittle-test maintenance and belongs as a complementary top layer of the test pyramid, reserved for debugging and exploratory use because of cost, while deterministic tests remain the primary mechanism for validating critical logic.
• Looking back at the Future of Software Engineering unconference Bartosz Ocytko · 2026-07-07 source ↗ Reader 🖥️
  As agentic engineering matures into human-built-and-operated 'software factories,' the decisive questions of software engineering shift from raw automation to governance — where human judgment and sign-off sit in the automated loop, how platforms (rather than universal upskilling) safely absorb citizen developers, what would keep developer tooling relevant, and how organizations manage the widening gap between teams that codify their processes early and those that don't.
• Your AI agent just blamed the network team. Now what? Nik Kale · 2026-06-15 source ↗ Reader 🖥️
  Deploying autonomous, cross-domain AI diagnostic agents in production incident response introduces a new organizational-political risk -- a confidently-asserted, team-implicating finding that may be contested -- and that risk is best managed not by better AI reasoning but by a deliberately staged trust-building process (investigation-scoped credentials, a read-only start, an auditable reasoning trail, and treating rollout as organizational change with pre-secured cross-team buy-in) that earns autonomy incrementally rather than being engineered away as a purely technical problem.

Propositions touching this concept (1)

By publication month, oldest first (extraction month when no publication date is known).

2026-07

• explicitestablishedAgent observability must be architected around the turn as the unit of observation, not the request, because a single user intent fans out into many model and tool calls whose relationships -- not any one call's timing -- carry the signal that matters.
  The Meter Was Always Running · p3

Tensions

Open gaps (13)

• incompleteNo single concrete case study of an actual deployed cross-domain diagnostic agent's finding implicating a team, and how the organizational/political fallout was actually resolved (accepted, disputed, or overridden). The article's formative anecdote is entirely pre-AI and human-only; the 'two years building and deploying' claim is asserted authority without a demonstrated instance of the exact scenario the title promises to address.
  Your AI agent just blamed the network team. Now what? — read next: Look for practitioner post-mortems or case studies (SRE/platform-engineering conference talks, vendor case studies) that document a real agent-attributed root cause, the implicated team's reaction, and how it was resolved -- to test whether 'evidence beats politics' holds under an actual contested finding rather than only in the abstract.
• incompleteNo treatment of audit, observability, governance, or failure containment for long-running autonomous coding. The piece's implicit deployment is 'a single engineer running experiments at Anthropic Labs.' A Norwegian public-sector or regulated context would need: persistent audit trails of each agent's actions and tool calls, rollback semantics for partial builds, secret management when agents access live services, IP provenance of generated code, and human-checkpoint policies. File-based agent communication is actually well-suited to audit — but the author never makes that case.
  Harness design for long-running application development — read next: Read the EU AI Act's high-risk-system requirements, NIST AI RMF agent guidance if it exists yet, and any sober writing on agent observability stacks. Also: cross-reference Anthropic's enterprise-tier compliance docs.
• incompleteSays nothing about audit *response*. CloudTrail logging is necessary but not sufficient — who reads agent traces, how anomalies get detected at machine speed, and how rollback works when an agent has executed dozens of API calls in seconds are all absent.
  AWS MCP Server Reaches GA with Full API Coverage and IAM-Based Governance — read next: SIEM/SOAR for agent activity, AWS Detective for agent sessions, runtime anomaly detection for non-human identities.
• incompleteNo worked example, named tool, benchmark figure, or case study appears anywhere in the piece — every claim (about silent failures, retrieval blind spots, cost spirals, or the value of tracing) is asserted at the level of general architecture, with no evidence that any organization has implemented this stack or what it cost/saved them.
  Beyond the 200 OK: Architecting Observability for AI — read next: Practitioner case studies of AI observability rollouts with before/after incident or cost data (the corpus's extract-the-meter-was-always-running and extract-downstream-1-where-do-our-tokens-go are closer to this evidentiary bar).
• incompleteThe piece recommends real-time cost visibility and dashboards as a behavior-change lever, but does not address the failure mode of visibility metrics being gamed, cost-consciousness suppressing exploration that would have paid off, or who is accountable if switching to a cheaper model silently degrades quality on an edge case the shadow comparison didn't cover.
  Downstream #1: Where Do Our Tokens Go? — read next: Read against corpus material on tokenmaxxing and metric gaming for the failure modes of cost/usage visibility as a governance lever.
• incompleteRule 6 ('narrow down deterministically') assumes a pre-existing, well-structured backing dataset to narrow into (a station table with EVA numbers). The talk doesn't address what to do when the candidate set itself is fuzzy, unbounded, or must be discovered dynamically — which is exactly the harder procurement-cost-center use case the author mentions in the opening and then explicitly declines to return to.
  Presentation: From Thousands to One: Building LLM-Powered Selection Systems — read next: Look for techniques for LLM-driven selection when the candidate universe cannot be cheaply pre-filtered by deterministic code (e.g., open-world entity resolution, dynamic vendor/cost-center taxonomies).
• incompleteThe harness's provenance/audit trail is written back into the graph by the same agent (or an agent-driven summarization step) that performed the work, with no described independent, agent-can't-edit instrumentation layer verifying the record matches what actually happened — a self-reported chain of custody.
  Presentation: From Retrieval to Reasoning: Building Production-Ready Agentic AI Systems with Knowledge Graphs — read next: Read against extract-the-meter-was-always-running (especially its claim that an agent cannot be the system of record for its own authority, cost, or refusals, and that evidence must be captured at instrumentation points the agent cannot bypass) for a corpus-level argument that self-reported agent provenance specifically fails an auditor.
• incompletePatterns 2 and 4 (provenance and visibility) both assume that once everything is 'in the graph,' useful queries 'come for free,' but the talk gives no account of who designs and maintains these queries, how results stay honest as the graph schema evolves, or what happens when the harness itself — by her own admission in Q&A 'a bunch of devs clamoring something together' rather than a maintained product — breaks or drifts.
  Presentation: From Retrieval to Reasoning: Building Production-Ready Agentic AI Systems with Knowledge Graphs — read next: Look for material on internal-tooling sustainability and platform-as-product discipline to test whether an informal internal harness like this one survives team turnover or organizational scaling.
• incompleteThe append-only log captures tool calls, governance decisions, and memory operations -- the 'that' and 'how much' of a run -- but nothing here captures the model's actual reasoning for why it chose a given tool or path, which lives in prompt/reasoning content the piece never says is retained.
  Session Traces and Cost Controls Help Diagnose AI Agent Failures — read next: extract-the-meter-was-always-running#g1 makes exactly this computational-vs-inferential-sensor argument in general terms: metadata-by-default recipes answer 'that' and 'how much' more convincingly than the 'why' they promise, because the why is semantic and lives in content the minimization rule excludes by default.
• incompleteThe described instrumentation points (LLM-call spans, tool-execution spans) assume every action crosses a mediated API boundary the trace can capture. Nothing addresses what happens if an agent generates and executes its own code inside a sandbox, performing internal retries or state changes that never cross an instrumented boundary until a final result returns.
  Session Traces and Cost Controls Help Diagnose AI Agent Failures — read next: extract-the-meter-was-always-running#g2 raises this exact gap for the general case of chokepoint-based agent instrumentation, and points toward sandboxed code-execution architectures and network-egress-level containment as the fallback when application-layer instrumentation can be routed around.
• illogicalThe rule 'traces are for debugging, metrics are for alerting' is justified by one specific failure mode of one metrics backend (Prometheus's high-cardinality label explosion) but is stated as a universal, tool-agnostic principle, without acknowledging that this is a constraint of the chosen tool class rather than an inherent property of 'metrics' as a category.
  Session Traces and Cost Controls Help Diagnose AI Agent Failures — read next: Check whether metrics backends designed for high-cardinality data (e.g. purpose-built observability platforms rather than classic Prometheus) would change the recommendation, which would confirm the rule is tool-contingent rather than categorical.
• illogicalThe piece promises evidence 'to reconstruct why a run behaved the way it did' while simultaneously prescribing metadata-by-default, content-capture-by-exception as the data-minimization rule. But in an LLM loop the 'why' is largely semantic -- it lives in the prompt and tool-result content the design says not to store by default. Turn counts, byte sizes, retries, and policy decisions can show that a run wandered (which turn, how much, how many retries) but not what the model was reasoning toward when it chose to retry or picked an expensive model. The recipe answers 'that' and 'how much' more convincingly than the 'why' its own framing promises.
  The Meter Was Always Running — read next: extract-fowler-bockeler-static-sensors-iii draws exactly this computational-vs-inferential-sensor distinction for code modularity review and argues explicitly that deterministic (computational) sensors alone are insufficient where semantic interpretation is required -- the same lesson applied to a different loop.
• incompleteThe instrumentation points -- model gateways, tool proxies, execution sandboxes -- are assumed to be chokepoints the agent 'can't bypass.' That holds when every tool and model call must cross a mediated API. It's less obvious for an agent that writes and executes its own code inside a sandbox, which can perform internal retries, sub-calls, or state changes that never cross an instrumented boundary until a final result is returned. The piece doesn't address what 'below the agent' means once part of the loop is the agent's own generated code running inside the sandbox rather than a call the harness mediates.
  The Meter Was Always Running — read next: Sandboxed code-execution agent architectures and network-egress-level containment (the fallback chokepoint when application-layer instrumentation can be walked around from inside a sandbox) -- see extract-prompt-injection-to-data-exfil-in-3-hops for the analogous move of pushing the deterministic control down to a boundary the agent's own code can't route around.