Presentation: The Agent Harness: Control Planes, Invariants, and Approval Boundaries for Production AI Agents

Vinoth Govindarajan · rss · 2026-09-21 source ↗ Reader 🖥️

practical · mixed — Argumentative in structure (five named design rules, each defended and generalized into a review checklist) but carried almost entirely by narrative case studies — real GitHub-issue incidents from the open-source OpenClaw project — rather than abstract argument. The talk is explicitly a call to action: run the 'Run Receipt Audit' on your own agent system.

Structure

Key terms

• Harness Agent Harness
  The non-model system around the LLM that decides whether a model's proposal becomes a committed, provable production action — the layer that owns state, orders mutations, bounds work, constrains authority, and proves outcomes. Not just 'the code around the model' in a loose sense; specifically the boundary that holds production accountability the model itself cannot hold.
• Receipt
  Evidence that survives after a run and proves what happened at the boundary the user actually relies on: what woke the system, what state it inherited, what was proposed, what policy evaluated, what executed, and what the user-visible edge confirmed. Sharply distinguished from a transcript (what the agent said) and a tool return value (what the tool claims happened inside its own boundary).
• Silent success
  A failure shape where the user-visible channel reports (or appears to report) success while the durable record behind it has a hole — worse than a crash because it gives the operator no reason to doubt the system.
• Lane
  The single-writer commit boundary the harness assigns per session: same-process writes serialized with a queue, cross-process writes protected with a lock, so that concurrency is possible across sessions but exactly one writer is active within a given mutable session boundary at a time.
• Own the state / state ownership
  Not a human or organizational owner, but the specific system boundary whose persistent record is the source of truth for a fact and that provides a replay path to reconstruct it later. 'State ownership is the boundary for recovery,' distinguished from mere storage ('storage is where just the bytes live').
• Approval (as a scoped object)
  A structured, time-bound execution artifact with actor, session, tools, arguments, lifetime, and outcome, that must be validated at the moment of use — not a persistent, vague memory that a human clicked something once. 'Approval is not a button, it is a scoped execution state.'
• Capability drift
  A specific failure where a tool disabled at the policy/UI control-plane layer still registers as available at the provider/runtime layer, so the model can still request something an operator believed was turned off — a divergence between the policy surface and the runtime surface.

Propositions

• explicitProduction AI agent failures largely occur not from model hallucination but from harness gaps, where the model behaves coherently but the surrounding system fails to record, order, bound, authorize, or prove its actions.conf: high
• explicitA model can only propose an action; the harness decides whether the proposal becomes reality, and the receipt is the evidence that it did.conf: high
• explicitA 'silent success' — where the user-visible channel reports success but the durable record behind it has a hole — is worse than a crash, because a crash gives a visible boundary and a replay point, while a silent success gives the operator no reason to doubt the system.conf: high
• explicitEvery fact an agent depends on later needs exactly one owner and one replay path; if a system cannot reliably reconstruct a fact later, it does not really own it.conf: high
  extendsArticle: Implementing Durable Workflows on Postgres Without an External Orchestrator · p3 — Applies the same 'one owner, one replay path' invariant this talk states in general harness terms to durable-workflow engineering's concrete mechanism (a unique-constraint checkpoint), grounding the same design principle in a specific, testable implementation.
• explicitInternal liveliness/control signals (such as a heartbeat token) must never be allowed to cross into user-facing delivery state; a system that misclassifies internal state as deliverable work can silently suppress real output for extended periods.conf: high
• explicitWhen two writers can touch the same mutable state, 'last writer wins' is not a valid consistency model; the harness must own and enforce a single ordered commit path (serialize same-process writes with a queue, protect cross-process writes with a lock).conf: high
  supportsArticle: Implementing Durable Workflows on Postgres Without an External Orchestrator · p2 — Both converge on 'exactly one worker/writer claims a mutable unit of work at a time' as the core concurrency invariant — this talk's harness-level 'lane' and Varma's database-level SKIP LOCKED queue are the same pattern at different layers.
• explicitThe correct concurrency model is parallel across sessions but single-writer within a session: many sessions and read-only work can run concurrently, but only one active writer is allowed per mutable session boundary.conf: high
• explicitOrdering failures are experienced by users as agent 'personality' (forgetful, dead, confused) rather than as an infrastructure bug, which makes ordering a product feature rather than an implementation detail.conf: high
• explicitIn production, silence from a tool call cannot be treated as neutral: every unit of work needs a defined terminal state (success, failure, timeout, cancellation) enforced via a lease, a watchdog, and an interrupt, or a single missing result can hold an entire session hostage.conf: high
  extendsArticle: Implementing Durable Workflows on Postgres Without an External Orchestrator · p4 — Generalizes the lease-and-sweeper pattern from workflow-step crash recovery to agent tool-call liveness and timeout handling — 'a lease gives runs and tools a deadline' is the same mechanism applied one layer up.
• explicitAn approval is a scoped, structured execution object — actor, session, tools, arguments, lifetime, outcome — that must be validated at the moment of use, not a persistent, vague memory that a human clicked something once; expired approvals should terminate rather than retry indefinitely.conf: high
  supportsGoogle's Genkit Ships Agents API with Detached Turns and Human-in-the-Loop for TypeScript and Go · p4 — Both treat a paused human-approval/interrupt point as requiring re-validation against runtime/session state at resume time rather than blind trust in whatever payload arrives — this talk supplies the concrete field list (actor, session, tools, arguments, lifetime, outcome) that Genkit's 'validates against session history' leaves unspecified.
  supportsGoogle's Genkit Ships Agents API with Detached Turns and Human-in-the-Loop for TypeScript and Go · p10 — This talk's insistence that an approval must be re-validated at execution time, and that expiry must terminate rather than loop, is the same stance as treating a resume payload as untrusted input by default.
• explicitA tool being schema-visible to the model (offered as an option) is not the same as a tool being executable; identity, policy, approval, arguments, and scope must all converge before an action happens, and if the policy surface and the runtime surface diverge (a 'disabled' tool that still registers as available), policy becomes decoration rather than a boundary.conf: high
• explicitA transcript proves only what the agent said, and a tool's return value proves only what happened inside its own execution boundary; neither is proof that the user-visible outcome actually occurred — only a receipt validated at the user-visible edge counts as proof.conf: high
  extendsThe Anatomy of an Agent Harness · p7 — Sharpens 'verification separates toy demos from production agents' into a stronger, more specific claim: verification is not enough unless proof specifically crosses the user-visible edge, not merely an internal rules-based/LLM-as-judge check.
• explicitThe specific failure modes OpenClaw's incidents illustrate (idempotency, ordering, locking, retries, state boundaries) are largely not new to distributed systems; what changes with agents is that a non-deterministic planner, dynamically chosen tools, and events arriving from many surfaces at once make these familiar failures easier to trigger and harder to explain.conf: high
• explicitcorroboratedThe Harness Blueprint (event -> session key -> lane -> throttle -> tools -> audit trail) generalizes across essentially all agent products — personal assistants, coding agents, support agents — because in every case the model's API is stateless between turns and the harness must reassemble working context each time.conf: high
  supports ←Coding Agent Development Workflows · p4 — Tune's hand-built verify→review→check→submit→autonomous-fix pipeline is an independent, practitioner-built instance of the same event→session→lane→throttle→tools→audit-trail harness blueprint, arrived at from a coding-agent angle rather than a production-agent-reliability angle.
• implicitBecause the harness, not the model, is what makes an agent auditable and recoverable, harness quality is the binding constraint on whether an agent can be safely operated in production, independent of model capability.conf: medium
• implicitTreating approval as a scoped, time-bound, re-validated object rather than a persistent capability grant is itself a structural defense against excessive-agency and prompt-injection-style failures, because it forces authority to be re-derived at the moment of action rather than assumed from an earlier interaction.conf: medium

Arguments

Narratives

• A user asked the agent to remember a refund for a customer; the assistant replied that it would remember for the next turn, and the user saw no errors — but the durable memory record was never actually written, so the future turn inherited a hole.
  → Anchors the talk's opening thesis that a failure which looks successful at the user-visible edge is more dangerous than a crash, and directly motivates 'own the state.'
• An agent went silent for 64 hours because an internal HEARTBEAT_OK liveliness token got misclassified as pending user-facing delivery work, so each new heartbeat saw pending work already in flight and kept skipping.
  → Illustrates that internal control signals must be architecturally separated from user-facing delivery state; the fix was making the state machine stricter, not the model smarter.
• Two callers independently loaded, modified, and saved the same commitment record (a classic read-modify-write race); neither writer was individually wrong, but with no serialization around the cycle, one user's correction silently disappeared.
  → Demonstrates the 'last writer wins is not a consistency model' failure and motivates a single ordered commit path per mutable state boundary.
• A session's log recorded a tool call whose matching result never arrived — the process was killed, the network dropped, or the tool timed out — leaving the loop waiting indefinitely while new user messages piled up behind the stuck work.
  → Motivates 'bound the work': every tool call needs a life cycle with leases, watchdogs, interrupts, and a terminal receipt.
• A user approved an elevated execution, but the follow-up run lost the scoped execution context needed to carry it out, and an expired callback kept retrying without ever terminating or informing the user why nothing was happening.
  → Motivates treating approval as a scoped, expiring object with actor/session/tools/arguments/lifetime/outcome rather than a vague, persistent memory of a click.
• A tool disabled at the control-plane/UI layer still registered as available at the provider/runtime layer, so the model could still see and request a capability an operator believed was turned off.
  → Illustrates 'capability drift' and motivates converging policy surface and runtime surface before any capability is shown to the model, tying the harness argument to OWASP's 'excessive agency' risk category.
• A tool reported that a message had been sent successfully, but the visible client rendered nothing, so the user never actually saw the message the system internally believed had been delivered.
  → Motivates 'prove the action': internal tool success is not external proof, and the receipt must specifically cross the user-visible edge.
• An OpenClaw contributor ('Peter') is said to have submitted roughly 600 pull requests in a single day by closing the loop on proof — checking screenshots, logs, and before/after evidence — before issuing each one.
  → Offered as evidence that a proof-required review culture, rather than raw model capability, is what enables high-velocity, low-incident production agent development.

Questions addressed

• ✅ Why do production AI agents fail beyond model hallucination?
  Because the harness — not the model — is where state ownership, mutation ordering, work bounding, authority scoping, and outcome proof actually live; the model can only propose.
• ✅ How should concurrent state mutations from multiple valid event sources (users, webhooks, heartbeats, sub-agents, tool results) be handled?
  One ordered commit path per mutable state boundary: parallel across sessions, single writer within a session, same-process writes serialized with a queue, cross-process writes protected with a lock.
• ✅ How should a harness distinguish an internal control signal (like a heartbeat) from genuine user-facing work?
  Internal signals must never be allowed to cross into delivery state; the state machine has to classify whether something is actually deliverable work before acting on it.
• ✅ What makes an approval trustworthy enough to authorize an elevated or irreversible action?
  It must be a scoped, structured, expiring object (actor, session, tools, arguments, lifetime, outcome), re-validated at execution time rather than treated as a standing memory of a past click.
• ✅ How do you know an agent's action actually happened from the user's perspective, rather than merely succeeding inside some internal boundary?
  Require a receipt validated at the user-visible edge; neither a transcript nor a tool's own return value is sufficient proof on its own.
• ⭕ Is this five-part harness architecture the only viable approach, or could other architectures also work for production agents?
  Raised explicitly by an audience member; Govindarajan answers only that the pattern generalizes because the model API is stateless and multi-session/locking pressures recur everywhere, but he does not claim it is the only workable architecture and says it may still be 'too early to say' whether alternatives exist.
• ⭕ Who or what is accountable when the harness itself has a design bug (e.g. the heartbeat state machine's misclassification), as distinct from cases where a human genuinely made a bad call?
  The talk treats these incidents purely as engineering fixes ('the fix was making the state machine stricter') without addressing the organizational-accountability question of who owns and is responsible for harness correctness itself.

Gaps

• incompleteThe 'harness owns the order' prescription (serialize same-process writes with a queue, cross-process writes with a lock) assumes a single reachable, reliable lock/queue authority. It does not address genuinely distributed harness deployments — multi-region or multi-process across data centers — where the lock/queue mechanism itself must be distributed and has its own failure modes (partition, lease expiry, split-brain).
  Read next: Read alongside extract-varma-durable-workflows-postgres, which operationalizes exactly this at the database layer (SELECT ... FOR UPDATE SKIP LOCKED, lease-and-sweeper crash recovery) and whose own gaps (g1, g5) already flag where 'exactly-once claim' guarantees and cross-engine locking semantics break down under contention and heterogeneity.
• incompleteThe claim that OpenClaw's contributor 'Peter' submitted roughly 600 pull requests in a day because of the proof-required review culture is offered with no independent verification, no baseline for comparison, and no accounting for PR size or quality — the same self-reported-evidence problem the corpus has flagged in other vendor/practitioner production-metrics talks.
  Read next: Read alongside extract-anthropic-details-how-it-contains-claude#g4, which raises an identical evidentiary-independence gap about self-reported production security metrics, to see whether unverified headline numbers are a systemic weakness across 'here is our production agent's numbers' talks.
• incompleteThe talk specifies what makes an approval trustworthy in the moment (scoped actor/session/tools/arguments/lifetime/outcome) but never addresses what happens when 'Approve' mode becomes routine at scale — the risk that a human approval gate degrades into a rubber stamp under high volume, which would silently defeat the entire scoping design it argues for.
  Read next: Read alongside extract-hbr-stop-automating-design-new-processes#g5 (the approval-fatigue-at-scale gap) and #p11 (irreversible actions should default to 'Approve' mode) to add the human-factors dimension this talk's authority-scoping design omits.
• incompleteThe talk's central contract ('a model proposes, the harness commits') implicitly treats model and harness as cleanly separable layers, but does not address whether that separation survives once a model is post-trained against a specific harness's proposal and approval format — i.e., whether the model itself can come to implicitly rely on, or probe for, known enforcement gaps in a harness it was trained inside.
  Read next: Read alongside extract-agent-harness-engineering#p5 and #p11 (model-harness co-training coupling, overfitting to specific tool implementations) to test whether 'model proposes, harness commits' remains a clean separation once the model has been trained inside a specific harness's proposal/approval loop.

Closes gaps elsewhere

• Google's Genkit Ships Agents API with Detached Turns and Human-in-the-Loop for TypeScript and Go · g2 — Gives a concrete, generalizable answer to what a resume/approval payload should validate and why: actor, session, tools, arguments, lifetime, and outcome, with expiry required to terminate authority rather than let a callback retry indefinitely — directly answering the gap's question of what exactly should be checked and what attack class (stale or over-broad authority) it guards against, beyond Genkit's specific interrupt-resume mechanism.

Related documents

• The Anatomy of an Agent Harness
• Article: Implementing Durable Workflows on Postgres Without an External Orchestrator
• Claude Code Source Leak Megathread
  Agent Governance (Internal) Agent Harness Agent Security
• The Claude Code active attack didn't stop. 294,842 secrets stolen from 6,943 machines. It evolved and now spreads through Python too and uses Claude Code itself to steal your secrets. The risk to your credentials just got bigger.
  Agent Governance (Internal) Agent Security Sandboxing
• GitLab Warns That AI Agent Sandboxes Are Only as Secure as Their Network Access
  Agent Governance (Internal) Agent Security Sandboxing
• GKE Security Blueprint Joins Growing List of Cloud AI Frameworks
  Agent Governance (Internal) Agent Security Sandboxing
• Beyond Zero: Google Publishes Successor to BeyondCorp
  Agent Governance (Internal) Agent Security Sandboxing
• Vibe Coding Agent Security And Evaluation
  Agent Harness Agent Security Sandboxing
• Technology Radar, Volume 34
  Agent Harness Agent Security Durable Execution & Workflow Checkpointing
• 5 Signs That Authenticated API Traffic Is Actually Malicious
  Agent Governance (Internal) Agent Security