Agent Harness Engineering

Addy Osmani · article · 2026-04-20 source ↗ Reader 🖥️

Agent Harness Agent Tool Management Agentic Loop Capability Provisioning (right-sizing to the task)Agent Context Files (CLAUDE.md / AGENTS.md)Context Rot & Context Limits Multi-Agent Orchestration

The performance gap between what current AI models can do and what users actually experience is primarily a harness-engineering gap — the scaffolding of prompts, tools, hooks, context management, and feedback loops wrapped around the model — and closing it requires treating that scaffolding as a continuously evolving engineering artifact shaped by observed failures.

practical · expository — A practitioner synthesis of an emerging engineering discipline. Primarily expository — surveys and organizes contributions from multiple sources (Viv Trivedy, HumanLayer, Anthropic, Simon Willison) into a coherent framework, with prescriptive guidance throughout.

Structure

Part	Role
Opening frame: Agent = Model + Harness	Establishes the core equation and shifts attention from model selection to harness design
The 'skill issue' reframe	Provides evidence that harness changes alone can dramatically improve agent performance (Top 30 to Top 5 on Terminal Bench)
The ratchet: every mistake becomes a rule	Introduces the core methodology — treat failures as permanent signals that become constraints
Working backwards from behaviour (component walkthrough)	Catalogues the concrete harness components: filesystem/git, bash/code execution, sandboxes, memory/search, long-horizon execution, hooks, AGENTS.md
Harnesses don't shrink, they move	Addresses the objection that better models will make harnesses obsolete
Harness-as-a-Service and where this is going	Projects the trend toward harness SDKs and dynamic harness assembly

Key terms

harness Agent Harness
Every piece of code, configuration, and execution logic that isn't the model itself — prompts, tools, context policies, hooks, sandboxes, subagents, feedback loops, and recovery paths
harness engineering Agent Harness
The discipline of designing, iterating, and maintaining the scaffolding around an AI model so it can reliably complete tasks; coined by Viv Trivedy
ratchet
The practice of converting each observed agent failure into a permanent constraint (rule, hook, or check) that prevents recurrence, only relaxing constraints when model improvements make them redundant
context rot Context Rot & Context Limits
The degradation of model reasoning and task-completion quality as the context window fills up
Ralph Loop Agentic Loop
An autonomous coding loop where a hook intercepts the model's attempt to exit and re-injects the original prompt into a fresh context window, forcing continuation against a completion goal
HaaS (Harness-as-a-Service) Agent Harness
The shift from building on LLM APIs (which give a completion) to building on harness APIs (which give a runtime with loop, tools, context management, hooks, and sandbox primitives)
skill issue
Agent failures that appear to be model limitations but are actually harness configuration problems

Propositions

explicitestablishedA decent model with a great harness beats a great model with a bad harness.conf: high
supports ←The Anatomy of an Agent Harness · p13 — Osmani p1: 'a decent model with a great harness beats a great model with a bad harness.' This piece's 'the harness is the product' with the TerminalBench 20-rank jump from a harness-only change (identical weights) is the same claim with a hard benchmark number.
supports ←Claude Code: Best practices for agentic coding · p4 — This canonical 2025 guide's whole premise — that a deliberately low-level model rewards the context/tools you engineer around it — is a concrete, early instance of 'a decent model with a great harness beats a great model with a bad harness.'
supports ←GitHub - musistudio/claude-code-router: Use Claude Code as the foundation for coding infrastructure, allowing you to decide how to interact with the model while enjoying updates from Anthropic. · p11 — 'A decent model with a great harness beats a great model with a bad harness' — ccr operationalizes this by pairing Claude Code's harness with cheaper/decent models.
supports ←Don't train the model, evolve the harness · p1 — Concrete instance of 'a decent model with a great harness beats a great model with a bad harness': a frozen open model reached frontier parity at ~7x lower cost purely via harness rewrite.
extends ←Downstream #1: Where Do Our Tokens Go? · p1 — Bai et al.'s empirical data (16,000 trajectories, 8 models, one harness) provides quantitative corroboration for the qualitative claim that a great harness matters more than a great model.
supports ←How Anthropic Builds And How Engineering Will Change Soon | Thariq Shihipar · p4 — Thariq's 'harness gets more complicated as the model gets smarter' directly reinforces the corpus's 'a decent model with a great harness beats a great model with a bad harness' claim.
extends ←Podcast: Strands Agents with Clare Liguori · p2 — Sharpens 'a decent model with a great harness beats a great model with a bad harness' into a directional claim: over-scaffolding doesn't just fail to help a capable model, it actively degrades it.
supports ←This Week in AI: Rethinking the Agent Harness · p1 — Independently asserts harness choice now matters more than model choice, backed by Stanford results on bare-model vs harnessed performance gaps.
refines ←GitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · p12 — Confirms harness-beats-model but conditions it: the harness is the dominant lever specifically at the small-model end of the capability spectrum.
explicitestablishedThe gap between what today's models can do and what users see them doing is largely a harness gap.conf: high
supports ←Claude Code: Best practices for agentic coding · p4 — The guide's insistence that you must supply CLAUDE.md, tools, and workflow to get good results is exactly the claim that the gap between what the model can do and what users see is largely a harness gap.
supports ←Don't train the model, evolve the harness · p6 — Cross-author corroboration (Osmani) with hard numbers: 'the gap is largely a harness gap' is exactly what five harnesses spanning 3.5%-80.1% on the same model/judge/tasks demonstrate.
extends ←From AI Code to Trusted Software: Harness Engineering in Practice · p3 — Both extractions use 'harness' for the scaffolding around AI-driven work, but extract-agent-harness-engineering frames the harness gap as a model-capability-elicitation problem ('the gap between what models can do and what users see'), while this article extends the term into an organizational-governance and accountability problem (permissions, fail-closed gates, evidence, observability) — a broadening of the same core concept worth surfacing on its concept trajectory.
extends ←Loop Engineering · p5 — Names a layer above the harness gap: if the harness gap explains why models underperform in a single session, loop engineering names the next-order gap — most people still operate the harness by hand, turn by turn, instead of building a system that runs it for them.
explicit⚡ disputed & corroboratedMost agent failures are configuration problems, not model problems.conf: high
supports ←Don't train the model, evolve the harness · p3 — 'Most agent failures are configuration problems, not model problems' — directly evidenced by the biggest single gain being file handling (landing the deliverable where the judge expects), not intelligence, with zero extra model tokens.
contradicts ←Jimmy Acton's Post · p3 — Competing dominant-cause attributions for the same phenomenon: Acton locates agent production failure in teams' conceptual gaps about the lower AI-stack layers, while the harness-engineering view locates it in configuration, explicitly not in the model or in stack literacy.
explicitcorroboratedEvery line in a good AGENTS.md should be traceable back to a specific thing that went wrong.conf: high
supports ←How Claude Code works in large codebases: Best practices and where to start · p3 — Both converge on keeping CLAUDE.md lean and purposeful (every line earning its place) as a load-bearing principle of harness design, here generalized into guidance for layering root vs. subdirectory files.
explicitcorroboratedModels get post-training coupled to the harness they were trained against, creating overfitting to specific tool implementations.conf: medium
extendsHarness design for long-running application development · p6 — Adds the reverse coupling: not only do harness components encode model-version assumptions that go stale, models are post-trained against specific harnesses and overfit to their tool implementations.
supports ←The Anatomy of an Agent Harness · p12 — Osmani p5: 'models get post-training coupled to the harness they were trained against, creating overfitting to specific tool implementations.' Anatomy's co-evolution principle states exactly this — Claude Code's model learned its specific harness, and changing tool implementations can degrade performance.
extends ←The Problem Is Prompt Debt · p8 — Extends the model-harness co-training overfitting insight specifically to hand-written prompts, with concrete evidence (Berkeley enterprise-lock-in study, GPT-4o usage share) that a prompt tuned against one model's trained-in behavior fails unpredictably on another.
refines ←Don't train the model, evolve the harness · p5 — Osmani p5 says models get post-training coupled to the harness they were trained against (overfitting to tool implementations). Akshay refines the practical consequence: because of that coupling, structural CODE-layer fixes are model-portable while PROMPT tuning is model-specific and can hurt a different family — the asymmetry the coupling predicts, observed.
explicitcontestedAs models improve, the space of interesting harness combinations doesn't shrink — it moves to new failure modes at higher capability levels.conf: high
contradicts ←Stop Overengineering Your Agent Harness · p6 — Directly conflicts with the other harness piece's claim that 'as models improve, the space of interesting harness combinations doesn't shrink — it moves to new failure modes': this essay argues harness complexity should shrink and features get removed outright as capability is absorbed.
explicitestablishedSeparating generation from evaluation into distinct agents outperforms self-evaluation because agents reliably skew positive when grading their own work.conf: high
supports ←20 Loop Design Patterns Every AI Engineer Should Know · p1 — Generate->Critique->Rewrite's core claim (the generator is not the best judge of its own output; a separate critic finds what it missed) is a concrete instance of the corpus principle that separating generation from evaluation into distinct agents beats self-evaluation because agents skew positive grading their own work. The thread's best-founded claim.
supports ←Claude Code: Best practices for agentic coding · p15 — The 'one Claude writes, another verifies' pattern is a concrete, earlier instance of separating generation from evaluation into distinct agents because self-grading skews positive.
supports ←Presentation: The Multi-Agent Approach: Building Reliable and Controllable Software Development Automation · p14 — Directly echoes 'separating generation from evaluation outperforms self-evaluation because agents skew positive grading their own work.'
supports ←Loop Engineering · p7 — Restates, at the loop layer, the same maker/checker separation-of-duties claim already established at the harness layer: self-evaluation is unreliable, a distinct evaluator agent is needed.
extends ←What Survived Contact · p14 — Agent-harness-engineering argues evaluation should be separated from generation into a distinct agent; this doc pushes the separation up to the human hirer, arguing the objective's evaluation (the oracle) is the one thing the person directing the model cannot delegate at all.
explicitestablishedTen focused tools outperform fifty overlapping ones because the model can hold the menu in its head.conf: medium
supports ←The Anatomy of an Agent Harness · p10 — Osmani p8: 'ten focused tools outperform fifty overlapping ones because the model can hold the menu in its head.' Anatomy decision 6 (tool scoping) supplies concrete evidence: Vercel removed 80% of v0's tools and got better results; Claude Code achieves 95% context reduction via lazy loading; 'expose the minimum tool set needed for the current step.'
supports ←Designing a Multi-Agent System for Engineering Support at Scale: A Case Study From Grab · p2 — Grab's production finding that 30+ exposed tools cause unpredictable selection independently confirms that a small curated toolset outperforms many overlapping ones.
explicitcorroboratedThe right harness for your codebase is shaped by your failure history and cannot be downloaded as a generic package.conf: high
supports ←Notes from FOSE Europe · p12 — The claim that the friction of each team building its own harness is generative — and that dedicated harness teams are a mistake — supports the idea that the right harness is shaped by local failure history and cannot be downloaded as a generic package.
explicitdevelopedTop coding agents (Claude Code, Cursor, Codex, Aider, Cline) look more like each other than their underlying models do — harness patterns are converging.conf: medium
extends ←Loop Engineering · p6 — One-year-later empirical update to the harness-convergence claim: convergence has moved up a layer, from harness features (tools, context) to loop features (automations, worktrees, sub-agents) now native in both major products.
implicitThe model-harness co-training loop means the 'best' harness for a given model is not necessarily the one the model was trained inside.conf: medium
implicitdevelopedHarness engineering will eventually move from static configuration to dynamic, compiler-like assembly of tools and context.conf: low
extends ←Presentation: From Copy-Paste to Composition: Building Agents Like Real Software · p8 — Both converge on tool assembly moving from static, hand-authored configuration toward something dynamic and composable; this talk supplies a concrete mechanism (a versioned virtual-tools mapping/override layer) for the general trend the other document only predicts.

Arguments

Engineers should invest in harness engineering rather than waiting for better models. strength: high

Explicit premises:

A decent model with a great harness beats a great model with a bad harness (author's direct experience).
On Terminal Bench 2.0, the same model (Claude Opus 4.6) scored far lower in Claude Code than in a custom harness.
Viv's team moved from Top 30 to Top 5 by changing only the harness.

Implicit premises:

Model improvements are outside the engineer's control, while harness improvements are within it.
The rate of harness improvement per unit of engineering effort exceeds the rate of model improvement per unit of waiting time.

Inferences:

Therefore the highest-leverage engineering work for agent performance is harness design, not model selection.

Harness engineering is a permanent discipline, not a temporary stopgap until models improve. strength: high

Explicit premises:

Every harness component encodes an assumption about what the model can't do on its own (Anthropic).
When models improve, old scaffolding becomes unnecessary but new scaffolding is needed for newly reachable tasks.
Opus 4.6 eliminated context-anxiety scaffolding but created need for multi-day memory policies and multi-agent coordination.

Implicit premises:

There will always be a frontier of tasks just beyond current model capabilities.
The gap between raw model capability and reliable task completion is structural, not incidental.

Inferences:

The harness moves with the capability frontier rather than being absorbed by it.

Agent mistakes should be treated as permanent signals that become harness constraints (the ratchet). strength: medium

Explicit premises:

Failures are usually legible and attributable to specific missing constraints.
Adding a convention to AGENTS.md, a hook that blocks a destructive command, or a back-pressure signal from typechecking can prevent recurrence.
Rules should only be removed when a capable model makes them redundant.

Implicit premises:

Agent failure modes are stable enough to be worth encoding permanently.
The cost of maintaining a constraint is lower than the cost of repeated failures.

Inferences:

A failure-driven, additive approach to harness configuration produces the most effective harness over time.

Narratives

Terminal Bench 2.0 benchmark comparison: Claude Opus 4.6 in Claude Code vs. the same model in a custom harness, where the custom harness moved from Top 30 to Top 5
→ Primary evidence that harness design, not model selection, is the dominant performance variable
An agent ships a PR with a commented-out test, which gets merged by accident. This leads to updating AGENTS.md, adding a pre-commit hook to grep for .skip() and xit(), and adding a reviewer subagent that flags commented-out tests as blockers.
→ Concrete illustration of the ratchet methodology — showing how a single failure cascades into three layers of prevention
Opus 4.6 killed the context-anxiety failure mode (premature task completion near context limits), eliminating a class of scaffolding the author had been writing six months earlier
→ Illustrates the 'harnesses move, not shrink' thesis — showing a real example of scaffolding becoming dead code as models improve
Fareed Khan's architectural breakdown of Claude Code showing all harness concepts as named components in a production system
→ Validates the theoretical framework by showing it instantiated in a shipping product

Questions addressed

✅ What is the most impactful lever for improving AI coding agent performance?
Harness engineering — the scaffolding around the model — rather than model selection or waiting for better models.
✅ What are the concrete components of an agent harness?
System prompts, tools/MCPs, filesystem/git, bash execution, sandboxes, memory/search, compaction, hooks, planning/verification loops, subagent orchestration, and observability.
✅ Will better models make harness engineering obsolete?
No — as models improve, the harness space moves rather than shrinks; new capabilities unlock new failure modes requiring new scaffolding.
✅ How should engineers respond when an agent makes a mistake?
Treat the mistake as a permanent signal and engineer a constraint (rule, hook, or check) that prevents recurrence — the ratchet.
✅ How do model training and harness design interact?
There is a feedback loop: useful harness primitives get standardized into products, used in training the next model generation, making the model better at those primitives, which enables new harness patterns.
✅ What is the appropriate size and governance model for AGENTS.md?
Keep it short (under 60 lines), earn each line by tracing it to a specific past failure — pilot's checklist, not style guide.

Gaps

incompleteThe article does not address when harness complexity itself becomes a liability — the scaling problem of accumulating ratchet rules. At what point does the constraint set become too large for the model to follow or for the engineer to maintain?
Read next: Look for work on constraint prioritization, AGENTS.md hygiene, or automated rule pruning in harness configurations.
closed byThe Anatomy of an Agent Harness — PARTIAL. Osmani g1 asks when harness complexity itself becomes a liability — the scaling problem of accumulating ratchet rules, and at what point the constraint set is too large. The anatomy supplies a directional criterion: the 'future-proofing test' (a design is sound only if performance scales up with more powerful models WITHOUT added harness complexity) plus the scaffolding-shrinks stance (Manus rebuilt 5x removing complexity; Anthropic deletes planning steps as models internalize them). So complexity that a stronger model would render redundant is, by this test, a liability. It does NOT answer the quantitative half of g1 — no threshold for 'too large for the model to follow / the engineer to maintain,' no maintenance-cost model — so this is a design principle, not the boundary the gap asks for.
incompleteThe Terminal Bench 2.0 data point (Top 30 to Top 5 via harness change) is presented without detail on what specific harness changes drove the improvement, making it impossible to evaluate which components had the most leverage.
Read next: Read Viv Trivedy's original 'Anatomy of an Agent Harness' post for the detailed breakdown.
incompleteThe article focuses almost exclusively on coding agents. It does not explore whether the harness-engineering framework applies to non-coding agent domains (research, design, operations, etc.) or where it might break down.
Read next: Look for harness-engineering writing in non-coding agent contexts — research agents, customer support agents, design agents.
closed byStop Overengineering Your Agent Harness — Extends the harness-engineering framework beyond coding agents to support/sales/enterprise agents via the Maven Assistant case study, showing the framework still applies to non-coding domains but resolves differently there — low context complexity needs routing/guardrails/handoffs rather than the coding-agent toolkit of memory/compaction/sub-agents.
incompleteThe model-harness co-training feedback loop is identified but its risks are not explored: overfitting to specific tool implementations, lock-in to specific harness vendors, or narrowing of model generality.
Read next: Read research on post-training specialization tradeoffs and whether tool-specific fine-tuning degrades general capability.
closed byThe Anatomy of an Agent Harness — This article names and illustrates the model-harness co-training overfitting risk with a concrete case (Codex-5.3's apply_patch tool logic making the model brittle to patch-method changes) — partially closing the gap's 'risks are not explored' finding, though without quantifying lock-in cost or generalization loss.
incompleteThe article does not discuss cost or economics of harness engineering — when is the investment in harness refinement worth it vs. simply accepting some failure rate or switching models?
Read next: Look for ROI frameworks for agent infrastructure investment.
closed byStop Overengineering Your Agent Harness — Supplies an explicit decision rule for when harness investment is worth it: map the task on action-complexity and context-complexity axes, keep the harness minimal when both are low, and add infrastructure only when a real observed failure demands it.

Agent Harness Engineering

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents