Harness engineering for coding agent users

Birgitta Böckeler · article · 2026-04-20 source ↗ Reader 🖥️

Agent Harness Agentic Engineering Ashby's Law (Requisite Variety)Coding Agents Feedback Loops Orchestration Tax / Human Review Bottleneck

Coding agent users should engineer an outer harness of feedforward guides and feedback sensors — distributed across computational and inferential execution types and across the change lifecycle — to reduce review toil and increase system quality, but the behavioural harness (ensuring functional correctness) remains the hardest unsolved problem.

practical · expository — A practitioner framework article that organizes emerging harness-engineering practice into a structured mental model using control-theory concepts (feedforward/feedback). Primarily expository with prescriptive elements — taxonomizes rather than argues a contested point.

Structure

Part	Role
Defining the outer harness for coding agent users	Narrows the general 'Agent = Model + Harness' equation to the user-configurable outer layer of coding agents, distinguishing it from the builder-built inner harness
Feedforward and feedback taxonomy	Introduces the two control directions: guides that steer before the agent acts, and sensors that observe after and enable self-correction
Computational vs inferential execution types	Cross-cuts the feedforward/feedback distinction with a second axis: deterministic (CPU) vs probabilistic (GPU/LLM) controls, creating a 2x2 framework
The steering loop	Describes the human's role as iterating on the harness itself — converting recurring issues into improved controls
Timing: keep quality left	Maps feedforward and feedback controls across the change lifecycle (pre-commit, post-integration, continuous) using shift-left principles
Regulation categories (maintainability, architecture fitness, behaviour)	Categorizes what the harness regulates into three domains of increasing difficulty
Harnessability	Argues that codebase properties (type systems, module boundaries, frameworks) determine how governable the codebase is
Harness templates and Ashby's Law	Proposes pre-defined topology-based bundles of guides and sensors, justified by variety reduction
The role of the human	Frames human developer experience as an implicit harness that explicit harnesses attempt to externalize, but only partially

Key terms

harness (outer / user harness) Agent Harness
The user-configurable layer of guides, sensors, and feedback loops that a coding agent user builds around the agent, on top of the agent builder's built-in harness
guides (feedforward controls) Feedback Loops
Controls that anticipate the agent's behaviour and steer it before it acts, increasing the probability of good first-attempt results. Examples: AGENTS.md, skills, code mods, LSP
sensors (feedback controls) Feedback Loops
Controls that observe after the agent acts and help it self-correct. Particularly powerful when signals are optimized for LLM consumption (e.g. custom linter messages with correction instructions)
computational controls Feedback Loops
Deterministic, fast, CPU-run controls: tests, linters, type checkers, structural analysis. Milliseconds to seconds; reliable results
inferential controls Feedback Loops
Semantic, probabilistic, GPU/NPU-run controls: AI code review, LLM-as-judge. Slower, more expensive, non-deterministic, but capable of richer semantic judgment
steering loop Feedback Loops
The human practice of iterating on the harness by converting recurring agent failures into improved feedforward and feedback controls
harnessability Agent Harness
The degree to which a codebase's properties (type system, module boundaries, framework abstractions) make it amenable to effective harness controls
harness templates Agent Harness
Pre-defined bundles of guides and sensors matched to common service topologies (API services, event processors, dashboards), justified by Ashby's Law of Requisite Variety as a variety-reduction move
regulation categories
Three domains the harness regulates: maintainability (internal code quality), architecture fitness (non-functional characteristics), and behaviour (functional correctness)

Propositions

explicitcontestedCoding agent users can and should build an outer harness on top of the agent builder's built-in harness.
contradicts ←How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code) · p12 — Wu reports simple setups outperform heavily customized ones, cutting against Böckeler's recommendation that users build an elaborate outer harness on top of the built-in one.
extends ←Loop Engineering: Building Blocks, Adoption, and Impact · p2 — Adopts Böckeler's inner/outer harness distinction (cited explicitly) and adds a further layer above the outer harness: the loop that invokes the harness repeatedly over time rather than once.
extends ←GitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · p12 — Generalizes the outer-harness thesis to its extreme: at the small-model end the harness, not the model, is the dominant performance lever.
explicitA well-built outer harness serves two goals: increasing the probability the agent gets it right initially (feedforward), and self-correcting issues before they reach human eyes (feedback).
explicitcorroboratedFeedforward-only or feedback-only harnesses are each insufficient; both control directions are necessary.
supports ←The Anatomy of an Agent Harness · p7 — Strongest cross-author link — the anatomy explicitly cites 'Martin Fowler's Thoughtworks team' framing verification as guides (feedforward) versus sensors (feedback). Boeckeler p3: feedforward-only or feedback-only harnesses are each insufficient; both control directions are necessary. Same source, same claim, arriving from Akshay's synthesis.
explicitdevelopedFeedback sensors are most powerful when their signals are optimized for LLM consumption, such as custom linter messages that include self-correction instructions.
extends ←Maintainability sensors for coding agents · p3 — Bockeler's harness-engineering piece states in the abstract that feedback sensors are most powerful with LLM-optimized signals, citing custom linter messages with self-correction instructions as the example; this document supplies the concrete case study (the no-explicit-any formatter, the cyclomatic-complexity before/after comparison) that instantiates and confirms that exact claim.
explicitdevelopedComputational controls are cheap, fast, and deterministic; inferential controls are expensive, slow, and non-deterministic but provide richer semantic judgment.
refines ←Three more static code analysis sensors · p15 — Identifies modularity as a domain where the cheap deterministic computational tier cannot carry the load and inferential LLM judgment is required.
explicitdevelopedControls should be distributed across the change lifecycle according to cost, speed, and criticality, with cheaper checks shifted as far left as possible.
extends ←Maintainability sensors for coding agents · p12 — The abstract claim that controls should be distributed across the change lifecycle by cost/speed/criticality, cheapest shifted left, is exactly what this article's sensor taxonomy (during coding session -> pipeline -> repeated -> runtime, computational sensors run fast and often, inferential ones on a slower cadence) operationalizes with a concrete, working example.
explicitThe maintainability harness is the easiest to build because substantial pre-existing tooling exists; the behavioural harness is the hardest and least solved.
explicitNeither computational nor inferential sensors reliably catch the highest-impact problems: misdiagnosis, overengineering, misunderstood instructions.
refinesHarness design for long-running application development · p3 — Böckeler qualifies evaluator optimism: even a tuned skeptical evaluator is an inferential sensor, and neither sensor class reliably catches the highest-impact failures (misdiagnosis, overengineering, misread instructions).
explicitestablishedMost current high-autonomy coding agent workflows rely excessively on AI-generated test suites for behavioural verification, which is not good enough yet.
extendsPresentation: AI-First Software Delivery: Balancing Innovation with Proven Practices · p8 — Takes the brittleness of post-hoc AI-generated tests further: reliance on AI test suites is not good enough for behavioural verification in high-autonomy workflows generally.
refines ←[FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero · p13 — Offers a condition under which AI-generated tests become trustworthy for behavioural verification: TDD ordering instruments behaviour before implementation exists.
supports ←Harness Engineering · p2 — The missing-verification gap flagged here is the same territory the later article names explicitly as the unsolved 'behavioural harness' problem, with the added detail that current workarounds (AI-generated test suites) aren't good enough yet.
supports ←Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation · p8 — Slack's insistence that deterministic tests remain primary for critical logic — with agentic testing only complementary — echoes the caution against over-relying on AI for behavioural verification, though here the AI executes rather than authors the tests.
explicitdevelopedA codebase's harnessability is determined by its technical properties: strong typing, clear module boundaries, and framework abstractions increase it.
supportsWhy Rust is different, with Alice Ryhl · p11 — Both claim rich type-system/compiler feedback is what makes a codebase correctable by agents — Böckeler's 'harnessability' names the property Ryhl attributes to Rust.
extends ←Harness engineering: leveraging Codex in an agent-first world · p8 — Böckeler's fuller article names 'harnessability' abstractly as a codebase property (strong typing, clear module boundaries, framework abstractions) that determines how governable a codebase is; this piece is the concrete worked example — a fixed layered dependency structure per business domain, mechanically enforced — showing exactly what high harnessability looks like built from scratch.
explicitcorroboratedCommitting to pre-defined service topologies is a variety-reduction move (per Ashby's Law) that makes comprehensive harnessing more achievable.
supports ←Harness Engineering · p5 — Same convergence-on-fewer-topologies intuition, later crystallized in the fuller article as an explicit variety-reduction move justified via Ashby's Law.
explicitcontestedHarnesses are an attempt to externalize the implicit knowledge that human developers bring, but can only partially succeed.
contradictsSkill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI · p2 — If harnesses can only partially externalize human implicit knowledge, agent failures are not reducible to operator skill issues.
contradictsAgent Skills: Google-Backed AI Coding Agent Workflow · p4 — Böckeler holds that externalizing developers' implicit knowledge can only partially succeed, against the premise that Markdown instructions suffice to transmit complex engineering judgment.
explicitdevelopedThe goal of a good harness is not to eliminate human input but to direct it to where it is most important.
extends ←From AI Code to Trusted Software: Harness Engineering in Practice · p9 — Bockeler states the harness's goal is to direct human input to where it is most important; Frisinger operationalizes that into a concrete review-altitude shift (auditing verdicts and evidence, sampling deeply only where evidence looks thin) plus a maturity arc describing exactly where the human checkpoint sits at each stage.
explicitcorroboratedLegacy codebases with high technical debt face a paradox: the harness is most needed where it is hardest to build.
supports ←Harness Engineering · p7 — The retrofitting skepticism here (expressed via the static-analysis-overload analogy) is the same legacy-harness tension the later article names explicitly as a paradox: the harness is most needed where it's hardest to build.
extends ←How Microsoft Developers Use AI in Real-World Coding | BRK103 · p4 — Bockeler argues legacy/high-entropy codebases are where a test harness is most needed but hardest to build; this talk's regex-interpreter and memory-pool examples show AI providing a lighter-weight partial workaround — recovering comprehension and generating documentation/tests for orphaned legacy code even without a full harness.
explicitcorroboratedBuilding the outer harness is an ongoing engineering practice, not a one-time configuration.
supports ←Harness engineering: leveraging Codex in an agent-first world · p17 — Both converge on the same claim: harness engineering (here, 'discipline in the scaffolding') is an ongoing engineering practice rather than a one-time configuration — this piece supplies the five-month lived narrative behind that abstract claim.
explicitCoding agents lack social accountability, aesthetic judgment, organizational memory, and the ability to distinguish load-bearing conventions from mere habits.
implicitThe cybernetic framing (feedforward + feedback = governor) accurately describes what harness engineering is doing.
explicitcorroboratedTeams may start choosing tech stacks partly based on what harnesses are already available, rather than purely on technical merit.
supports ←Harness Engineering · p3 — This piece's early, tentative hypothesis ('teams may use harnesses as a starting point, then shape them over time') is the same claim the author later states almost identically and more confidently in her fuller framework article — this is the earlier form of that idea.

Arguments

Both feedforward and feedback controls are necessary for effective agent harnessing strength: high

Explicit premises:

Guides (feedforward) anticipate behaviour and steer before the agent acts
Sensors (feedback) observe after the agent acts and enable self-correction
Feedback-only leads to repeated mistakes; feedforward-only means rules are never validated

Implicit premises:

Agent behaviour is sufficiently unpredictable that anticipation alone cannot prevent all errors
Self-correction is only valuable if the initial guidance constrains the space of possible errors

Inferences:

By elimination — each direction alone is shown to be insufficient, therefore both are needed.

The behavioural harness is the most important unsolved problem in harness engineering strength: medium

Explicit premises:

Maintainability harness has abundant pre-existing tooling
Architecture fitness harness can leverage fitness functions
Behavioural harness currently relies on AI-generated tests, which are not trustworthy enough
Correctness is outside any sensor's remit if the human didn't clearly specify requirements

Implicit premises:

Functional correctness is ultimately more important than maintainability or architecture fitness
The quality of AI-generated tests cannot be assumed to match human-written specification tests

Inferences:

Argument from difficulty gradient: easiest problems have solutions, hardest problem (behaviour) does not yet.

Pre-defined service topologies make comprehensive harnessing more achievable strength: high

Explicit premises:

Ashby's Law states a regulator must have at least as much variety as the system it governs
An LLM-based coding agent can produce almost anything
Committing to a topology narrows the output space

Implicit premises:

The current variety of possible agent outputs exceeds what any practical harness can regulate
Topology constraints are acceptable trade-offs for the governance gains

Inferences:

Deductive application of Ashby's Law: reduce system variety to make regulation feasible.

Harnesses can only partially externalize human developer experience strength: high

Explicit premises:

Humans bring implicit knowledge: conventions, aesthetic judgment, organizational context, social accountability
Agents lack all of these qualities
Externalizing and making explicit what experience provides is expensive and must be prioritized

Implicit premises:

Some human knowledge is tacit and resists formalization (Polanyi's paradox)
Social accountability and aesthetic judgment cannot be replicated by rules or sensors

Inferences:

From the gap between tacit and explicit knowledge: if not all can be formalized, externalization must be partial.

Controls should be shifted as far left in the lifecycle as possible strength: medium

Explicit premises:

Earlier detection means cheaper fixes (CI/CD principle)
Fast/cheap controls (linters, fast tests) can run pre-commit
Expensive controls (mutation testing, broad review) should run post-integration

Implicit premises:

The cost of fixing agent errors scales with how far they propagate before detection
This CI/CD principle transfers directly from human-authored to agent-authored code

Inferences:

Argument by analogy: shift-left economics apply equally to agent-produced code.

Narratives

OpenAI team documented their harness: layered architecture enforced by custom linters, structural tests, and recurring 'garbage collection' drift scans with agent-suggested fixes. Their conclusion: 'Our most difficult challenges now center on designing environments, feedback loops, and control systems.'
→ Primary external validation that harness engineering is the central challenge, not model capability. Supports the lifecycle distribution argument (P6) and the ongoing-practice thesis (P15).
Stripe's minions system uses pre-push hooks with heuristic-based linter selection, emphasizes shift-left, and uses 'blueprints' to integrate feedback sensors directly into agent workflows.
→ Illustrates computational feedback sensors integrated into the agent loop and the shift-left principle in production at scale. Supports P6 and P4.
Thoughtworks teams tackle architecture drift with both computational and inferential sensors — improving API quality with agents + custom linters, improving code quality with a 'janitor army'.
→ Shows the computational/inferential distinction playing out in real team practice. Supports P5 and P15.
Colleagues using the 'approved fixtures' pattern selectively for behavioural verification, but acknowledging it is not a wholesale answer to the test quality problem.
→ Illustrates both a partial solution and the incompleteness of the behavioural harness. Supports P9 and P7.

Questions addressed

✅ How should coding agent users think about and structure the controls they build around their agents?
Use a 2x2 framework: feedforward guides vs feedback sensors, crossed with computational vs inferential execution types. Both directions are necessary.
✅ What mental model from control theory applies to agent harnessing?
The cybernetic governor model: feedforward anticipation combined with feedback self-correction, regulated by the human steering loop.
✅ Where in the software change lifecycle should different types of controls be placed?
Shift left: cheap/fast controls pre-commit, expensive controls post-integration, continuous drift detection outside the change lifecycle.
✅ What can and cannot be effectively harnessed today?
Maintainability is easiest (pre-existing tooling). Architecture fitness is feasible (fitness functions). Behavioural correctness is the hardest and least solved.
✅ What is the role of the human developer when agents are harnessed?
The human steers the harness by iterating on guides and sensors. The goal is not to eliminate human input but to direct it where it matters most.
✅ How do codebase properties affect harness effectiveness?
Harnessability varies: strong typing, clear module boundaries, and framework abstractions increase it. Legacy codebases face a paradox — harness is most needed where hardest to build.

Gaps

incompleteThe behavioural harness is acknowledged as 'the elephant in the room' but no concrete framework or methodology is proposed for solving it beyond mentioning approved fixtures as a partial pattern.
Read next: Read the approved fixtures pattern (lexler.github.io/augmented-coding-patterns) and explore specification-based testing approaches (property-based testing, contract testing) as potential behavioural harness components.
incompleteThe article does not address how to evaluate harness quality or coverage — it raises the question ('If sensors never fire, is that high quality or inadequate detection?') but offers no answer.
Read next: Explore meta-testing approaches: mutation testing for harnesses, harness coverage metrics, or adversarial testing of the harness itself.
incompleteThe cybernetic framing (feedforward/feedback, Ashby's Law) is adopted as the organizing metaphor but not critically examined. Control theory assumes a well-defined desired state and measurable error signal — software development often has neither.
Read next: Read Stafford Beer's work on viable systems and cybernetic management to assess how well control-theory concepts transfer to inherently creative/ambiguous engineering tasks.
incompleteNo discussion of the cost-benefit analysis of harness engineering itself — when is the investment in building and maintaining a harness worthwhile vs. simply relying on human review?
Read next: Look for empirical studies or experience reports quantifying harness ROI in agent-assisted development.
incompleteThe interaction between guides and sensors — coherence, contradiction, priority resolution — is raised as an open question but not explored.
Read next: Explore policy conflict resolution in cybernetic and multi-agent systems literature.

Closes gaps elsewhere

Presentation: AI-First Software Delivery: Balancing Innovation with Proven Practices · g6 — Answers who builds the verification infrastructure: coding agent users themselves build an outer harness as an ongoing engineering practice, distributing computational and inferential controls across the lifecycle by cost and criticality.
The Future Of Software Engineering · g2 — Harness engineering operationalizes the unnamed 'middle loop': feedforward/feedback controls, sensor design optimized for LLM consumption, and lifecycle placement of checks as an ongoing supervisory engineering practice.

Harness engineering for coding agent users

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Closes gaps elsewhere

Related documents