Harness Engineering

Birgitta Böckeler · article · 2026-02-17 source ↗ Reader 🖥️

theoretical · expository reaction piece with speculative argument — A short (1126-word) 'thinking out loud' response to OpenAI's own 'Harness engineering' write-up and Mitchell Hashimoto's 'engineer the harness' post, from Böckeler's personal exploration series on Martin Fowler's site. Mostly theoretical — testing hypotheses about where harness engineering leads — but closes with a practical call to reflect on 'what's your harness today?'. Note: this is a distinct, earlier, much shorter piece by the same author than the existing corpus extraction extract-bockeler-harness-engineering (a later, fuller framework article, 'Harness engineering for coding agent users', published separately with a feedforward/feedback taxonomy).

Structure

Key terms

• Harness Agent Harness
  The tooling and practices — mixing deterministic approaches (linters, structural tests) and LLM-based approaches (context-providing agents, periodic 'garbage collection' agents) — used to keep AI coding agents in check and their output maintainable, adopted from OpenAI's and Mitchell Hashimoto's usage.
• Garbage collection (for harnesses)
  Agents that run periodically (not on each change) to find inconsistencies in documentation or violations of architectural constraints, fighting entropy and decay in the codebase and its knowledge base over time.
• Harness-friendliness / AI-friendliness (of a tech stack)
  A proposed new criterion for choosing a technology stack or codebase topology: whether a good, pre-built harness already exists for it, potentially outweighing traditional criteria like developer ergonomics or technical merit.

Propositions

• explicitOpenAI's harness for a 1-million-plus-line, 5-month, no-manually-typed-code product mixes deterministic and LLM-based components across three categories: context engineering (a continuously enhanced knowledge base plus access to dynamic context like observability data), architectural constraints (enforced by both LLM agents and deterministic linters/structural tests), and 'garbage collection' agents that periodically fight documentation and architectural decay.conf: high
• explicitOpenAI's write-up documents measures for long-term internal quality and maintainability, but has a real, named gap: it does not describe how functional or behavioral correctness is verified.conf: high
  supportsHarness engineering for coding agent users · p9 — The missing-verification gap flagged here is the same territory the later article names explicitly as the unsolved 'behavioural harness' problem, with the added detail that current workarounds (AI-generated test suites) aren't good enough yet.
• explicitHarnesses are likely to become the next generation of 'service templates': organizations will pick from a small set of harnesses for common application topologies, then diverge from them over time, risking the same forking and synchronization problem that already affects service-template adoption.conf: low
  supportsHarness engineering for coding agent users · p18 — This piece's early, tentative hypothesis ('teams may use harnesses as a starting point, then shape them over time') is the same claim the author later states almost identically and more confidently in her fuller framework article — this is the earlier form of that idea.
• explicitBuilding maintainable, trustworthy AI-generated code at scale requires constraining the solution space — specific architectural patterns, enforced boundaries, standardized structures — rather than allowing LLMs the unconstrained flexibility to generate in any language or pattern that early AI-coding hype assumed.conf: medium
• explicitAs coding shifts from typing to steering generation, organizations may converge on fewer tech stacks and codebase topologies than today, selected partly for 'AI-friendliness' — availability of a good harness — rather than purely for human developer experience or technical merit.conf: low
  supportsHarness engineering for coding agent users · p11 — Same convergence-on-fewer-topologies intuition, later crystallized in the fuller article as an explicit variety-reduction move justified via Ashby's Law.
• explicitIf codebase design patterns can be reliably harnessed, these topologies — not natural language prompting itself — may become the real abstraction layer for AI-assisted development, contrary to the common assumption that natural language is the new universal interface.conf: low
• explicitRetrofitting a harness onto an existing, high-entropy legacy codebase may often not be worth the effort, because such codebases are frequently 'so non-standardized and full of entropy' that harnessing would surface an overwhelming volume of violations — comparable to running a static-analysis tool for the first time on a codebase that's never had one.conf: medium
  supportsHarness engineering for coding agent users · p14 — The retrofitting skepticism here (expressed via the static-analysis-overload analogy) is the same legacy-harness tension the later article names explicitly as a paradox: the harness is most needed where it's hardest to build.
• explicitA genuinely effective harness is substantially more work than generating and maintaining a set of Markdown rules/instruction files — it requires extensive deterministic tooling plus significant code-design work, since code design itself is a large part of what an agent uses as context.conf: high

Arguments

Narratives

• OpenAI's own report: 5 months of harness-building work, resulting in 'a real product that's now over 1 million lines of code' built with 'no manually typed code at all.'
  → Grounds every hypothesis in the piece in a single, striking existence proof that a harness-first approach can scale to production size — while the author herself flags it as vendor-reported and unverified.
• The static-analysis-tool analogy: harnessing an un-harnessed legacy codebase is like 'running a static code analysis tool on a codebase that's never had one, and then drowning in alerts.'
  → Makes the legacy-retrofitting cost concrete via an experience most engineers recognize, supporting argument 2's skepticism about retrofitting harnesses onto old codebases.

Questions addressed

• ✅ What does a harness actually consist of, concretely, beyond the word itself?
  Partially: via the author's own three-category reading of OpenAI's write-up (context engineering, architectural constraints, garbage collection).
• ⭕ Does a good harness verify functional correctness, not just maintainability?
  Explicitly named as missing from OpenAI's write-up; the author sets the gap aside rather than resolving it within this piece.
• ⭕ Will harnesses converge tech-stack choices and codebase topologies the way service templates and golden paths already do for infrastructure?
  Posed as a genuine open question the single OpenAI case cannot settle; offered as speculation to test, not a conclusion.
• ⭕ Is retrofitting a harness onto an existing legacy codebase worth the effort?
  The author leans toward 'often not,' via the static-analysis-overload analogy, but treats it as a case-by-case judgment call rather than a settled answer.

Gaps

• incompleteThe author explicitly flags that OpenAI's write-up omits behavioral/functional verification, but then 'leaves that gap aside' for the rest of the piece without returning to it — so the single most consequential missing piece of the harness (does the code actually work, not just stay maintainable) goes unanalyzed in an otherwise fairly thorough piece.
  Read next: Read the author's own later, fuller article on harness engineering, which names the behavioural harness explicitly as 'the hardest and least solved' problem.
• uninformedThe piece explicitly asks readers to 'assume we can trust OpenAI's representation of the success of this,' naming OpenAI's vested interest in the claim — but then doesn't actually discount or stress-test any of the specific numbers (5 months, 1M+ lines, zero manually typed code) in light of that named incentive.
  Read next: Look for independent replications or critiques of OpenAI's harness-engineering claims from teams without a vendor stake in the outcome.
• incompleteThe service-template/golden-path analogy raises a fork-and-synchronization risk ('other teams often struggle to incorporate updates') but doesn't explore what would be structurally different about harness drift versus template drift — e.g., whether a harness's LLM-based components (which can self-update via 'garbage collection' agents) might resist staleness in ways a static service template cannot.
  Read next: Platform-engineering literature on golden-path/service-template staleness, and case studies on whether 'self-healing' harness components solve drift once deployed across multiple teams.
  closed byHarness engineering: leveraging Codex in an agent-first world — This primary source shows concretely how a harness's LLM-based components can resist the staleness that a static service template cannot: golden principles are encoded once, then enforced continuously by recurring background Codex tasks that scan for deviations, update quality grades, and open small, mostly auto-merged refactoring PRs — replacing a non-scaling manual weekly human cleanup ritual ('20% of the week' spent on 'AI slop'). This is a structurally different, self-correcting drift-resistance mechanism, answering the gap's open question about whether harness drift differs from template drift.
• incompleteThe speculation about tech-stack convergence ('AI-friendliness' as a selection criterion) doesn't address who bears the switching cost if an organization's current stack turns out to harness poorly — existing codebases, team skills, and integrations don't disappear just because a stack is less 'AI-friendly.'
  Read next: Organizational case studies on stack-migration cost under an AI-friendliness criterion, as distinct from traditional migration drivers (performance, hiring, vendor support).

Related documents

• Harness engineering for coding agent users
• AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
  Agent Harness Context Engineering
• The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2
  Agent Harness Context Engineering
• The Anatomy of an Agent Harness
  Agent Harness Context Engineering
• How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code)
  Agent Harness Golden Path
• Claude Code: Best practices for agentic coding
  Agent Harness Context Engineering
• Don't train the model, evolve the harness
  Agent Harness Context Engineering
• Downstream #1: Where Do Our Tokens Go?
  Agent Harness Context Engineering
• Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
  Agent Harness Context Engineering
• 12-Factor Agents: Patterns of reliable LLM applications — Dex Horthy, HumanLayer
  Agent Harness Context Engineering