Harness engineering: leveraging Codex in an agent-first world

Ryan Lopopolo (OpenAI) · article · 2026-02-11 source ↗ Reader 🖥️

Agent Governance (Internal)Agent Harness Agent Context Files (CLAUDE.md / AGENTS.md)Feedback Loops Golden Path Intent Debt Orchestration Tax / Human Review Bottleneck Progressive Disclosure

Building a product with zero manually-written code at scale (one million-plus lines, five months, a team that grew from three to seven engineers with throughput that increased rather than degraded) required redefining engineering work from writing code to designing environments, pushing all team knowledge into repository-local legible artifacts, mechanically enforcing architectural invariants and documentation freshness, and relaxing human-paced process norms (blocking merge gates, manual cleanup) in favor of continuous, agent-driven correction — because human time and attention, not code production, is the one truly scarce resource the whole system must be organized to protect.

practical · mixed — A first-person engineering retrospective/case study (OpenAI's own internal build) that narrates five months of experience and distills it into explicit, prescriptive lessons at each turn. This is the primary source both corpus extractions by Birgitta Böckeler (extract-bockeler-harness-engineering and extract-boeckeler-harness-engineering) react to and build on.

Structure

Part	Role
We started with an empty git repository	scene-setting / evidence
Redefining the role of the engineer	central reframing
Increasing application legibility	technique
We made repository knowledge the system of record	technique
Agent legibility is the goal	principle
Enforcing architecture and taste	technique
Throughput changes the merge philosophy	process change
What 'agent-generated' actually means / Increasing levels of autonomy	scope statement
Entropy and garbage collection	problem + solution
What we're still learning	conclusion / epistemic honesty

Key terms

Agent legibility
The property of a system — its application state, logs, metrics, architecture, and institutional knowledge — being directly inspectable and reasonable-over by the agent from within the repository and running application, on the principle that anything the agent can't access in-context while running effectively doesn't exist to it.
AGENTS.md as table of contents, not encyclopedia
A short (~100 line) instruction file that serves as a map with pointers into a deeper, structured docs/ directory (the actual system of record), rather than trying to contain all the agent's needed context directly.
Golden principles
Opinionated, mechanical rules encoded directly into the repository (e.g. 'prefer shared utility packages over hand-rolled helpers,' 'validate data boundaries, don't probe YOLO-style') that are enforced continuously by recurring background Codex tasks, replacing the earlier practice of humans manually cleaning up drift once a week.
Progressive disclosure Progressive Disclosure
Structuring the knowledge base so agents start from a small, stable entry point (the short AGENTS.md) and are taught where to look next (design docs, architecture docs, execution plans) rather than being overwhelmed with everything up front.
Enforce boundaries centrally, allow autonomy locally
The team's own explicit analogy between how they govern Codex (mechanically enforced architectural invariants, with freedom in how solutions are expressed inside them) and how a platform engineering organization leads many human teams.

Propositions

explicitOver five months, a team of initially three (growing to seven) engineers built an internal product to roughly one million lines of code, with zero manually-written code across application logic, tests, CI configuration, documentation, observability, and tooling — all written by Codex — in an estimated one-tenth the time hand-writing it would have taken.conf: high
explicitcontestedPull-request throughput per engineer increased, not decreased, as the team grew from three to seven engineers (averaging 3.5 PRs per engineer per day), contrary to the usual expectation that coordination overhead erodes per-person throughput as a team grows.conf: high
contradictsThe Orchestration Tax is You · p4 — Osmani's structural claim (by Amdahl's Law, parallelizing agent production cannot exceed the throughput of the serial human-judgment step) is in tension with this piece's reported empirical result — per-engineer PR throughput rising, not falling, as the team grew from three to seven — a genuine corpus tension rather than a resolved disagreement, since this piece never isolates whether human judgment time per PR fell or whether PR size/risk fell instead.
explicitThe engineering team's primary job shifted from writing code to designing environments, specifying intent, and building feedback loops that let Codex agents do reliable work — summarized by the team as 'humans steer, agents execute.'conf: high
explicitWhen the agent failed at a task, the team's diagnostic response was never 'try harder' — it was always to ask what capability was missing and make it both legible and enforceable for the agent, treating failure as a signal about the environment rather than about the model's effort or ability.conf: high
explicitA single large 'AGENTS.md' instruction file fails in four specific, predictable ways at scale — it crowds out task/code/doc context, makes all guidance equally unprioritized ('too much guidance becomes non-guidance'), rots because no one maintains a monolithic file, and resists mechanical verification (coverage, freshness, ownership, cross-links) — so AGENTS.md should function as a short table-of-contents pointing into a structured docs/ directory that is the actual system of record.conf: high
extendsThe Intent Debt · p18 — Osmani argues AGENTS.md should be an intent ledger, not auto-generated config; this piece supplies the structural reason it must also stay short and point elsewhere — a monolithic file suffers context scarcity, non-guidance, rot, and unverifiability, so the ledger has to be a short map into a separately-enforced docs/ system of record, not the encyclopedia itself.
explicitBecause anything the agent can't access in-context while running effectively doesn't exist to it, all team knowledge relevant to a decision — including a Slack discussion that aligned the team on an architectural pattern — must be pushed into repository-local, versioned artifacts (markdown, schemas, executable plans), or it remains invisible to every future agent run exactly as it would be invisible to a new human hire three months later.conf: high
supportsThe AI-Native SDLC playbook · p7 — Both independently arrive at the same requirement from opposite directions in the same batch: institutional/team knowledge must become repository-resident, versioned, and machine-readable (CLAUDE.md/skills; docs/ as system of record) or it is effectively invisible to every future agent session.
explicitThe team mechanically enforces the knowledge base's integrity: dedicated linters and CI jobs validate that documentation is up to date, cross-linked, and structured correctly, and a recurring 'doc-gardening' agent scans for documentation that no longer reflects real code behavior and opens its own fix-up pull requests.conf: high
explicitRather than micromanaging implementation details, the team enforces architectural invariants — a fixed, strictly validated layered dependency structure per business domain, with cross-cutting concerns entering through a single explicit 'Providers' interface — mechanically via custom linters and structural tests, while leaving agents free within those boundaries; the team explicitly likens this to how a platform engineering organization leads many human teams: 'enforce boundaries centrally, allow autonomy locally.'conf: high
extendsHarness engineering for coding agent users · p10 — Böckeler's fuller article names 'harnessability' abstractly as a codebase property (strong typing, clear module boundaries, framework abstractions) that determines how governable a codebase is; this piece is the concrete worked example — a fixed layered dependency structure per business domain, mechanically enforced — showing exactly what high harnessability looks like built from scratch.
explicitArchitectural rigor that human-only teams typically postpone until they have hundreds of engineers (strict layering, validated dependency direction) became an early prerequisite for this three-to-seven-person, fully agent-generated codebase, because the constraints are specifically what allow speed without decay or architectural drift.conf: high
explicitCustom lint error messages are deliberately written to inject remediation instructions directly into the agent's context, turning a lint failure into an actionable fix-it prompt for the agent rather than a bare rule violation.conf: high
explicitAs agent throughput increased, conventional engineering norms (strict blocking merge gates, blocking on flaky-test investigation before merging) became counterproductive and were relaxed, because once agent throughput far exceeds human attention, corrections are cheap and waiting is expensive — a tradeoff the team explicitly says 'would be irresponsible in a low-throughput environment' but is 'often the right tradeoff' here.conf: high
explicitOnce enough of the development loop (testing, validation, review, feedback handling, recovery) was encoded into the system, the repository crossed a threshold where, given a single prompt, Codex can drive an entire feature end to end — reproduce a bug, record video evidence, implement and validate a fix, open a PR, respond to feedback, remediate build failures, escalate only when judgment is required, and merge — but this capability is explicitly scoped by the authors as dependent on this specific repository's structure and investment and 'should not be assumed to generalize... at least, not yet.'conf: high
explicitFull agent autonomy introduces its own entropy problem: Codex replicates existing patterns in the repository, including suboptimal ones, so quality drifts over time; the team's initial manual response — one engineer-day per week ('20% of the week') spent removing 'AI slop' — did not scale.conf: high
explicitThe team's working answer to agent-driven entropy is to encode 'golden principles' (opinionated, mechanical rules) directly into the repository and run a recurring background-agent cleanup process that scans for deviations, updates quality grades, and opens small, fast-reviewable (under a minute), often auto-merged refactoring pull requests — paying down resulting technical debt continuously in small increments rather than letting it compound into painful bursts.conf: high
explicitHuman taste and judgment get captured once — through review comments, refactoring PRs, and user-facing bug reports — and are then enforced continuously and mechanically on every subsequent line of code, rather than being re-applied by a human reviewer each time a similar case recurs.conf: high
explicitThe whole account is explicitly bounded by acknowledged uncertainty: the team states it does not yet know how architectural coherence evolves over years in a fully agent-generated system, where human judgment adds the most leverage, or how the system will change as underlying models become more capable.conf: high
explicitThe team's own retrospective frames its central lesson as: discipline in agent-generated software now shows up in the scaffolding (environments, feedback loops, control systems) rather than in the code itself — reframing 'engineering discipline' away from code-level craft and toward environment design as the locus of rigor.conf: high
supportsHarness engineering for coding agent users · p15 — Both converge on the same claim: harness engineering (here, 'discipline in the scaffolding') is an ongoing engineering practice rather than a one-time configuration — this piece supplies the five-month lived narrative behind that abstract claim.

Arguments

Rigorous architectural constraint (strict layering, validated dependencies, mechanically enforced invariants) should be adopted as an early-stage prerequisite for agent-generated codebases, not deferred until headcount scale. strength: high

Explicit premises:

Agents are most effective in environments with strict boundaries and predictable structure.
Without architectural constraint, agent-replicated patterns (including suboptimal ones) drift and compound over time (entropy).
Once an invariant is encoded as a mechanical check, it applies to every future agent run automatically, unlike a norm that depends on an individual human remembering and applying it.

Implicit premises:

The usual reason organizations defer heavy architectural governance — it isn't worth the overhead at small team scale — assumes a human rate of code production; that assumption breaks once an agent can produce the violation volume of a much larger team.

Inferences:

The breakeven point for 'when architecture investment pays for itself' shifts dramatically earlier once code is agent-generated, because production volume scales independently of human headcount.

Blocking, human-attention-gated merge norms should be relaxed as agent throughput increases, even though this would be the wrong call for a human-paced team. strength: medium