Harness design for long-running application development

anthropic.com · article · 2026-03-24 Reader 🖥️

practical · mixed — Primarily practical: argues for specific design patterns (GAN-inspired generator/evaluator, three-agent planner/generator/evaluator architecture, stripping components when models improve) in service of building long-running autonomous coding harnesses. Mixed form: expository (defining failure modes like context anxiety, self-evaluation leniency), narrative (the retro-game-maker and DAW case studies), and argumentative (a methodological stance about how to evolve harnesses as models improve). Vendor undertone: every recommendation is anchored to specific Claude model versions (Sonnet 4.5 → Opus 4.5 → Opus 4.6) and Anthropic-built scaffolding (Claude Agent SDK, frontend-design skill, Playwright MCP).

Structure

Key terms

• Harness Agent Harness
  The orchestration scaffolding surrounding one or more Claude agents — agent roles, prompts, tools (MCPs), inter-agent communication mechanisms, and loop logic — that enables the model to perform a complex task that it cannot do in a single inference pass. Implicitly treated as a stack of removable 'components,' each encoding an assumption about a specific model weakness.
• Long-running Long-Running & Managed Agents
  Multi-hour autonomous coding sessions (concretely: 3.8 to 6 hours in the case studies) that produce 'rich full-stack applications' end-to-end without human intervention. Not formally defined by a time threshold but operationalized in the examples.
• Context anxiety Context Rot & Context Limits
  A model behavior in which the agent prematurely wraps up work as it approaches what it believes is its context window limit. Distinct from actual coherence loss; it is the model's anticipatory truncation of effort. Author claims Sonnet 4.5 exhibits this strongly; Opus 4.6 largely eliminated it.
• Context reset Context Rot & Context Limits
  Clearing the context window entirely and starting a fresh agent, combined with a structured handoff artifact that carries the previous agent's state and next steps. Contrasted with compaction.
• Compaction Context Rot & Context Limits
  Summarizing earlier parts of a conversation in place so the same agent can keep going on a shortened history. Preserves continuity but does not eliminate context anxiety because the agent is still 'the same agent.'
• Self-evaluation leniency Multi-Agent Orchestration
  The empirical tendency of an agent asked to grade its own output to praise the work confidently even when quality is mediocre. Especially pronounced for subjective tasks lacking binary verification, but also present on verifiable tasks. Not fully eliminated by separating evaluator from generator, but greatly reduced.
• Generator / Evaluator (GAN-inspired pattern) Multi-Agent Orchestration
  An agent architecture borrowed conceptually from Generative Adversarial Networks: one agent produces, a separate agent grades against explicit criteria and writes critique, feedback flows back to generator for next iteration. Author concedes the evaluator is itself an LLM still 'inclined to be generous,' but separating it makes skepticism tractable.
• Sprint contract Agent Planning Artifacts
  A negotiated agreement between generator and evaluator agents, before code is written, defining what 'done' means for a chunk of work — specifically: what will be built and how completion will be verified. Bridges the gap between a high-level spec and testable implementation. Communicated via files.
• Load-bearing (harness component) Riding Model Improvements
  A piece of the harness without which the final output's quality measurably degrades. The methodological test is to remove components one at a time and measure impact. Whether a component is load-bearing depends on the model version.
• Planner / Generator / Evaluator (three-agent harness) Multi-Agent Orchestration
  The Anthropic-Labs canonical architecture: planner expands a short user prompt into a full product spec; generator builds in sprints (V1) or continuously (V2) against the spec; evaluator drives the running app via Playwright MCP and grades against criteria with hard thresholds, returning detailed feedback.
• Grading criteria Evals
  Explicit, written prompts shared between generator and evaluator that convert subjective judgments (e.g. 'is this design good?') into concrete graded dimensions (design quality, originality, craft, functionality). Criteria wording itself steers the output — e.g. 'museum quality' pushes designs toward visual convergence.
• AI slop AI Slop
  Highly generic, telltale AI-generated visual patterns — explicitly: 'purple gradients over white cards,' unmodified stock components, library defaults. The evaluator's originality criterion is explicitly designed to penalize these.

Propositions

• explicitestablishedLong-running coding agents predictably fail in two specific ways: context-window coherence loss (including context anxiety, where the model prematurely wraps up) and self-evaluation leniency (where the model praises its own mediocre work).conf: high
  supports ←LLMs Corrupt Your Documents When You Delegate · p6 — DELEGATE-52's finding that degradation worsens with longer interactions and larger inputs independently corroborates the long-running coherence-loss failure mode.
  supports ←Project Glasswing: what Mythos showed us · p3 — Quantifies the context-coherence failure mode in the field: a single session on a 100k-line repo covers ~0.1% of the surface before compaction discards earlier findings.
  supports ←Prompting Claude Fable 5 · p14 — Same 'context anxiety' failure mode (the model prematurely wraps up as it senses limits). The Fable 5 docs confirm it as a named behavior and add a concrete mitigation the harness-design piece did not: don't surface remaining-token countdowns to the model, or add a reassurance.
• explicitestablishedContext resets — clearing the window entirely and starting a fresh agent with a structured handoff — address context anxiety in a way that compaction does not, because a fresh agent has no awareness of approaching limits.conf: high
  supports ←[FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero · p3 — Pocock independently converges on clearing context over compacting, matching Anthropic's finding that fresh-agent resets with handoffs beat compaction.
  supports ←Claude Code: Best practices for agentic coding · p12 — Recommending /clear plus a scratchpad handoff and context-quarantining subagents is a coarse precursor to the harness paper's structured context-reset-over-compaction finding.
• explicitestablishedTuning a standalone evaluator to be skeptical is more tractable than making a generator critical of its own work.conf: high
  supports ←20 Loop Design Patterns Every AI Engineer Should Know · p1 — Same separate-evaluator claim, corroborated by the harness paper's finding that tuning a standalone skeptical evaluator is more tractable than making a generator critical of its own work.
  refines ←Harness engineering for coding agent users · p8 — Böckeler qualifies evaluator optimism: even a tuned skeptical evaluator is an inferential sensor, and neither sensor class reliably catches the highest-impact failures (misdiagnosis, overengineering, misread instructions).
  supports ←Claude Code: Best practices for agentic coding · p15 — Separating a verifier with fresh context anticipates the harness paper's finding that a standalone skeptical evaluator is more tractable than making a generator critical of its own work.
  supports ←Netflix Open-Sources Agentic Workflow for Causal Inference · p3 — Netflix's separate actor and critic agents catching early-adopter bias and a failed placebo test is a concrete case of a standalone evaluator being more tractable than a self-critiquing generator.
  supports ←Project Glasswing: what Mythos showed us · p11 — Independent Cloudflare corroboration: a separate adversarial reviewer with a different prompt and model beats asking the original agent to check its own work.
  extends ←Project Glasswing: what Mythos showed us · p11 — Cloudflare extends the standalone-skeptical-evaluator claim by showing adversarial review works best with a different prompt AND different model, in production security triage.
• explicitdevelopedEven subjective qualities like aesthetic design can be made gradable by encoding design principles as explicit criteria that the model can grade against.conf: high
  refines ←Generative AI in the Real World: Aishwarya Naresh Reganti on Making AI Work in Production · p11 — Reganti conditions criteria-based LLM grading: only after rubrics are codified from sufficient human-annotated examples should LLM judges replace human judges.
• explicitThe wording of grading criteria steers model output independent of and prior to the evaluator-feedback loop — including stylistic convergence (e.g. 'museum quality' pushing toward a particular look).conf: high
• explicitdevelopedEvery component in a harness encodes an assumption about a current model weakness; those assumptions are model-version-dependent and go stale as models improve.conf: high
  extends ←Agent Harness Engineering · p5 — Adds the reverse coupling: not only do harness components encode model-version assumptions that go stale, models are post-trained against specific harnesses and overfit to their tool implementations.
• explicitestablishedWhen a stronger model lands, the right methodology is to remove harness components one at a time, measure the impact on output, and keep only the load-bearing pieces.conf: high
  supports ←The Anatomy of an Agent Harness · p11 — Anthropic p7: when a stronger model lands, remove harness components one at a time, measure impact, and keep only the load-bearing pieces. Anatomy generalizes the same prescription via the scaffolding metaphor and future-proofing test (complexity should shrink as models improve; Manus rebuilt 5x removing complexity).
  supports ←Prompting Claude Fable 5 · p2 — Both prescribe stripping harness scaffolding as model capability rises. The Fable 5 docs make it official Anthropic guidance ('capability improvements are a good prompt to re-evaluate which instructions, tools, and guardrails are still needed'), matching the harness-design methodology of removing components one at a time and keeping only the load-bearing pieces.
• explicitcorroboratedThe evaluator is worth its cost when the task sits beyond what the current model does reliably solo; on tasks within the model's solo capability, the evaluator becomes unnecessary overhead.conf: high
  supports ←Don't train the model, evolve the harness · p7 — Anthropic's harness paper: the evaluator (and by extension harness) is worth its cost only when the task sits beyond what the model does reliably solo. Akshay generalizes and corroborates: harness gains flatten exactly at the boundary where residual misses become real capability gaps and the model must carry the work.
• explicitPlanner agents should specify product context and high-level technical design but NOT granular technical details, because spec errors cascade downstream.conf: high
• explicitcontestedInter-agent communication via files (where one agent writes, another reads and responds) is sufficient orchestration for a three-agent harness.conf: high
  contradicts ←The Production Gap: 5 Patterns for Building Long-Running AI Agents · p22 — Turing Post argues natural-language orchestration is not durable because LLMs shortcut it, against Anthropic's claim that file-based inter-agent messages are sufficient orchestration (albeit at three-agent vs fleet scale).
• explicitThe space of interesting harness combinations does not shrink as models improve — it moves outward to more complex problems.conf: high
• explicitOut-of-the-box Claude is a poor QA agent — it identifies real issues then talks itself into approving the work, and tests superficially rather than probing edge cases.conf: high
• implicitThe harness-vs-solo quality differential justifies a 20x cost multiplier ($200 vs $9, 6h vs 20min) when the task sits beyond solo capability.conf: medium
• implicitThe right cadence for re-examining harness design is at each model release.conf: high
• explicitdevelopedHuman review of model traces on realistic problems remains essential to harness tuning — there is no shortcut around reading logs to find where the agent's judgment diverges from yours.conf: high
  refines ←How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code) · p10 — Wu adds model self-introspection on mistakes as a harness-debugging technique alongside the Labs post's human trace-reading.
• explicitEven within a working evaluator-led harness, deeply nested features remain under-tested and small layout/interaction issues persist — verification has continuous headroom.conf: high
• explicitModalities beyond text/visual fundamentally bottleneck the evaluator pattern: Claude can't hear, so musical-quality QA falls back to numeric inspection rather than felt judgment.conf: high

Arguments

Narratives

• The Dutch art museum design: by iteration 9, Claude had produced a clean dark-themed museum landing page that the author found 'visually polished but largely in line with my expectations.' On iteration 10, the model scrapped the approach and reimagined the site as a 3D room with checkered floor in CSS perspective, artwork on walls, doorway-based navigation. The author calls it 'the kind of creative leap that I hadn't seen before from a single-pass generation.'
  → Implicitly argues that the generator-evaluator iteration loop is not merely a quality-floor mechanism but unlocks emergent creative outputs the model would not produce in single-pass. It also implicitly supports the proposition that iteration count matters — quality is non-monotonic and the 10th iteration was qualitatively different from the 9th.
• Retro game maker, solo vs. harness: a one-sentence prompt produced (solo, $9, 20 min) an app with broken entity wiring and a non-functional play mode; the same prompt produced (full harness, $200, 6 hr) a 16-feature spec across 10 sprints, with a working play mode, richer editors, and an AI-assisted level generator. The author notes that an unjumpable wall in the level revealed the agent's lack of common-sense game-design.
  → Carries the core empirical argument that harness lift is real and justifies 20x cost on Opus 4.5. Also implicitly carries a quieter argument: that the gap between 'looks impressive' and 'actually works when you try to use it' is the killer issue, and that Playwright-driven evaluators close it.
• DAW build on simplified V2 harness with Opus 4.6: 3h 50m, $124.70, three rounds of build-then-QA. Round 1 produced a beautiful-but-shallow app (timeline clips not draggable, no instrument UI panels, no EQ curves); QA caught it; Round 2 still left audio recording as stub-only and effects as numeric sliders rather than graphical EQ curves.
  → Implicitly argues that even with a simpler harness and a more capable model, the generator-without-evaluator pattern would have shipped a non-functional 'display-only' app. The evaluator's value persists at the capability frontier.
• Specific evaluator findings: 'Rectangle fill tool allows click-drag to fill a rectangular area' → 'FAIL — Tool only places tiles at drag start/end points instead of filling the region. fillRectangle function exists but isn't triggered properly on mouseUp.' / 'Delete key handler at LevelEditor.tsx:892 requires both selection and selectedEntityId...' / 'PUT /frames/reorder route defined after /{frame_id} routes. FastAPI matches reorder as a frame_id integer.'
  → Carries the implicit claim that LLM evaluators can produce file-and-line-number-precise QA findings — not vague 'the layout feels off' critique. This is doing real argumentative work because it answers an implicit skeptic ('LLM QA can't possibly produce actionable bug reports').
• Evaluator tuning loop: author repeatedly reads QA logs, finds where the evaluator's judgment diverges from his own, and updates the evaluator's prompt. Multiple rounds. Even then, the evaluator misses deeply nested features and subtle interaction issues.
  → Implicitly supports the claim that harness design is a craft requiring sustained human judgment — there is no auto-generated evaluator. Quietly undercuts vendor narratives of 'just plug in two agents and they bootstrap each other.'
• DAW QA can't evaluate musical taste because Claude can't hear: the agent could set tempo and key, lay down melody and drums, adjust mixer levels, add reverb — all via tools — but the author cannot use the QA loop to evaluate whether the music sounds good.
  → Implicitly carries a modal-capability boundary argument: the evaluator pattern is bounded by the evaluator's perceptual modality. Where the model lacks a sense (hearing, embodiment, etc.), the iteration loop degrades.

Questions addressed

• ✅ How do we get Claude to produce high-quality frontend designs that aren't bland defaults?
  Generator + evaluator with explicit weighted grading criteria that explicitly penalize 'AI slop' patterns and reward originality; 5-15 iteration loops; few-shot calibrated evaluator with Playwright MCP for live navigation.
• ✅ How do we get Claude to build complete full-stack applications without human intervention over multi-hour sessions?
  Three-agent harness (planner expands prompt to spec; generator works in sprints or continuously; evaluator drives the running app via Playwright with hard-threshold criteria); on weaker models, add sprint contracts and context resets; on stronger models, strip those away.
• ✅ How do we know when a harness component is still earning its place vs. has become obsolete?
  Methodical one-at-a-time removal, measuring impact on final output quality. Treat every component as a hypothesis about a current model weakness.
• ✅ When is the multi-agent harness worth its cost vs. just running the model solo?
  Worth the cost when the task sits beyond what the current model handles reliably solo; otherwise the evaluator is 'unnecessary overhead.'
• ⭕ How do we calibrate where 'beyond solo capability' lies for an arbitrary task?
  Not addressed — the author's own calibration is implicit and experience-based ('our builds were at the edge of what the generator could do well solo').
• ⭕ How do we manage cost, audit, observability, and failure containment for long-running autonomous coding in regulated or production contexts?
  Not addressed. The piece is a Labs / case-study post; no engineering treatment of cost ceilings, audit trails, sandboxing, secret management, IP provenance, or rollback.
• ⭕ How should harnesses be designed when the generator and evaluator share the same base model's biases?
  Acknowledged ('the evaluator is still an LLM that is inclined to be generous towards LLM-generated outputs') but not solved — the author relies on prompt-engineered skepticism and few-shot calibration without a structural answer.
• ⭕ What happens when the next model introduces new failure modes rather than just resolving old ones?
  Not addressed — the simplification narrative assumes monotonic capability improvement.

Gaps

• incompleteNo engagement with cost ceilings or economic boundaries. $124-$200 per multi-hour autonomous run is presented as routine; there is no discussion of when this stops scaling (e.g. teams running this 100x/day, regulated environments with per-call cost controls, or comparison to a human contractor at equivalent output).
  Read next: Read pieces that treat agent-cost economics directly — Simon Willison's pricing analyses, the smallcode argument for smaller models, any IT-procurement perspective. Also: Anthropic's own enterprise pricing docs as the implicit deployment context.
• incompleteNo treatment of audit, observability, governance, or failure containment for long-running autonomous coding. The piece's implicit deployment is 'a single engineer running experiments at Anthropic Labs.' A Norwegian public-sector or regulated context would need: persistent audit trails of each agent's actions and tool calls, rollback semantics for partial builds, secret management when agents access live services, IP provenance of generated code, and human-checkpoint policies. File-based agent communication is actually well-suited to audit — but the author never makes that case.
  Read next: Read the EU AI Act's high-risk-system requirements, NIST AI RMF agent guidance if it exists yet, and any sober writing on agent observability stacks. Also: cross-reference Anthropic's enterprise-tier compliance docs.
• incompleteThe 'simplify when models improve' methodology assumes monotonic capability advance. It does not address the (empirically real) case where a new model has different — not strictly better — failure modes. E.g. Sonnet 4.5's context anxiety vs. Opus 4.6's regression on some niche tasks. The methodology is silent on what to do when component removal *reveals* a new weakness rather than confirming an old strength.
  Read next: Look for empirical model-version-comparison work — Aider/SWE-bench longitudinal results, internal Anthropic evals if available, and practitioner accounts of regressions (e.g. Simon Willison's lab notebooks).
• incompleteSelf-loop critique: a same-base-model evaluator shares the generator's biases. The author concedes this once and moves on, but it is a deep structural problem. What's the right intervention — a different model family as evaluator? A symbolic / non-LLM checker? Adversarial-prompt mining? The piece declines to engage.
  Read next: Read on adversarial / red-team LLM-as-judge literature, also on cross-family evaluator setups (e.g. Claude-grades-GPT or vice versa), and the formal-verification-meets-LLM thread.
  closed byProject Glasswing: what Mythos showed us — Answers the same-base-model evaluator-bias question with a working intervention: Cloudflare's adversarial review uses a second finding-suppressing agent with a different prompt and a different model (p11).
• incompleteThe piece collapses 'long-running' to 'multi-hour build of a single demo app from a one-line prompt.' It does not address the other kind of long-running: agents maintaining state and intent across hours/days while interleaving with humans, other systems, or external events. Goal drift over time is not discussed; only context drift within a single session is.
  Read next: Read on goal stability / instruction-following over long horizons, agent persistence (Cognition Devin postmortems, Anthropic's own memory work), and the agent-as-coworker thread.
  closed byThe Production Gap: 5 Patterns for Building Long-Running AI Agents — Addresses the other kind of long-running the gap flags: multi-day agents interleaving with humans and events, via checkpointed server-process patterns, live-state HITL pauses, distinct memory stores, and naming behavioral/memory drift as a production failure mode (p3, p6, p8, p9, p20).
• incompleteVendor-positioning shapes which problems get named. Every named failure mode (context anxiety, self-eval leniency) is one that future Anthropic models will improve on; less attention to: hallucinated APIs, supply-chain risks of generated code, on-prem deployment, smaller-model substitution, fallback when the vendor is down or rate-limits, multi-vendor portability. The implicit message is 'wait for the next Anthropic model and your harness gets thinner' — a pleasant story for Anthropic.
  Read next: Cross-read with smallcode-AI-coding-agent-small-llms (already in this repo) and any practitioner piece on multi-provider agent fallback. Read Böckeler's framing for a vendor-neutral conceptual model.
  closed byGitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. — Supplies the smaller-model-substitution and vendor-independence perspective the Anthropic piece omits: a local-only harness for 7B-20B models with cloud escalation only as fallback (p2, p10, p11).
• incompleteThe criteria-as-stylistic-steering observation ('museum quality' produces visual convergence) is surfaced and then dropped. It is treated as a quirky finding rather than a serious concern about AI-generated homogenization. At scale, if many teams use Anthropic-suggested criteria, the result is a deep convergence in the look of AI-generated software.
  Read next: Read on cultural homogenization risk in generative AI (Naomi Klein, Ted Chiang, and design-criticism writing). Also: the 'AI slop' discourse and any empirical work on generated-design diversity.
• incompleteThe 'long-running' threshold is operationalized only through examples (3.8 to 6 hours) — never defined. A reader cannot tell whether the prescriptions apply to a 30-minute task, a multi-day task, or only to the specific case study durations. Böckeler's harness theory and the smallcode piece operate at different time scales without acknowledgment.
  Read next: Cross-reference extract-bockeler-harness-engineering.json (theoretical-control framing — variety reduction, feedforward, feedback sensors) which is time-scale-agnostic; the smallcode piece's smaller-task framing; and reach for any time-scale-explicit treatment of agent autonomy windows.
• uninformedThe piece is silent on the existence of a broader harness-engineering theoretical literature (Böckeler's adaptive-control framing, the smallcode engineering-instinct view) — even though Anthropic Labs is presumably aware. As a result, named phenomena are coined fresh ('context anxiety') without acknowledging or engaging with adjacent theory. This is not necessarily a mistake — it may reflect the piece's narrow Labs-experiment scope — but it does mean the reader has to do the cross-mapping work themselves.
  Read next: Maintain the syntopical map (already in this repo as syntopical-harness-engineering-for-ai-coding-agents.json) and treat this article as one empirical data point, not a theoretical contribution.

Related documents

• Anthropic's Designs Three-Agent Harness Supports Long-Running Full-Stack AI Development
  Agent Harness Context Rot & Context Limits Long-Running & Managed Agents Multi-Agent Orchestration
• Prompting Claude Fable 5
  Agent Harness Capability Provisioning (right-sizing to the task) Long-Running & Managed Agents Multi-Agent Orchestration
• Agent Harness Engineering
  Agent Harness Capability Provisioning (right-sizing to the task) Context Rot & Context Limits Multi-Agent Orchestration
• [FULL WORKSHOP] AI Coding For Real Engineers - Matt Pocock, AI Hero
  Agent Planning Artifacts Capability Provisioning (right-sizing to the task) Context Rot & Context Limits Multi-Agent Orchestration
• The Anatomy of an Agent Harness
  Agent Harness Context Rot & Context Limits Evals Multi-Agent Orchestration
• Don't train the model, evolve the harness
  Agent Harness Evals Multi-Agent Orchestration Riding Model Improvements
• No Vibes Allowed: Solving Hard Problems in Complex Codebases – Dex Horthy, HumanLayer
  Agent Harness AI Slop Context Rot & Context Limits Multi-Agent Orchestration
• How to Actually Prompt Fable 5, Straight From the Team That Built It
  Agent Harness Long-Running & Managed Agents Multi-Agent Orchestration Riding Model Improvements
• Accelerating Delivery: How AI Agents Can Own the Initial Code Draft
  Agent Harness Context Rot & Context Limits Multi-Agent Orchestration
• How Anthropic's product team moves faster than anyone else | Cat Wu (Head of Product, Claude Code)
  Agent Harness Evals Riding Model Improvements