Don't train the model, evolve the harness
Akshay · tweet · 2026-04-06 Reader 🖥️
Agent HarnessAgentic LoopContext EngineeringEvalsMulti-Agent OrchestrationRecursive Self-ImprovementRiding Model Improvements
For a model you don't control — a frozen, rented commodity whose weights you can't change — the reliable leverage on agent performance lives in the harness (the code that feeds the model context, runs its tool calls, and decides when a run ends), not in the weights: a benchmark score conflates model and harness, so most apparent 'model failures' are harness failures (often mundane output-plumbing, not reasoning), and the disciplined way to capture that leverage is an automated fitness-selection loop that adds one mechanism at a time and keeps it only when it clearly beats the current best — a process that compounds until the harness runs out of tricks and the residual misses are genuine capability gaps.
practical · argumentative thread (with an appended expository deep-dive) — The thread proper is argumentative: a thesis ('don't train, evolve the harness') defended by one worked example and three ablation-style findings. The quoted article is expository/taxonomic — it defines the harness, enumerates 12 components and 7 design decisions, and asserts rather than argues. Prescriptive intent throughout (what to invest in, where to look when an agent fails), so practical.
Structure
| Part | Role |
|---|---|
| The thesis and the worked case | States 'don't train the model, evolve the harness' and grounds it in the Hugging Face experiment: a frozen open model at 0% on a hard legal-agent benchmark, driven to Sonnet-4.6 parity at ~7x lower cost per task by an automated loop that rewrote only the code around it. |
| Diagnosis — what the 0% was really measuring | Shows the model did the legal analysis correctly but misfiled the deliverable (wrong name, scratch folder, or never written); the judge only grades files saved in the exact place under the exact name. So the 0% measured the harness, not legal reasoning. |
| The mechanism — automate the harness search | Hand-tuning the harness is slow and model-specific, so they automated it: a Claude proposer adds exactly one mechanism per iteration; an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound. |
| Three findings about where agents actually fail | Biggest single gain was file handling, not intelligence (zero extra tokens); code fixes transferred across models but prompt playbooks did not (a smaller same-family model gained 14 points; tuned prompts hurt a different family); harness choice dominated — five harnesses spanned 3.5% to 80.1% on the same model/judge/tasks. |
| The ceiling and the lesson | Gains eventually flatten; the residual misses look like real capability gaps and the model must carry the work. Lesson: until the harness is fixed, it is impossible to know whether the model or the harness failed. |
| Appended deep-dive: The Anatomy of an Agent Harness | Defines the harness (von Neumann analogy, three levels of engineering), enumerates 12 production components and 7 design decisions, and argues via the scaffolding metaphor that harness complexity should DECREASE as models improve (Manus rebuilt 5x removing complexity; co-evolution principle; the future-proofing test). |
Key terms
- harness Agent Harness
The complete non-model software wrapping the LLM — the runtime that feeds it context, runs its tool calls, and decides when a run ends; concretely the 12 components (orchestration loop, tools, memory, context management, prompt construction, output parsing, state, error handling, guardrails, verification loops, subagent orchestration). 'If you're not the model, you're the harness.' - evolve (the harness)
Improve the harness by an automated fitness-selection loop that ADDS and REMOVES mechanisms to fit the current model — not monotonic growth. The quoted deep-dive is explicit that complexity should shrink as models absorb capabilities. Opposed to 'train'/'fine-tune' (change the model's weights). - the automated harness loop (proposer + outer loop)
A Claude 'proposer' adds exactly one mechanism per iteration; an outer loop accepts it only if it clearly beats the current best. Accepted mechanisms therefore compound. This is the concrete method behind 'evolve' — recursive self-improvement of the code around a fixed model. - benchmark-score conflation
A benchmark score measures the model AND its harness together; until the harness is fixed you cannot attribute a failure to either. The 0% legal-agent score measured file-plumbing, not reasoning. - harness thickness / scaffolding
How much logic lives in the harness versus the model. Scaffolding is temporary construction infrastructure — removed when the building is complete; as models improve, harness complexity should decrease (Anthropic deletes planning steps as models internalize them).
Propositions
- explicitFor a frozen model with zero weight changes, rewriting only the harness can match a frontier model's headline benchmark metric at roughly 7x lower cost per task — the leverage sits in the code around the model, not in the weights.conf: highsupportsAgent Harness Engineering · p1 — Concrete instance of 'a decent model with a great harness beats a great model with a bad harness': a frozen open model reached frontier parity at ~7x lower cost purely via harness rewrite.
- explicitcorroboratedA benchmark score measures the model and its harness together; until the harness is fixed it is impossible to know which one failed.conf: highsupports ←Downstream #1: Where Do Our Tokens Go? · p6 — Both converge on treating model and harness as a coupled unit of evaluation rather than judging a model in isolation.
- explicitMany apparent 'model failures' are mundane harness failures: the single biggest gain came from file handling — an automatic step that lands the deliverable exactly where the judge expects — not from intelligence, and with zero extra model tokens.conf: highsupportsAgent Harness Engineering · p3 — 'Most agent failures are configuration problems, not model problems' — directly evidenced by the biggest single gain being file handling (landing the deliverable where the judge expects), not intelligence, with zero extra model tokens.
- explicitcorroboratedHarness improvement can be automated as a selection loop: a Claude proposer adds exactly one mechanism per iteration and an outer loop keeps it only if it clearly beats the current best, so accepted mechanisms compound.conf: highsupports20 Loop Design Patterns Every AI Engineer Should Know · p20 — The proposer + keep-if-better outer loop is a concrete production instance of the Workflow-Optimization pattern (the system measures quality/cost and modifies its own workflow), and of Prompt/Config Optimization (p19) — supplying the real numbers that thread lacked.supports ←The Anatomy of an Agent Harness · p14 — CENTERPIECE. The sibling's strategy automates harness improvement with a proposer that 'adds exactly one mechanism per iteration' and an outer loop that keeps it only if it beats the current best. This anatomy supplies WHAT those mechanisms are: the named components and seven decision axes are the labeled search space the fitness-selection loop searches over. Anatomy = the what; strategy = the search. The sibling's note says this deep-dive was written EARLIER, so it is the prior/foundational piece the strategy stands on — hence 'supports' (not refines/extends, which would imply it postdates and improves the strategy). The load-bearing hinge is p14's separability premise: one-mechanism-at-a-time selection is only valid if the components are actually decoupled (see g1/g3).
- explicitdevelopedCode-layer harness fixes transfer across models, but prompt playbooks do not — the same harness lifted a smaller same-family model by 14 points, while the tuned prompts hurt a different model family on tasks it could already finish.conf: highrefinesAgent Harness Engineering · p5 — Osmani p5 says models get post-training coupled to the harness they were trained against (overfitting to tool implementations). Akshay refines the practical consequence: because of that coupling, structural CODE-layer fixes are model-portable while PROMPT tuning is model-specific and can hurt a different family — the asymmetry the coupling predicts, observed.extends ←The Self-Building Agent: A LangChain4j Experiment · p5 — Extends the harness-vs-model-transfer finding with a more extreme case: the identical multi-agent harness and task goes from complete failure (unbounded tool-calling loop) to full success purely by swapping the underlying model, with zero changes to the harness itself.
- explicitHarness choice dominates all other factors: with the same model, judge, and tasks, five different harnesses scored anywhere between 3.5% and 80.1%.conf: highsupportsAgent Harness Engineering · p2 — Cross-author corroboration (Osmani) with hard numbers: 'the gap is largely a harness gap' is exactly what five harnesses spanning 3.5%-80.1% on the same model/judge/tasks demonstrate.
- explicitHarness gains eventually flatten; the residual misses look like real capability gaps, and past that point the model itself must carry the work.conf: highsupportsHarness design for long-running application development · p8 — Anthropic's harness paper: the evaluator (and by extension harness) is worth its cost only when the task sits beyond what the model does reliably solo. Akshay generalizes and corroborates: harness gains flatten exactly at the boundary where residual misses become real capability gaps and the model must carry the work.
- implicitdevelopedThe model is a fixed, substitutable commodity you rent rather than an asset you improve, so leverage accrues to the layer you own and can iterate cheaply — the harness — because model gains are exogenous and arrive on someone else's schedule.conf: highextends ←Is Open-Source AI Really the Dangerous Path? · p6 — Both identify the harness (not the rented/commodity model) as where leverage accrues; this article generalizes that engineering-and-benchmark claim into a business-power and open-vs-closed-AI framing (who controls the harness, and to what end).
- implicitcorroborated'Evolve the harness' means running a fitness-selection process that adds AND removes mechanisms to fit the current model — not monotonically growing scaffolding; the same author's deep-dive holds that harness complexity should DECREASE as models improve (Manus rebuilt 5x removing complexity; the future-proofing test rewards performance that scales with stronger models WITHOUT added complexity).conf: highrefinesPrompting Claude Fable 5 · p2 — NOT a contradiction — the apparent opposition ('evolve/grow' vs 'prune') is lexical. Fable-5 p2 prescribes PRUNING on the MODEL-CAPABILITY axis: a stronger model absorbs compensatory hand-holding, so remove it. Akshay's thesis operates on the LOCUS-OF-LEVERAGE axis: for a FIXED, rented model, the harness is where the marginal gain lives. These are different axes, so no conflict. Akshay refines Fable-5's undifferentiated 'prune scaffolding' by specifying WHAT survives pruning: his finding that 'code fixes transfer across models, prompt playbooks do not' (p5) says keep the portable code-layer mechanisms and cut the model-specific prompt hand-holding. And Akshay's own quoted deep-dive endorses the prune stance outright (Manus rebuilt 5x removing complexity; the future-proofing test rewards NOT adding complexity), confirming 'evolve' = fitness-selection that adds AND removes.supports ←Stop Overengineering Your Agent Harness · p7 — The named repeated-rebuild examples (Manus 5x, LangChain Open Deep Research, Claude Code) are concrete corroborating instances of 'evolve the harness' meaning add-and-remove to fit the current model, not monotonic growth.
- implicitFor most teams the practical choice is harness-vs-nothing, not harness-vs-fine-tuning: because you cannot change a rented frontier model's weights, harness engineering is the only lever actually available, and the 'don't train' framing reframes where to look rather than adjudicating a live tradeoff.conf: medium
- explicitTwo products built on identical models can differ wildly in performance based solely on harness design; the harness (orchestration loop, tools, memory, context management, verification, guardrails) is the product and is not a commodity layer.conf: high
- explicitModels are increasingly post-trained with specific harnesses in the loop, so model and harness co-evolve; the tight coupling means changing tool implementations can degrade performance, and a sound harness is one whose performance scales up with more powerful models without added complexity.conf: high
Arguments
Evolve the harness rather than train the model. strength: high
Explicit premises:
- A frozen model plus a rewritten harness matched a frontier model at ~7x lower cost per task, with zero weights changed.
- Five harnesses on the same model/judge/tasks spanned 3.5% to 80.1% — the harness dominates outcome.
- The harness search can be automated by a cheap proposer + keep-if-better outer loop whose accepted mechanisms compound.
- Code-layer harness fixes transfer across models.
Implicit premises:
- You rent a frontier model whose weights you cannot change, so training is not actually available as a lever for most teams.
- Harness iteration is cheap, fast, and controllable, whereas training is expensive, slow, and gated by the model owner.
- The binding constraint for your task is the harness, not the model's capability ceiling — the load-bearing and undischarged assumption.
Inferences:
- Therefore spend marginal effort on the layer you own and can iterate cheaply (the harness) rather than on the model.
A benchmark score (even a 0%) does not tell you the model failed. strength: high
Explicit premises:
- The judge only grades files saved in the right place under the exact requested filename.
- The model did the legal analysis correctly but kept saving it under the wrong name, in a scratch folder, or not at all.
- Five harnesses on the same model produced scores from 3.5% to 80.1%.
Implicit premises:
- A benchmark score is a joint function of model capability and harness quality.
- Harness failure modes (output plumbing, tool orchestration) are systematically mistaken for reasoning failures.
Inferences:
- Therefore fix the harness before attributing any failure to the model; 'the next time your agent fails, look at the harness.'
Narratives
- Hugging Face took a frozen open model scoring 0% on a hard legal-agent benchmark, left its weights alone, and let an automated loop rewrite only the surrounding code. By the end the system essentially matched Sonnet 4.6 on the headline metric at ~7x lower cost per task, zero weights changed. The 0% turned out to be misfiling (correct analysis saved under the wrong name / in a scratch folder / never written); the biggest single gain was an automatic file-handling step, not intelligence; code fixes transferred across models while tuned prompts did not; and five harnesses on the same model spanned 3.5-80.1%.
→ The entire empirical spine of the thesis — one worked case carrying every generalization (leverage, portability, dominance, ceiling). - LangChain changed only the infrastructure wrapping their LLM (same model, same weights) and jumped from outside the top 30 to rank 5 on TerminalBench 2.0; a separate research project reached a 76.4% pass rate by having an LLM optimize the infrastructure itself, surpassing hand-designed systems.
→ A second, independent instance that the harness — not the model — drives benchmark rank; corroborates the thread's single case. - Manus was rebuilt five times in six months, each rewrite removing complexity: complex tool definitions became general shell execution; 'management agents' became simple structured handoffs.
→ Demonstrates that 'evolving' a harness includes shrinking it — the concrete basis for reconciling the title with 'prune your scaffolding as the model improves.'
Questions addressed
- ✅ When an agent underperforms on a benchmark, is it the model or the harness that failed?
Largely the harness, until harness gains flatten — and you cannot attribute failure to the model until the harness is fixed, because the score measures both together. - ✅ Can harness tuning be made systematic rather than slow, manual, and model-specific?
Yes — automate it with a proposer that adds one mechanism per iteration and an outer loop that keeps it only if it clearly beats the current best, so accepted mechanisms compound. - ✅ Do harness improvements port across models?
Partially: code-layer fixes transfer (a smaller same-family model gained 14 points), but prompt playbooks do not and can even hurt a different model family.
Gaps
- incompleteNo binding-constraint diagnostic. The whole thesis presupposes the harness — not the model's capability ceiling — is what limits your task, but the piece offers no way to tell in advance whether a given task is harness-bound or capability-bound; you only learn which after gains flatten. The headline example (file-plumbing, 0% -> parity) is a selected case where the harness was the entire story and does not generalize to reasoning-bound tasks. This is the Meadows leverage-point question left unanswered.
Read next: Read Boeckeler's adaptive-control harness framing and Anthropic's 'remove components one at a time, keep only the load-bearing' methodology, looking for a procedure that predicts, before investment, which constraint binds — e.g. a cheap probe that separates 'model got the answer but the wrapper lost it' from 'model can't do this at all.' - incompleteNo eval-construction guidance, yet the whole method depends on it. The automated proposer + keep-if-better loop presupposes a reliable task judge to select against — and the cited judge was a brittle exact-filename grader that itself created much of the 'failure.' The piece never says how to build the objective the loop optimizes, which is where the real difficulty (and the risk of optimizing to a bad metric) lives.
Read next: Read the evals literature and the extract-20-loop-design-patterns Score-and-Retry / Judge Ensemble / LLM-as-judge patterns for how to construct and calibrate the gradable objective a self-optimizing harness loop requires.closed byPresentation: From AI Agent Demo to Production: Automated Testing and Evaluation — That gap flags the self-optimizing harness-evolution loop's total dependence on a reliable task judge with no guidance on how to build one (its own judge was a brittle exact-filename grader). This talk supplies the missing eval-construction methodology: build the judge around simulation-generated scenarios with automatically verifiable, rule-checkable goals rather than a single brittle exact-match grader. - incomplete'Don't train' vs 'evolve the harness' is a soft strawman. Fine-tuning is never actually analyzed — no cost, no cases where training genuinely IS the right lever (domain shift, latency/cost at scale, proprietary capability the frontier lacks). For a rented frontier model the real contrast is harness-vs-nothing, so the title over-generalizes into 'never train.'
Read next: Read a fine-tuning cost/benefit source to bound when weight-changing is warranted, so the 'harness-first' heuristic has an explicit boundary rather than an absolute. - incompleteThe automated 'add one mechanism, keep if better' loop has no pruning or garbage-collection step, so accreted mechanisms can ratchet harness complexity upward — precisely the cost the quoted deep-dive warns against (scaffolding should shrink; tight model-coupling creates lock-in). The maintenance and portability cost of an ever-growing auto-generated harness is named in the abstract but never quantified or bounded.
Read next: Read on harness co-training overfitting and the 'delete components one at a time' methodology; look for how an automated harness search can be given a complexity penalty or removal operator so it thins rather than only accretes. - incompleteSecond-hand, single case, no first-party numbers. The entire empirical spine is one relayed Hugging Face experiment (7x cost, 14-point transfer, 3.5-80.1% spread) with no reproducible benchmark, dataset, task definition, or independent replication — thin ground for a rigorous adopter to act on.
Read next: Read the source Hugging Face post directly and seek independent harness-ablation studies before treating the 7x / 3.5-80.1% figures as generalizable.
Closes gaps elsewhere
- 20 Loop Design Patterns Every AI Engineer Should Know · g4 — Supplies the missing evidence for the self-optimization / workflow-optimization loop family specifically (p19/p20): a documented case where an automated 'add one mechanism, keep if it beats the current best' loop moved a frozen model from 0% to frontier parity at ~7x lower cost, with the honest boundary that gains compound then flatten. Evidences that pattern, not all 20.
Related documents
- The Anatomy of an Agent Harness
- 20 Loop Design Patterns Every AI Engineer Should Know
- AI Root Cause Analysis Shifts from Model Reasoning to Context Engineering
- Harness design for long-running application development
- How to Actually Prompt Fable 5, Straight From the Team That Built It
- Prompting Claude Fable 5
- Stop Overengineering Your Agent Harness
- Agent Harness Engineering
- Accelerating Delivery: How AI Agents Can Own the Initial Code Draft
- The Almedra Token-Efficiency Benchmark, Updated For GLM 5.2