This Week in AI: Rethinking the Agent Harness

Michelle Smith · rss · 2026-05-22 source ↗ Reader 🖥️

practical · mixed — Mixed news roundup and argumentative editorial. The first two sections are expository news summary; the third (the 'harness' section) is the load-bearing argumentative section and is practical — it argues that practitioners should build their own agent harnesses with skills rather than wrap a model or buy a productized agent.

Structure

Key terms

• agent harness Agent Harness
  The scaffolding around a model — tools, skills, control loops, environment integrations — that determines what the agent can actually do, exemplified by Claude Code. Treated as a layer distinct from both the model and the bespoke application.
• age of the unharnessed agent
  Berryman's label for the current moment: agents are accessible to everyone (not just developers), and the relevant practice has moved from building bespoke agents to building general agent harnesses populated with skills.
• skills Agent Skills
  Natural-language descriptions of how to do a task, given to a general-purpose agent harness; readable and modifiable by domain experts without developer intermediation.
• open agent protocol Agent-to-Agent Protocols
  Berryman's term for a not-yet-existing standard whereby an agent moving between environments (e.g., Obsidian to Wikipedia) is handed environment-specific skills by each environment.
• harness era / harness phase Agent Harness
  The latest stage in an evolutionary arc (document completion → chat loops → tool-calling agents → DAG workflows → harnesses) characterized by general-purpose, skill-driven agents.
• Mythos Mythos
  Anthropic's internal frontier model that surfaced thousands of unknown vulnerabilities; treated here as evidence that capability has crossed a threshold significant enough to move policy.
• Project Glasswing Project Glasswing
  Anthropic's restricted-access program for the Mythos model, distributed to trusted defensive partners rather than publicly released.

Propositions

• explicitThe harness you build around a model now matters more than which model you pick.conf: high
  supportsAgent Harness Engineering · p1 — Independently asserts harness choice now matters more than model choice, backed by Stanford results on bare-model vs harnessed performance gaps.
• explicitcorroboratedAI product development has evolved through a sequence of phases: document completion → chat loops → tool-calling agents → DAG-based workflows → the harness era.conf: high
  supports ←Podcast: Strands Agents with Clare Liguori · p1 — Clare's first-person account of Strands moving away from workflow-heavy design is a concrete case of the 'DAG-based workflows -> harness era' phase transition this proposition asserts at the industry level.
• explicitEach successive phase of AI product development added capability but also complexity, and each generated a new class of reliability and control problems.conf: high
• explicitA step change in model capability in late 2026 (November-December of 'last year' from the article's perspective, i.e., late 2025) enabled the harness phase.conf: high
• explicitcorroboratedReplacing a bespoke application with a skills-driven agent harness lets domain experts read and modify agent behavior in plain English, without developer intermediation.conf: high
  supports ←Zero to Agent in 30 Minutes: Build a Workflow Agent with John Berryman · p3 — Independent convergence within the same author's O'Reilly series: both credit natural-language agent logic with letting domain experts read and modify behavior directly, without a developer/engineer intermediary.
• explicitSkills-driven harnesses give domain experts a kind of control that wrapped commercial agent products do not.conf: high
• explicitStanford research shows performance gaps between a bare model and a well-designed harness now often matter more than which underlying model is used.conf: medium
  supportsExternalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering · p1 — Stanford evidence that harness design gaps outweigh model choice backs the claim that agent capability is now delivered by reorganizing the runtime, not changing weights.
• explicitdevelopedFrontier model capability now moves U.S. policy in weeks, not years (six weeks from light-touch policy to FDA-style review proposals after Mythos).conf: high
  extends ←How the 4 New AI Models Change How You Work · p1 — The government-imposed release pause now ending, with a backlog arriving all at once, is the downstream operating state of the FDA-style pre-release review regime that oreilly-rethinking-agent-harness observed emerging in weeks after Mythos — policy that moved fast enough to gate releases and create a backlog.
• explicitHyperscaler compute commitments (Colossus 1, 3.5 GW Google/Broadcom, 9 GW Stratos) presuppose that model capability will keep scaling.conf: high
• explicitWhether the assumption of continued capability scaling holds will determine what is economically viable to build in the next decade.conf: high
• explicitThere will eventually be no internet for humans.conf: medium
• explicitA future 'open agent protocol' will let agents move between environments and receive environment-specific skills from each environment.conf: medium
• implicitReadability-by-domain-experts is the right measure of a good harness.conf: high
• implicitcorroboratedA skill written in plain English is a kind of executable code.conf: high
  extendsAgent Skills: Google-Backed AI Coding Agent Workflow · p4 — Takes markdown-instruction sufficiency further: a plain-English skill is a kind of executable code that domain experts can read and modify directly.
  supports ←Zero to Agent in 30 Minutes: Build a Workflow Agent with John Berryman · p8 — Berryman's step 5 (write skills as natural language instead of coding a state machine) is a concrete building instance of the claim that a plain-English skill is itself a kind of executable code.
• implicitHarness design is a layer with its own engineering discipline, separable from model R&D and from application development.conf: high
• implicitThe previously dominant evaluation criterion — which model scores highest on benchmarks — was a buying decision (not e.g. a research or safety decision).conf: high
• implicitThe right unit of comparison is bare model vs. well-designed harness (not bare model vs. bare model, or harness vs. harness).conf: medium
• implicitcontestedFrontier capability that surfaces real defensive value (Mythos finding vulnerabilities) justifies restricted access rather than open release.conf: medium
  contradicts16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden · p15 — Claims restricted access is justified for defensively valuable frontier capability, against Strumke's claim that vendor withholding is not a security mechanism at all.
• explicitBuilding your own agent is now within reach of everyone, not just software developers.conf: high

Arguments

Narratives

• Berryman's client engagement: he replaced a bespoke application with an agent harness plus skills derived from interviews with the client's domain experts; the experts can now read the agent's behavior in plain English and potentially fix issues themselves.
  → The single load-bearing piece of evidence for the unity. It is meant to show — without arguing abstractly — that the skills-driven harness pattern produces a qualitatively different relationship between domain experts and software.
• The 27-year-old OpenBSD bug surfaced by Mythos, alongside thousands of other vulnerabilities across OSes, browsers, and financial infrastructure.
  → Anchors the claim that frontier model capability has crossed a threshold material enough to force a policy response; the age of the bug makes the model's capability feel categorically new.
• Box Elder County, Utah approving the Stratos data center — 40,000 acres, 9 GW, twice the size of Manhattan, equivalent to nine nuclear reactors, backed by Kevin O'Leary, approved over local protests.
  → Sensationalizes the scale of compute commitments; implicitly argues that the industry is acting as if capability scaling is far from done.
• Berryman's demo of his personal agent moving from an Obsidian notebook into Wikipedia and back, carrying context across environments.
  → Concretizes the 'open agent protocol' idea — agent receives skills from each environment — and gives the otherwise speculative protocol a visible referent.
• Berryman's evolutionary arc: document completion → chat loops → tool-calling agents → DAG-based workflows → harness era.
  → Periodization narrative; positions the harness phase as the natural successor rather than one option among many, lending historical inevitability to the unity.

Questions addressed

• ✅ What is the most important shift in AI engineering practice right now?
  The shift from model selection to harness design, and within harness design, to skill-authoring by domain experts.
• ✅ Should organizations build their own agents or buy wrapped products?
  Build — specifically a general harness populated with skills — to give domain experts direct readability and control.
• ✅ Is frontier-model capability moving fast enough to require new policy mechanisms?
  Yes — six weeks from Mythos disclosure to FDA-style review proposals is taken as evidence.
• ⭕ Will model capability keep scaling enough to justify multi-gigawatt buildouts?
  Explicitly flagged as the open question whose answer determines what's economically viable.
• ⭕ What does 'rethinking the agent harness' actually consist of — which prior assumptions are being overturned, and which new ones replace them?
  Not directly answered. The article asserts that harnesses now matter more than models and that skills are the mechanism, but does not enumerate which prior harness assumptions are being discarded (e.g., DAGs, tool-calling orchestration, deterministic control flow) or articulate new design principles to replace them.
• ⭕ Under what conditions does a skill-driven harness fail?
  Not addressed. The Berryman anecdote is presented as 'worked perfectly' with no counter-cases, failure modes, or boundary conditions.
• ⭕ How is harness quality evaluated — what replaces the dethroned 'highest benchmark score' criterion?
  Hinted at (expert readability, control) but not operationalized; no benchmarks, no evaluation framework, no failure modes are proposed.
• ⭕ Is the 'harness era' compatible with the hyperscaler buildout thesis, or are they in tension?
  Both narratives are presented in adjacent sections without reconciliation. The article does not notice that the harness thesis (value moves up the stack) is in mild tension with the buildout thesis (the bet is on capability still scaling at the model layer).

Gaps

• incompleteThe 'rethinking' is asserted rather than performed. The article never enumerates the prior harness assumptions being overturned (e.g., DAG-based orchestration, deterministic tool dispatch, code-defined skills, developer-mediated change) nor articulates the new principles replacing them beyond 'skills written in English.' For a piece titled 'Rethinking the Agent Harness,' the actual rethink is one anecdote and one neologism.
  Read next: Berryman & Ziegler, 'Prompt Engineering for LLMs' (the cited O'Reilly book); Anthropic's own Claude Code design docs; the Stanford paper at arXiv:2603.28052; Birgitta Böckeler's 'harness engineering' piece for an attempt to actually decompose the harness layer.
• incompleteSingle-case generalization. The unity ('build your own harness') rests almost entirely on one consulting client where it 'worked perfectly.' No failure modes, no boundary conditions, no comparison to cases where a bespoke app or a wrapped product would still be the right choice.
  Read next: Post-mortems of agent deployments at scale; the Cognition/Devin retrospectives; harness-engineering writeups from teams shipping to production rather than consultants.
• incompleteNo engagement with the assumption that natural-language skills are a stable substrate. Treating English prose as code elides the prompt-brittleness, drift, and version-control problems that practitioners shipping Claude Code-style systems already report. The Berryman quote slides past this in a half-sentence.
  Read next: Empirical work on prompt brittleness across model versions; writing on 'skills as code' from teams maintaining them in production (e.g., harness-engineering threads); the user's existing extractions on skills as protocol vs. skills as prose.
  closed byThe Problem Is Prompt Debt — Breunig directly substantiates that gap's concern — that treating natural-language prose as a stable substrate elides real brittleness/drift problems — by naming the phenomenon ('prompt debt'), explaining its causal mechanism (imprecise language against a probabilistic model, plus the need to repeatedly 'fight the weights'), and documenting concrete brittleness examples (Fable's system prompt repeating one copyright rule six times; Claude Code repeating a tool-call instruction seven times).
• incompleteThe relationship between the three sections is unargued. Mythos, the compute buildout, and the harness thesis are juxtaposed without the article noticing that they imply different theories of where AI value is moving — model layer (compute), capability frontier (Mythos), or harness layer (Berryman). The implicit reconciliation would be the most interesting move and isn't made.
  Read next: Articles framing 'value migration in the AI stack' explicitly; investor letters or analyst notes that try to resolve which layer captures economic returns.
• incompleteThe 'open agent protocol' is named without being specified. No design sketch, no examples of what an environment-supplied skill looks like, no comparison to existing proposals (MCP, A2A, agent-to-agent protocols). The reader is left with a direction but no testable proposal.
  Read next: Anthropic's Model Context Protocol spec; agent interoperability proposals from Google, OpenAI, and the LangChain ecosystem; the future-cast section in the next Berryman appearance.
• uninformedSchneier's challenge to Mythos's uniqueness is mentioned and then immediately set aside ('either way, it's a problem that's coming'). The article does not engage with the substantive question of whether the policy response is calibrated to actual differential capability or to perceived capability — a question whose answer matters for whether FDA-style review is wise.
  Read next: Schneier's full post on Mythos; CAISI's framing; counter-arguments from researchers who replicated Mythos-like findings on cheaper public models.
• illogicalMild equivocation on 'harness.' Claude Code (a specific product), 'the agent harness' (a layer), and Berryman's bespoke consulting build (a specific instance) are used interchangeably. A reader could leave thinking 'use Claude Code' and 'build your own harness' are the same recommendation, when the article's argument actually requires the latter and only points at the former as an example.
  Read next: Explicit harness taxonomy work — what distinguishes a product, a framework, and a bespoke harness — e.g., Böckeler's 'harness engineering' piece.

Closes gaps elsewhere

• Agent Skills: Google-Backed AI Coding Agent Workflow · g1 — Cites Stanford research empirically comparing bare models against well-designed harnesses, the independent evaluation of structured agent instruction frameworks the gap asked for.

Related documents

• 16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden
• Agent Skills: Google-Backed AI Coding Agent Workflow
• Agent Harness Engineering
• Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
  Agent Harness Agent Skills Agent-to-Agent Protocols
• Meta's new model is Muse Spark, and meta.ai chat has interesting new tools
  Agent Harness Mythos Project Glasswing
• Zero to Agent in 30 Minutes: Build a Workflow Agent with John Berryman
  Agent Harness Agent Skills Non-Engineers Building
• Project Glasswing: what Mythos showed us
  Agent Harness Mythos Project Glasswing
• Why cultivating agency matters more than cultivating skills in the AI era | Max Schoening (Head of Product, Notion)
  Non-Engineers Building Riding Model Improvements
• Agent Skills
  Agent Harness Agent Skills
• AI Sovereignty: Bargaining with Big Tech and the Promise of Full Stack Open Source AI
  Agent Harness Agent-to-Agent Protocols