Audit your Agent files

AddyOsmani.com · rss · 2026-08-27 source ↗ Reader 🖥️

Agent Config Decay Agent Harness Agent Memory Agent Skills Agent Context Files (CLAUDE.md / AGENTS.md)Hooks MCP (Model Context Protocol)Progressive Disclosure

Coding-agent configuration (CLAUDE.md/AGENTS.md, skills, hooks, MCP servers, memory) has a half-life because models and harnesses keep improving past what the rules were written for, so instead of accumulating it indefinitely or purging it wholesale, developers should audit it on a recurring cadence — using tools like /doctor, questioning whether each item's original reason still holds, and testing removal before committing to it.

practical · mixed — Argumentative-practical guide (you should audit your agent configuration on a cadence) built on cited empirical research and reinforced by personal-experience narrative and expository walkthroughs of tooling (/doctor vs claude doctor, skill loading budget).

Structure

Part	Role
Opening framing: skill/CLAUDE.md fatigue in developer discourse, and the tension between official 'delete periodically' guidance and the fear that deleting will silently degrade quality	context
Why agent configuration rots: the four-step ratchet cycle (error -> add rule -> file grows -> adherence drops), illustrated with the author's own CLAUDE.md failure modes	evidence / mechanism
Cited empirical research: the June 2026 100-repo config-smell study, the 288-run/17-task context-file study, the personalized-vs-generic-skill study, the source-vs-summary study, and Anthropic's 80%+ system-prompt cut	evidence
What the author found auditing his own setup with /doctor, including the forgotten 'anti-slop' skills and the 250-to-25 anecdote	narrative evidence
Clarifying what an installed skill actually costs (discovery budget vs. invocation-time body load) and distinguishing /doctor from claude doctor	conceptual clarification
Prescribed workflow: audit on a cadence, then test removal (archive, run /doctor and /memory, remove, test without, decide)	conclusion / action guide

Key terms

half-life (of agent configuration)
The rate at which a written instruction's value decays as the underlying model and harness improve past the state that justified writing it — the file itself doesn't shrink on its own, so staleness accumulates silently.
/doctor
The in-session Claude Code command that audits configuration: flags unused skills/MCP servers/plugins relative to context cost, slow hooks, duplicated CLAUDE.md guidance, and always-loaded content the agent could read from the repo instead. Reports findings before changing anything.
audit
Not deletion by default — the act of asking, for each existing rule/skill/hook, whether the original reason it was added is still true, then testing removal before deciding.
listing budget
The share of context window (defaulting to 1%) reserved for skill discovery — names and descriptions of all installed skills, always loaded; descriptions get shortened first when this budget is exceeded. Distinct from a skill's body, which loads only on invocation.
instruction value expires
The author's summary lesson from the cited research: a written rule's usefulness is time-bound, not permanent, so files should be archived-and-reconsidered rather than only ever grown; hard constraints should move into tests/hooks/permissions rather than staying as prose.

Propositions

explicitCoding-agent configuration has a half-life: as models improve and harnesses add capabilities, instructions written for an older state stay behind and lose value, but the file doesn't shrink on its own.conf: high
extendsYour AGENTS.md might be a big problem · p1 — Adds the temporal/model-improvement dimension to why bloat degrades performance: it's not just size, it's that instructions written for an older model/harness state go stale as the underlying system improves.
explicitestablishedAGENTS.md/CLAUDE.md bloat forms through a self-reinforcing four-step cycle: the agent errs once in a way that stings, a rule is added (reasonable in isolation), the file grows and dilutes what matters, and adherence drops, producing more errors and more rules.conf: high
supportsYour AGENTS.md might be a big problem · p2 — Same reactive rule-by-rule accretion mechanism identified independently — a failure prompts one more rule, and the file only ever grows.
supports ←How Claude Code works in large codebases: Best practices and where to start · p6 — Plugins and centralized ownership are presented here as the organization-scale fix for the same tribal-knowledge/bloat dynamic that the other article's bloat-formation cycle describes at the scale of one developer's personal setup.
supports ←Just Talk To It - the no-bs Way of Agentic Engineering · p7 — His own description of an 800-line AGENTS.md file built entirely from the agent appending notes after incidents is a first-person instance of the self-reinforcing configuration-bloat cycle the other document formalizes.
explicitA June 2026 study of 100 popular repositories found configuration smells were near-universal and often co-occurring: lint-related leakage in 62%, context bloat in 42%, and skill leakage in 35%.conf: high
explicit⚡ disputed & corroboratedAcross 288 runs on 17 real tasks, context files (AGENTS.md/CLAUDE.md) did not produce a clear gain in Claude Code/Codex task correctness, though they did change how the agents worked (e.g. running more targeted tests when warned a suite was slow).conf: high
contradicts ←Encoding Team Standards · p2 — Complicates this article's optimism that encoding a standard as an executable instruction guarantees consistent application: across 288 runs on 17 real tasks, context files (AGENTS.md/CLAUDE.md) did not produce a clear correctness gain, suggesting 'executable' does not by itself guarantee 'reliably executed.'
supports ←Stop Using /init for AGENTS.md · p1 — Both find context files don't cleanly improve correctness at scale (audit-your-agent-files: 288 runs, no clear gain; here: efficiency gain measured but correctness untested) — convergent caution against assuming AGENTS.md helps output quality.
contradicts ←Just Talk To It - the no-bs Way of Agentic Engineering · p7 — He reports his large AGENTS.md 'works incredibly well' and that GPT 'honors entries' far more than Claude did, while the controlled 288-run study across 17 tasks found context files produced no clear correctness gain — a direct tension between his subjective impression and the measured result.
explicitPersonalized skills built from a single developer's interaction history performed no better than a skill borrowed from someone else; a generic skill pooled from many developers was more useful overall, with personalization looking more promising only when the same preference recurred across several similar tasks.conf: high
explicitProse summaries of code answer far fewer behavioral questions than the source code itself (4 of 45 vs. 27 of 45), because summaries smooth over the small details that matter — so context files should point agents at real code rather than describe it.conf: high
explicitThe real cost of an unused, still-installed skill is not primarily token burn per prompt (only names/descriptions load for discovery, inside a budget defaulting to 1% of the context window; the body loads only on invocation) — it is degraded discoverability among a long list, accidental triggering/conflict between similar skills, unowned maintenance surface, and the owner's own loss of ability to reason about the setup.conf: high
refinesAgent Skills · p11 — Osmani's earlier piece frames the cost of clutter as token-level degradation requiring progressive disclosure; this piece clarifies the mechanism precisely — an installed skill's body is not loaded until invoked, so the real cost of an unused skill is discovery/confusion, not per-prompt tokens.
explicit`/doctor` (inside a Claude Code session, the configuration audit) and `claude doctor` (in a shell, installation diagnostics) are two distinct commands that people commonly conflate.conf: high
explicitcorroboratedIf a rule must always hold, it should be encoded in a test, hook, or permission rather than left as prose the model might deprioritize as the file grows — because instruction value expires and prose is the least durable place to store a hard constraint.conf: high
supportsYour AGENTS.md might be a big problem · p7 — Both converge on 'fewer, well-chosen instructions beat exhaustive coverage' — here operationalized as moving hard constraints into tests/hooks/permissions rather than prose.
supports ←Stop Using /init for AGENTS.md · p11 — Both argue that an always-true rule belongs in a test/hook/build-pipeline check rather than as prose competing for instruction budget — convergent on 'encode it, don't just write it down.'
explicitAuditing agent configuration should run on a recurring cadence (every couple of weeks to monthly) via a five-step archive-audit-remove-test-decide sequence, rather than as a one-off purge or never happening at all.conf: high
implicitPeople keep configuration items around less because they are still useful and more because removing them feels unsafe — usually an unexamined bet that the model and harness have not actually improved since the rule was written.conf: medium
explicitA test-removal run is a smoke test, not proof of safety to delete: agent outcomes vary even at low temperature, so a single run isn't decisive, and the run failing in exactly the way the skill was written to prevent counts as the skill re-earning its place.conf: high

Arguments

You should audit and selectively prune agent configuration on a recurring cadence, rather than either accumulating it indefinitely or deleting it wholesale on a fixed schedule. strength: medium

Explicit premises:

Bloat forms through a reactive ratchet cycle (error -> rule -> growth -> adherence drop) that nobody deliberately chooses.
Multiple independent empirical studies (100-repo config-smell study, 288-run context-file study, personalization study, source-vs-summary study, Anthropic's own 80% system-prompt cut) converge on the same finding: instruction value expires and unaudited accumulation carries real cost.
The dominant cost of clutter is confusion (discoverability, accidental triggering, forgotten ownership) rather than raw per-prompt token burn.
Removal is cheap to reverse when preceded by an archive step, which lowers the risk of testing a deletion.

Implicit premises:

A reason that justified adding an item can become invalid without anyone noticing, because nothing forces re-examination.
The 'delete everything periodically' guidance is hard to trust and act on precisely because there's no cheap, low-risk way to verify the deletion was safe — which the archive+test step is designed to supply.
Developers systematically underestimate how much configuration they've accumulated (the 250-skills anecdote, the author's own forgotten anti-slop skills) because nothing surfaces it without an explicit audit.

Inferences:

An organization's instinct to keep everything is a status-quo/loss-aversion bias rather than evidence the items are still earning their place.
Anthropic's willingness to cut 80%+ of its own system prompt with no measured internal-eval loss suggests professional harness builders don't trust their own accumulated instructions any more than individual developers should trust theirs.

Narratives

The author runs /doctor on his own setup and is shocked to find several 'anti-slop' writing skills he'd installed months earlier to experiment with and completely forgotten about, with no idea whether they were triggering together, conflicting, or being ignored.
→ Shows (rather than argues) that the real cost of clutter is loss of the owner's own ability to reason about the setup, not token burn — supports proposition 7.
A tweet the author recalls: someone audited their skills and went from 250 down to 25.
→ Illustrates how far unaudited accumulation can go and normalizes aggressive pruning as a reasonable outcome — supports the article's central prescription.

Questions addressed

✅ Do context files (AGENTS.md/CLAUDE.md) actually help coding agents succeed at tasks?
No clear gain in correctness across 288 runs/17 tasks; they change workflow behavior (e.g. targeted testing) rather than success rate. Best used for things the model can't infer from code (expensive operations, generated files, unusual conventions), not generic advice or code summaries.
✅ Do personalized skills (built from one developer's own history) outperform generic, community/pooled skills?
No — a personal skill performed about the same as a borrowed one; a generic pooled skill did better overall. Personalization pays off mainly when a preference recurs across many similar tasks.
✅ What does an installed-but-rarely-used skill actually cost you?
Not primarily tokens (bodies load only on invocation; discovery uses a small budget). The real costs are degraded discoverability, accidental triggering/conflict, unowned maintenance, and loss of the owner's own mental model of the setup.
⭕ How do you tell, at audit time, whether a surviving rule protects a real live constraint versus documenting a system that no longer exists?

Gaps

incompleteThe prescribed 'test without them' step is explicitly called a smoke test rather than proof, and the author notes agent outcomes vary even at low temperature — but no protocol is given for how many repeated runs, on how many representative tasks, would constitute sufficient evidence before committing to a deletion. Without that, 'test removal' risks becoming as arbitrary as the accretion it's meant to replace.
Read next: Empirical work on how many repeated agent runs are needed to distinguish a real capability regression from run-to-run variance — akin to statistical power analysis applied to agentic evals.
incompleteThe entire audit workflow (archive, /doctor, test-removal, decide) is framed around a single developer's personal setup. It doesn't address shared/team CLAUDE.md, skills, or hooks where multiple developers depend on the same configuration — deleting something 'nobody remembers installing' carries social and coordination risk (who else relies on it? who approves the removal?) that a solo audit cadence doesn't surface.
Read next: How team- or org-level agent configuration should be governed, versioned, and reviewed for pruning — ownership and change-control for shared CLAUDE.md/skill packs, analogous to platform-engineering governance of shared tooling.
closed byHow Claude Code works in large codebases: Best practices and where to start — Directly addresses the team/org scale that article's single-developer audit workflow didn't cover: shared CLAUDE.md and skills governed by a DRI or agent-manager role, plugins distributing a working setup org-wide, and cross-functional working groups for shared ownership of what multiple developers depend on.
uninformedThe cited personalization study used an LLM-based developer simulator rather than real developers (a caveat the author names), and the context-file study covers a specific harness/model pairing (Claude Code and Codex). The article doesn't discuss whether either finding would transfer to other harnesses, to less tool-augmented agents, or to real (non-simulated) developer populations, despite building a general prescription on top of both.
Read next: Follow-up studies replicating the personalization and context-file findings with real developer cohorts and across multiple harnesses, to test whether the 'generic beats personal' and 'context files don't move correctness' results are harness-specific artifacts or general findings.

Closes gaps elsewhere

Anatomy of a perfect Openclaw setup · g1 — Provides the missing evidence the earlier gap flagged as absent: a June 2026 study of 100 repositories quantifying how common configuration bloat/smells are (62% lint leakage, 42% context bloat, 35% skill leakage), plus Anthropic's own finding that cutting 80%+ of Claude Code's system prompt produced no measurable loss on internal evals — direct (if imperfect) data bearing on instruction adherence versus file size.
Claude Code: Best practices for agentic coding · g4 — Directly supplies what that gap said was missing: an account of the bloat failure mode the earlier best-practices piece omitted, a progressive-disclosure account for skills (listing budget defaulting to 1% of context, body loaded only on invocation), and a concrete audit-cadence workflow (archive, /doctor, remove, test, decide) in place of the unbounded 'append a rule' habit.