The Problem Is Prompt Debt
Drew Breunig · rss · 2026-07-30 source ↗ Reader 🖥️
EvalsJagged Capability (On Rails / Off Rails)Non-Deterministic SoftwarePrompt DebtPrompting as CommunicationTechnical Debt
Because natural language is an imprecise specification language for a probabilistic model, hand-tuning prompts to define durable system behavior accumulates a distinct 'prompt debt' — slowing iteration, becoming illegible to teams, and locking the application to one model's idiosyncrasies — which can only be paid down by specifying behavior with measurable evals rather than prose and letting automated search (e.g. DSPy, GEPA) generate and optimize the prompt itself.
practical · mixed — Practical: the piece argues for a concrete change in how teams should build AI systems (stop hand-tuning prompts). Form is mixed — expository diagnosis of a mechanism, short narrative examples (leaked system prompts, a controlled study), and an argumentative prescription in the closing section.
Structure
| Part | Role |
|---|---|
| Opening: natural-language prototyping's power and its trap | context |
| 'The prompt debt trap' — three compounding symptoms (slowing iteration, team illegibility, model lock-in) | evidence / diagnosis |
| 'Why prompt debt happens' — imprecise language + probabilistic models, 'fighting the weights', model-specific fixes that don't port | causal mechanism |
| 'Preventing prompt debt' — two principles (measure not prose; search not hand-write) plus a history-of-engineering analogy | prescription / conclusion |
Key terms
- prompt debt Prompt Debt
The compounding cost of using hand-written natural-language prompts as the specification for a system's durable behavior: incremental patches accumulate into a brittle, illegible, model-specific artifact. - fighting the weights Prompt Debt
Repeating an instruction with escalating severity because the desired behavior conflicts with what the model was trained to default to — the specific mechanism that makes prompts grow long and brittle. - jagged frontier Jagged Capability (On Rails / Off Rails)
Borrowed from Ethan Mollick to describe the leading edge of what models can do, used here to justify treating coding-agent practitioners as the relevant field to learn from.
Propositions
- explicitNatural-language prompts let AI applications be prototyped extremely quickly, and for one-off tasks this is optimal — but as a way to build reliable systems, the natural-language prompt is a trap.conf: high
- explicitNatural language was never meant to be a specification language for engineering, and treating it as one quietly caps what you can build.conf: highextendsFragments: February 9 · p9 — Breunig supplies the causal mechanism (imprecise language + a probabilistic model) and concrete evidence for Fowler's more general observation that natural-language prompting moves software sideways into non-determinism.
- explicitPrompt debt has three compounding symptoms in sequence: it first slows iteration (quick 'hot fixes' regress earlier instructions), then incapacitates the team (the prompt becomes illegible even to its author, let alone colleagues), and finally locks the application to a single model (fixes tuned to one model behave differently on another).conf: high
- explicitTeams that try to manage prompt complexity by breaking a brittle prompt into templates assembled at run-time do not escape the problem — the templates themselves evolve into 'a thicket of conditions.'conf: high
- explicitThe imprecision of natural language, combined with the probabilistic nature of language models, means different phrasings of the same intent can produce very different outputs — demonstrated by a study in which a clinical question asked in a patient's voice versus a physician's voice, with identical facts, flipped a model (Opus) from declining all ten times to answering all ten times.conf: high
- explicitIt is not only word choice that matters — spurious, seemingly unrelated statements in a prompt can shift model behavior unpredictably (a Harvard study found that merely stating which NFL team a user rooted for changed refusal rates on sensitive topics), which is why prompts grow more brittle with every added fix: a new instruction meant to quell one error can silently change how the model interprets a separate instruction that worked yesterday.conf: high
- explicitRepeating an instruction with escalating severity ('fighting the weights') is necessary specifically when the desired behavior conflicts with a model's trained-in defaults, and once recognized this pattern is visible in system prompts everywhere — e.g. ChatGPT's image-generation prompt once told the model eight times not to keep talking after returning a generated image, because the model was trained to always continue the conversation.conf: high
- explicitPrompt fixes are tailored to a specific model's idiosyncratic behavior and do not port to another model's weights, because models are not cleanly versioned software — different weights produce different, often undocumented behavior changes — so a prompt that works on GPT-4o may fail on GPT-5.5, and a Berkeley-led study found enterprises stay on older models specifically because newer ones break their existing agents.conf: highextendsAgent Harness Engineering · p5 — Extends the model-harness co-training overfitting insight specifically to hand-written prompts, with concrete evidence (Berkeley enterprise-lock-in study, GPT-4o usage share) that a prompt tuned against one model's trained-in behavior fails unpredictably on another.
- explicitPrompt debt's model lock-in is not the result of frontier labs engineering a clever competitive moat — it is the incidental byproduct of evolving a lossy natural-language specification against a probabilistic model.conf: highrefinesAI models can disappear overnight. Is your engineering team built to survive it? · p6 — Refines what a 'model-agnostic harness' actually requires: keeping tools/routing outside the model isn't sufficient for portability if the prompt itself is a hand-tuned artifact overfit to one model's idiosyncrasies — the prompt must also be decoupled, via evals plus a searched/generated prompt, for real model portability.
- explicitThe first principle for preventing prompt debt is to specify a system's behavior with measurements — evaluations, metrics, typed specifications — rather than prose, because these are legible, shared artifacts colleagues can read and contribute to, restoring the collaboration that brittle hand-written prompts prevented.conf: high
- explicitdevelopedThe second principle is to stop hand-writing prompts altogether: once behavior is defined by measurable evals, the prompt itself becomes something to search for — via systems like DSPy or GEPA — rather than something to craft by hand, because the space of possible wordings is too vast for productive human effort and is exactly the kind of search problem LLMs are suited to.conf: highextends ←Google's AlphaEvolve Reaches General Availability with Evolutionary Code Optimization as a Service · p11 — Both describe the same pattern -- LLM-driven candidate generation searched against measurable evals until convergence -- applied to different targets: AlphaEvolve optimizes code/algorithms, DSPy/GEPA optimizes prompts. AlphaEvolve is a productized, enterprise-scale instance of the same eval-driven search idea.
- explicitOnce prompts are generated against evals rather than hand-tuned, an application is no longer bound to a particular model — evaluating a new model becomes a matter of hours rather than weeks, converting model deprecation or regulatory removal from a 'fire drill' into a routine 'chore.'conf: high
- explicitHand-writing prompts is following the same trajectory as other engineering practices that mature disciplines eventually stop doing manually — hand-written assembly gave way to compilers, hand-tuned queries to query planners, manual memory management (mostly) to garbage collection — and prompt-writing is no different.conf: high
- implicitcontestedThe article assumes that lessons from coding-agent practitioners — a narrow, code-generation-specific population at the 'jagged frontier' — transfer cleanly to specifying AI application behavior in general, even though coding and general application behavior may differ in how measurable or evaluable they are.conf: mediumcontradicts ←Co-Existence and the End of Co-Intelligence · p3 — Mollick predicts coding-agent autonomy gains will generalize broadly to other fields; Breunig argues lessons from the 'jagged frontier' coding population don't transfer cleanly to less measurable/evaluable domains.
- implicitThe remedy assumes that evals/metrics can be made comprehensive enough to substitute for everything a full natural-language specification would otherwise carry — including tone, brand voice, and nuanced judgment calls — rather than only the more easily scorable parts of behavior.conf: medium
Arguments
Hand-tuned natural-language prompts should not be used to specify durable system behavior. strength: high
Explicit premises:
- Natural language is imprecise and models are probabilistic, so identical intent phrased differently yields different outputs.
- Repeated, escalating instructions are needed to fight a model's trained-in defaults, and each addition risks silently regressing an unrelated earlier instruction.
- Fixes are tailored to one model's idiosyncrasies and do not reliably port to another model's weights.
- These effects compound into three symptoms: slowing iteration, team illegibility, and model lock-in.
Implicit premises:
- Reliability, maintainability, and model-portability are necessary properties of a 'system' (as opposed to a one-off task) that natural language cannot deliver at scale.
- Anything meant to be durable needs a legible, checkable, non-prose form that a team — not just its original author — can reason about.
Inferences:
- If prompt debt is structurally caused by pairing imprecise language with a probabilistic model, the same logic would apply to any human-authored natural-language artifact kept as the durable source of truth for behavior — not only prompts narrowly, but e.g. long prose rationale documents or PRDs — yet the author never generalizes the argument this far.
You should stop hand-writing prompts and instead let automated search (DSPy, GEPA) generate and optimize them against evals. strength: medium
Explicit premises:
- Once metrics/evals exist, prompt search becomes a well-defined optimization problem.
- The search space of natural-language phrasings is too vast for productive manual human effort.
- LLMs are well suited to exploring large combinatorial search spaces.
Implicit premises:
- Automated prompt-search systems reliably produce prompts at least as good as hand-tuned ones.
- A machine-searched prompt does not need to be human-legible itself, as long as the evals that define its target behavior are legible.
Inferences:
- A machine-searched prompt could end up even less human-legible than a hand-tuned one, which would leave the 'team illegibility' symptom of prompt debt unresolved rather than fixed — an implication the author doesn't address; he relocates the legibility requirement onto the evals rather than defending legibility of the resulting prompt.
Narratives
- Fable's leaked system prompt repeats one specific copyright-guidance rule six times across six differently named sections (search_instructions, search_usage_guidelines, mandatory_copyright_requirements, hard_limits, self_check_before_responding, critical_reminders).
→ Concretely illustrates the 'slowing iteration' / illegibility symptoms of prompt debt — a rule so unreliable it had to be duplicated under six different framings. - Claude Code's system prompt tells Opus seven times to return multiple tool calls in a single response.
→ A second real-world instance of repeated instructions, reinforcing that this pattern is widespread across leading coding agents, not an isolated anecdote. - ChatGPT's old image-generation prompt instructed the model eight times not to reply after returning a generated image, because the model had been trained to always keep the conversation going.
→ Illustrates 'fighting the weights' — repetition made necessary by a conflict between the desired behavior and the model's trained-in default. - A recent study asked an identical clinical question in a patient's voice and then a physician's voice; the framing alone flipped a model's (Opus's) refusal behavior from declining all ten times to answering all ten times.
→ Provides a controlled, quantified before/after demonstration of the imprecision-plus-probabilistic-model mechanism the article's causal argument depends on. - Datadog's state-of-AI-engineering report found GPT-4o was the most-used model in observed production traffic — with the author noting he's heard from inference providers that usage of GPT-4o-vintage models can exceed 50% of all calls.
→ Illustrates model lock-in as an observed, present-day phenomenon rather than a hypothetical risk.
Questions addressed
- ✅ Why does the promise of rapid natural-language prototyping fail to translate into reliable, evolvable production systems?
Because natural language is imprecise and models are probabilistic, hand-written prompts accumulate 'prompt debt': brittleness from repeated/conflicting instructions, illegibility to a team, and lock-in to one model's idiosyncrasies. - ✅ How can a team pay down prompt debt once it has accumulated?
Specify behavior with measurable evals/typed specs instead of prose, then let automated search (DSPy, GEPA) generate and optimize the actual prompt rather than hand-writing it. - ✅ Is there a way to build AI-driven behavior that stays portable across models?
Yes — by decoupling the specification (evals) from the prompt (a searched, regenerable artifact), swapping models becomes a matter of hours of re-evaluation rather than weeks of prompt rewriting. - ⭕ Can the eval suite itself become stale, incomplete, or gamed — i.e., does specifying behavior 'with measurements' avoid an equivalent kind of debt, or does it just relocate the problem to a different artifact?
- ⭕ Do DSPy/GEPA-style automated prompt optimization systems actually produce reliably good behavior in practice, or are they proven only in narrow benchmarks?
Gaps
- incompleteThe proposed remedy assumes evals can be built comprehensively enough to substitute for everything natural language previously carried — intent, edge cases, tone, nuanced judgment calls — but the piece never addresses how to author a sufficiently complete eval suite, nor whether some behaviors resist reduction to scorable metrics at all, nor the risk that the eval itself becomes the thing being gamed once it becomes the target.
Read next: Read about eval-suite design methodology and Goodhart's-law-style failure modes for LLM evals — does 'specify with measurements' just relocate prompt debt into 'eval debt'? - incompleteThe article never addresses whether an auto-searched (DSPy/GEPA-optimized) prompt is itself legible or auditable. The remedy is shown to solve model lock-in and hand-tuning effort, but the 'team illegibility' symptom the article itself names as a problem could persist or worsen if the resulting prompt is an opaque, machine-optimized artifact nobody reads before it ships — the argument relocates the legibility requirement onto the evals without showing the generated prompt inherits that legibility.
Read next: Look for practitioner accounts of reviewing/auditing DSPy- or GEPA-optimized prompts in production, to see whether illegibility actually recurs one layer down. - incompleteThe remedy is framed as a mechanical software-engineering maturation (assembly to compilers, hand-tuned queries to planners) without engaging its own 'fighting the weights' evidence that some brittleness comes from the model's specific training (RLHF priors), not merely from imprecise language. If some workarounds exist only because a human noticed and reverse-engineered a model's trained-in quirk, it's unclear whether automated prompt search can discover the same workaround, or whether it only automates the trial-and-error of finding one.
Read next: Look for empirical case studies of DSPy or GEPA actually resolving a documented 'fighting the weights' scenario, versus cases where automated search failed to find a workaround a human later found by hand. - uninformedNo cost or effort data is given for adopting the eval-driven, search-based approach — building and maintaining a comprehensive eval suite and running DSPy/GEPA optimization loops is asserted to be a net win over hand-tuning, without any measurement of the engineering investment, timeline, or team skill required to do it well.
Read next: Look for case studies quantifying the setup and maintenance cost of DSPy/GEPA-based prompt optimization pipelines against the hand-tuning baseline they replace.
Closes gaps elsewhere
- This Week in AI: Rethinking the Agent Harness · g3 — Breunig directly substantiates that gap's concern — that treating natural-language prose as a stable substrate elides real brittleness/drift problems — by naming the phenomenon ('prompt debt'), explaining its causal mechanism (imprecise language against a probabilistic model, plus the need to repeatedly 'fight the weights'), and documenting concrete brittleness examples (Fable's system prompt repeating one copyright rule six times; Claude Code repeating a tool-call instruction seven times).