Fragments: February 9
Martin Fowler · article · 2026-02-09 source ↗ Reader 🖥️
Cognitive DebtThe Genie (Beck)Intrinsic Motivation & MakingMental Models & UnderstandingNon-Deterministic SoftwareTest-Driven Development
As LLMs absorb more of the work of building software, the pressing questions are less about model capability than about what humans stand to lose — the team's understanding of the system (cognitive debt), the intellectual craft-joy of model-building, and even the meaning of 'source code' as behavior shifts from deterministic text toward non-deterministic natural-language prompting.
mixed · mixed — A fragmentary, reflective collection of thoughts from an open-space gathering (under Chatham House Rule) plus curated links. It poses open questions and floats speculations about practice rather than arguing a single sustained thesis.
Structure
| Part | Role |
|---|---|
| Skeptic of one's own skepticism (opening frame) | Sets an epistemic stance: decades of over-hyped 'snake oil' tooling justify skepticism, but genuine skepticism must also doubt itself, keeping the inquiry open rather than dismissive |
| Cognitive debt and consolidating understanding | Names the risk that delegating work to LLMs erodes the team's domain and system understanding, and proposes a deliberate consolidation step (by analogy to TDD's refactor) plus a tactic (asking the LLM to explain its code) |
| LLMs as 'drug dealers' (misaligned incentives) | Crystallizes a critique that LLMs supply output without any stake in the long-term health of the system or its humans, because they renew context every cycle |
| The fun of programming and model-building | Distinguishes two joys of programming — delivering features (which LLMs improve) and model-building (which LLMs may erode) — and suggests model-building may instead become a key skill for working with LLMs |
| The future of source code and non-determinism | Asks whether 'source code' survives when prompts elicit behavior by raising abstraction while moving sideways into non-determinism, and speculates that a non-human, token-minimal semantic model (in the spirit of Language Workbenches) could become the durable representation |
| Scala 'lab-leak' joke and elsewhere-on-the-web links | Lightens the tone (an overheard joke) and curates external reading on AI-generated open-source contributions, how transformers work, and AI advertising |
Key terms
- cognitive debt Cognitive Debt
The loss of the development team's understanding of both the domain and the software when much of the building work is sent off to LLMs, leaving the team knowing less about what was built and why - The Genie The Genie (Beck)
Fowler's metaphor for the LLM (from his 'who-is-llm' article) — a powerful entity you delegate to but cannot fully rely on to keep track of everything - model building
The process of coming up with abstractions that help one reason about the domain the code supports; a distinct intellectual pleasure of programming, separate from feature delivery - sideways move into non-determinism Non-Deterministic Software
The trade-off whereby eliciting behavior through prompts and natural-language context raises the level of abstraction but yields non-deterministic behavior rather than deterministic execution - future source code Non-Deterministic Software
A speculated durable, non-human, possibly non-textual semantic representation of a system (in the spirit of Language Workbenches' persisted models) designed to maximize expression with minimal tokens, rendered to humans via projectional editors - LLMs as drug dealers
An overheard framing that LLMs supply output ('give us stuff') but care neither about the resulting system nor about the humans who develop and use it, because each cycle renews their context
Propositions
- explicitGenuine skepticism about LLMs must include skepticism of one's own skepticism, so wholesale dismissal of LLMs is unwarranted despite a history of over-hyped tooling.conf: high
- explicitcorroboratedWhen much of the building work is delegated to LLMs, the development team no longer gains as much understanding of the domain and the software it is building — incurring 'cognitive debt'.conf: highsupports ←Five studies that are changing how I think about AI in software engineering · p10 — Storey's cognitive-debt mechanism (accepting AI code without building the mental model) matches Fowler's 'delegating building work to LLMs means the team no longer gains as much understanding — cognitive debt.'
- implicitJust as TDD's refactor step is where developers consolidate their understanding and embed it into the codebase, teams need an analogous deliberate step to ensure they understand what the LLMs produced.conf: medium
- explicitdevelopedAsking an LLM to explain complex code it wrote — even in a playful form such as a fairy tale — is a practical technique for the team to recover understanding.conf: mediumextends ←My AI Kept Pushing Me to Ship, So I Asked It Why · p6 — Where Fowler frames an LLM's lack of long-term stake in the system as a liability for stewardship, this article shows the same statelessness is an asset when the task is cold, undefended auditing of another session's biased behavior.
- explicitBecause an LLM renews its context with every cycle, it has no stake in the long-term health of the system or the humans who develop and use it.conf: medium
- explicitDelivering useful features is one source of programming's enjoyment, and it will only improve as LLMs become more capable.conf: high
- explicitModel-building — devising abstractions to reason about the domain — is a distinct, valued part of programming that LLMs threaten to erode by drawing attention away from it.conf: high
- explicitModel-building skill may become more important, not less, as a means of working effectively with LLMs.conf: medium
- explicitdevelopedNatural-language prompts and context can elicit a lot of behavior and raise the level of abstraction, but they do so by moving software sideways into non-determinism.conf: highextends ←The Problem Is Prompt Debt · p2 — Breunig supplies the causal mechanism (imprecise language + a probabilistic model) and concrete evidence for Fowler's more general observation that natural-language prompting moves software sideways into non-determinism.
- implicitEven in an LLM-driven world there remains a need for a persistent, durable statement of a system's (non-deterministic) behavior.conf: medium
- explicitIdeas from Language Workbenches — a persisted semantic model that need not be textual or directly human-comprehensible, made readable via projectional editors — may be reintroduced by the rise of LLMs.conf: medium
- implicitThe future 'source code' may be a non-human deterministic representation designed to maximize expression with minimal tokens.conf: low
Arguments
Delegating software work to LLMs threatens to erode the team's understanding of what is built and why, so teams should engineer a deliberate step to consolidate that understanding. strength: medium
Explicit premises:
- Normally, building a system is how developers gain understanding of the domain and the software.
- When much of that work is sent to LLMs, the team may learn less.
- In TDD, the refactor step is where developers consolidate understanding and embed it in the codebase.
Implicit premises:
- Lost team understanding has real downstream consequences for maintaining and evolving the system.
- Understanding does not accrue automatically from LLM-produced code; it must be deliberately recovered.
- A practice analogous to refactoring could serve as the missing consolidation step for LLM-driven work.
Inferences:
- Therefore a deliberate understanding-consolidation practice (e.g., asking the LLM to explain its code) is needed to offset cognitive debt.
LLMs may diminish the joy of programming, but that joy — especially model-building — is worth protecting and may itself be the key to working well with LLMs. strength: medium
Explicit premises:
- Programmers fear LLMs will remove much of the fun from programming, not just their jobs.
- Feature-delivery joy improves as LLMs get more capable.
- Model-building is a separate joy that LLMs may cause developers to neglect.
Implicit premises:
- The abstractions produced by model-building are what let humans reason about, and steer, LLM-driven systems.
- Neglecting model-building would weaken both the craft and the human's ability to direct the LLM.
Inferences:
- Therefore model-building should be preserved and may be reframed as an essential LLM-collaboration skill rather than a casualty of it.
Narratives
- In TDD, refactoring is the step where developers consolidate their understanding and embed it into the codebase — offered as a template for a still-missing step that would let teams understand what LLMs are up to.
→ Grounds the proposed remedy for cognitive debt in an established, familiar practice - Almost two decades ago Fowler became interested in Language Workbenches — tools built on a persisted semantic model with projectional editors — which never had significant impact but whose ideas the rise of LLMs might now revive.
→ Uses prior art that failed to take hold to lend plausibility to a speculation about future source code - An overheard line — 'LLMs are drug dealers, they give us stuff, but don’t care about the resulting system or the humans that develop and use it' — paired with the observation that the LLM renews its context every cycle.
→ Distills the misaligned-incentive critique into a memorable image
Questions addressed
- ⭕ Does sending much of the building work to LLMs mean the team learns less, and what are the consequences?
Raised as the problem of 'cognitive debt'; a deliberate consolidation step and asking the LLM to explain its code are floated, but the consequences and remedies are left open. - ⭕ Do we need a step, analogous to TDD's refactor, to ensure the team understands what the LLMs build?
Posed as an open need; the fairy-tale explanation tactic is offered as one instance, but no defined practice, cadence, or acceptance criteria is given. - ⭕ Will LLMs remove the fun from programming?
Partly: feature-delivery joy is expected to improve, while model-building joy is threatened — though model-building may become a key skill for collaborating with LLMs. - ⭕ In the age of LLMs, will there still be such a thing as 'source code', and what will it look like?
Left open; speculates it may become a non-human, possibly non-textual, token-minimal semantic model in the spirit of Language Workbenches, with projectional editors for humans.
Gaps
- incompleteThe post names 'cognitive debt' as a risk but offers no way to detect or measure it, and no evidence about its actual consequences — leaving the central worry asserted rather than examined.
Read next: Look for empirical work on skill and comprehension erosion under AI-assisted development (e.g., the MIT 'cognitive debt' study) and metrics for team understanding. - incompleteThe proposed understanding-consolidation step 'analogous to TDD's refactor' is gestured at but never specified — no defined practice, cadence, artifact, or acceptance criteria beyond the fairy-tale-explanation tactic.
Read next: Read on comprehension practices for AI-written code: decision records/ADRs, code walkthroughs, and structured explanation or knowledge-consolidation rituals. - illogicalThe 'drug dealer' critique infers indifference to long-term system health from the technical fact that an LLM renews its context each cycle, conflating statelessness with a normative lack of care; it does not consider that persistent memory or explicit long-term-context mechanisms could change the incentive.
Read next: Read on agent memory and long-running-context approaches that could give an LLM stake in a system's ongoing health. - incompleteThe speculation that a non-human, non-textual semantic model becomes 'future source code' does not address how humans would review, debug, version, or trust a representation they cannot read directly — the very problems that helped keep Language Workbenches and intentional programming marginal.
Read next: Read critiques of projectional editing and intentional programming, and work on reviewability/verification of AI-generated artifacts.
Related documents
- How Do You Know That?
- AI coding creates two kinds of debt. You're only measuring one
- Guillaume Barrois' Post
- Cognitive Surrender
- Comprehension at AI Speed: Building a Context Store for Evolutionary Architecture
- Eating My Own Dog Food: How I Used the Framework to Write the Post About the Framework
- Fragments: February 13
- Fragments: February 4
- Who is LLM?
- Martin Fowler & Kent Beck: Frameworks for reinventing software, again and again