Mental Models & Understanding

Genuine human comprehension of how systems behave and why — the capability AI can erode through recognition-without-understanding, and cannot substitute for in a crisis.

Also known as: mental model, mental models, mental model ownership, widen system understanding / mental model, illusion of explanatory depth (forståelsesdybde-illusjon), illusion of explanatory depth, gjenkjenne vs forstå, insight vs summarization, systems thinking

Differential Diagnosis (Diagnostic Reasoning)Mechanical Sympathy (for LLMs)

How authors use it

Oldest first (extraction date when no publication date is known).

AuthorTheir termTheir meaning
Eating My Own Dog Food: How I Used the Framework to Write the Post About the Framework
Marc Millstone · 2026-05
mental model ownershipThe ability to explain and defend every part of your work -- framework, evidence, citations -- without consulting the source, as an engineer would in an incident review.
Designing Data-Intensive Applications: The Cloud & Doing the Right Thing
The Pragmatic Engineer · 2026-05-05
systems thinkingAnalyzing an entire system including human interactions and feedback loops, not just the computerized parts, to predict unintended consequences of data-driven decisions
Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents)
J. Paul Reed · 2026-05-22
Widen system understanding / mental modelThe breadth of an engineer's internal model of how the system actually behaves. Reed argues this is the human capability AI cannot substitute for — and that delegating to AI without maintaining the model is a strategic error masquerading as efficiency.
Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents)
J. Paul Reed · 2026-05-22
Insight vs summarizationSummarization is what AI can produce — frequency-weighted compression of what is present. Insight is what humans produce — surfacing what is absent, counterintuitive, or invisible (the incident caused by returning from vacation, not leaving). The two are not on a spectrum; they are different operations.
4) Morten Eikeland: Det finnes alltid et hierarki
Truls Jørgensen og Audun Fauchald Strand — Heis Podcast · 2026-06-01
illusion of explanatory depth (forståelsesdybde-illusjon)Eikeland's Norwegian rendering of the Rozenblit/Keil cognitive psychology finding. People believe they understand mechanisms (klima, digitalisering, AI) when in fact they merely gjenkjenner the begrep. The illusion is unmasked when one asks 'how does the mechanism actually work?' instead of 'what is your view?' At the strategy level in ledergrupper, asking for mekanismer in place of meninger reliably produces forbannelse and sometimes ejection from the room, because the leaders' own inkompetanse is suddenly visible.
4) Morten Eikeland: Det finnes alltid et hierarki
Truls Jørgensen og Audun Fauchald Strand — Heis Podcast · 2026-06-01
gjenkjenne vs forståRepeated use of a generalised begrep makes us recognise it; recognition is not understanding. Organisations using a word over and over (digitalisering, AI, agile, smidig, hierarki) drift toward feeling they share a meaning when they only share a label. The fix is to drop down into mekanisme — to be forced to explain how the thing actually works.
Netflix CPTO on AI and the future of product and tech roles | Elizabeth Stone
Lenny Rachitsky (host); Elizabeth Stone (guest, Chief Product and Technology Officer, Netflix) · 2026-07-19
Systems thinkingA lightweight, repeatable habit — not a formal discipline — of taking 'one click' outward from the immediate problem to question the broader assumptions ('what am I assuming is true about the broader space?') before executing, deliberately bounded so it doesn't become paralysis.

Documents (29)

Propositions touching this concept (43)

By publication month, oldest first (extraction month when no publication date is known).

2023-07

2025-01

2025-03

2025-11

2026-04

2026-05

2026-06

2026-07

2026-08

2026-09

2026-10

Tensions

Is code a durable artifact to be maintained, or an ephemeral byproduct regenerated from a more durable source (spec/objective)?
from: What is the future of software development in the agentic/AI age?
Assessment: The most-cited open question of the corpus, and genuinely unresolved (Herlein: 'no consensus here. None.'). The 'code is the better spec' counter is strong — code is unambiguous, executable, deterministic, whereas a prose spec is reinterpreted each run, which sits in tension with verifiability being the quality bar. Ruby's move is the most defensible: it dissolves the code-vs-spec fight by locating durability in the evaluation rather than any representation. Verraes and Ördög expose the load-bearing empirical unknown — nobody has data on regenerate-vs-patch cost or on long-term evolvability of agent-maintained code.
Where does the non-delegable human asset now live — the harness/platform, the oracle, the fundamentals, or the developer's understanding?
from: What is the future of software development in the agentic/AI age?
Assessment: This is the deepest structural disagreement — every author agrees a human residue remains but nominates a different residue, and the choice determines what to invest in and what to teach. The harness-centric (Edwards-Alexander), oracle-centric (Ruby), fundamentals-centric (Nilsson/Bangser), and understanding-centric (Fowler) camps are not merely emphases: they imply different career ladders, different hiring, and different failure modes. Ruby's oracle and Bangser's operational-learning are the most operationally concrete; Nilsson's is the least argued. The absence in the corpus of anyone who holds 'nothing is non-delegable' is itself telling (see unexamined assumptions).
Is preserving human understanding of the system a first-order value, or is comprehension safely offloadable so long as output is verifiable?
from: What is the future of software development in the agentic/AI age?
Assessment: The optimist-practitioner vs. skeptic-humanist axis. Fowler and Nilsson treat comprehension and craft as intrinsic goods whose loss is a harm even when output is fine; Herlein and Ocytko treat comprehension as instrumental — replaceable by verification and platform guardrails. The humanist camp has the stronger long-run argument (Fowler's cognitive-debt case and Ocytko's own 'who-guards-the-guards' problem both show verification and guardrails are themselves fallible artifacts that someone must understand), but the offloadable camp is describing what is actually happening at scale. The crux is whether verifiability can ever be robust enough to make understanding optional — which Ruby's own Goodhart gap (a passable oracle diverging from true intent) suggests it cannot fully be.

Open gaps (77)