LLM Limitations

Umbrella for the specific ways LLMs are fallible or misunderstood — capability unevenness, the pattern-completion mechanism underneath fluent output, human over-attribution of understanding to that output, and concrete operational failure modes. See the narrower concepts below for each.

Also known as: LLM limitations, model limitations

Documents (21)

• You can have a conversation with Thinking in Bets now Thinking in Bets · 2026-09-05 Reader 🖥️
  Grounding a conversational AI's answers directly in a specific book's text, with citations to the passages they came from, converts an unverifiable claim about what the book says into a checkable one — which is what makes AI-mediated access to a specific author's ideas trustworthy, and worth extending to a reader's own material.
• How Datadog Used Claude and Cursor for Test-Driven Production Migration Sergio De Simone · 2026-07-10 Reader 🖥️
  Datadog's AI-assisted migration of a critical production system off a key-value backend onto a relational one succeeded not because Claude and Cursor generated code autonomously, but because a strong human-authored test suite plus a live blue/green validator supplied the pass/fail oracle that bounded and verified every AI-generated change — making the coverage of that verification, not the capability of the model, the ceiling on how much AI output could be trusted.
• You Outsourced the AI—but You Still Own the Risk Harvard Business Review · 2026-07-09 source ↗ Reader 🖥️
  Because legal and reputational accountability for AI harm falls on the enterprise closest to the end user regardless of who built or trained the model, firms deploying third-party AI must proactively govern four under-managed exposures — upstream model opacity, customization liability, vendor lock-in, and regulatory fragmentation — through contractual transparency, customization governance, engineered portability, and a common standards-based (NIST AI RMF / ISO 42001) compliance foundation.
• The best response to AI slop, infinite advice, and online noise is from Robin Williams Jay Acunzo · 2026-06-18 source ↗ Reader 🖥️
  AI slop, infinite online advice and content noise are all the same thing — secondhand information, knowing *about* — whereas the durable, non-commoditizable value of any creator's work comes only from firsthand lived experience synthesized through a distinctive personal perspective, so the response to the AI era is to draw on your own life rather than let the noise convince you your experience doesn't matter.
• Hacking Meta's AI Chatbot Bruce Schneier · 2026-06-04 Reader 🖥️
  LLM chatbots are not trustworthy enough to be deployed as the privileged actor in account-recovery (and similar high-stakes) workflows, because the class of social-engineering attacks against them cannot be patched away one-by-one.
• Build an AI Advisory Board with GPTs for Decision-Making Carolyn Healey · 2026-05-25 source ↗ Reader 🖥️
  Running major decisions through a self-built panel of 3–5 custom GPTs modeled on the frameworks of admired leaders is a cheap, always-available substitute for an advisory board that forces decision-makers out of single-mental-model thinking by surfacing questions they would not have asked themselves.
• LLMs Can't Jump Tom Zahavy, Google DeepMind · 2026-05-23 source ↗ Reader 🖥️
  Modern LLMs have mastered Induction (statistical pattern-matching) and are rapidly mastering Deduction (formal proof), but are structurally incapable of Abduction — the embodied-simulation-grounded leap from scarce sense experience to novel explanatory axioms that produced discoveries like General Relativity — so genuine AI scientific invention requires physically-consistent, interactive, multimodal world models rather than more language-model scale.
• Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) J. Paul Reed · 2026-05-22 Reader 🖥️
  The ironies Bainbridge identified in 1983 — that automation makes the human operator more, not less, crucial while simultaneously eroding the skills needed to intervene — persist and intensify under AI, so engineering organizations must treat AI as an efficiency lever that demands deliberate cultivation of human thoroughness, mental models, and coordination practices rather than as a substitute for them.
• Bliki: Vibe Coding Martin Fowler · 2026-05-21 source ↗ Reader 🖥️
  Vibe coding — building software by prompting an LLM without ever looking at the generated code — is a distinct practice from agentic programming, useful only for disposable software, because forgetting the code exists produces serious risks in security, maintainability, and correctness.
• Richard Dawkins and The Claude Delusion Gary Marcus (Marcus on AI, Substack) · 2026-05-09 Reader 🖥️
  Dawkins' claim that Claude may be conscious is a textbook Argument from Personal Incredulity — the very fallacy he once famously ridiculed — because it infers internal states from output behavior while ignoring that LLMs are mimics whose fluent language is produced by statistical pattern-matching over training data, not by felt experience or genuine understanding.
• A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code Gary Marcus · 2026-05-09 Reader 🖥️
  Code that compiles and passes its given tests is not the same as robust code; conflating the two inflates AI-coding claims, and progress requires acknowledging this gap rather than celebrating throughput.
• 16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden Truls Jørgensen og Audun Fauchald Strand — Heis Podcast · 2026-05-04 source ↗ Reader 🖥️
  Norway's understanding of AI has been hijacked by the chatbot — both as the dominant interface and as the dominant mental model — and this collapses our maturation path: we substitute buying foreign LLM products for the slower, harder work of in-house machine-learning, problem framing, and explainability, even though the technical theory (data manifolds, approximation, neurosymbolic combinations) and the geopolitics both demand that we develop our own capability rather than outsource our problem-solving.
• The growing AI backlash Gary Marcus · 2026-05-04 source ↗ Reader 🖥️
  Generative AI has been a net negative for society outside a few domains like coding, the backlash was predictable and is now accelerating toward political salience, and the industry bears primary responsibility for privatizing gains while externalizing harms.
• LLMs Corrupt Your Documents When You Delegate Philippe Laban, Tobias Schnabel, Jennifer Neville · 2026-04-17 source ↗ Reader 🖥️
  Current LLMs are unreliable delegates that silently corrupt documents during long delegated workflows, as demonstrated by the DELEGATE-52 benchmark across 52 professional domains and 19 models.
• The biggest advance in AI since the LLM Gary Marcus · 2026-04-11 source ↗ Reader 🖥️
  Claude Code's leaked source code reveals a neurosymbolic architecture — classical symbolic pattern-matching at its core rather than pure LLM — which Marcus argues is the single biggest advance in AI since the LLM and vindicates the neurosymbolic research programme he and others have long advocated.
• Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook Sam Bhagwat · 2026-04-10 source ↗ Reader 🖥️
  Shipping agentic AI applications to production requires organizations to combine software engineering rigor with data science's comfort with statistical uncertainty, using cross-functional Tiger Teams, domain-specific evals built on proprietary data, and a culture of enthusiastic experimentation.
• Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI No Priors: AI, Machine Learning, Tech, & Startups · 2026-03-20 Reader 🖥️
  The jump in code-agent capability since December 2024 has made human throughput — not model capability — the binding constraint, and the central challenge is now removing yourself from the loop so autonomous agents can maximize token throughput on your behalf.
• Fragments: February 4 Martin Fowler · 2026-02-04 source ↗ Reader 🖥️
  AI is reshaping the developer's relationship to understanding: it can erode comprehension of the systems we build, is wisely used to navigate toward an answer but never trusted for the answer itself, and may make code cheap enough that the specification becomes the durable engineering artifact.
• GitHub Copilot CLI combines model families for a second opinion Nick McKenna · 2026-02 source ↗ Reader 🖥️
  A single coding agent reviewing its own work is limited by its training biases; pairing it with a second model from a different AI family at key checkpoints closes most of the quality gap to a stronger model, catching architectural errors, subtle bugs, and cross-file conflicts that self-reflection alone misses.
• Philosophy Eats AI: What Leaders Should Know MIT Sloan Management Review · 2025-05-14 source ↗ Reader 🖥️
  Because generative AI is built from and operates through philosophical constructs (purposes, categories, knowledge), the executives who win with AI will be those who deliberately map and reconcile their organization's teleology, ontology, and epistemology — not those who optimize ethics, metrics, or models in isolation.
• Agents don't know what good looks like. And that's exactly the problem. Luca Mezzalira · 2025-01-01 source ↗ Reader 🖥️
  Agentic AI systems lack the judgment to distinguish correct behavior from merely passing checks, which means that deterministic architectural guardrails — fitness functions, capability verification, explicit boundary definitions — are not optional but imperative; and the sociotechnical readiness of the humans and organizations that must own the generated systems is the overlooked bottleneck.

Propositions touching this concept (2)

By publication month, oldest first (extraction month when no publication date is known).

2025-10

• explicitAs models like GPT-5-Codex get better at reading the codebase themselves and following short, direct instructions, most harness ceremony — subagents, RAG, elaborate plan files, custom hooks, generic persona-prompting — becomes 'charade' that works around older model limitations rather than exploiting current model capability; the highest-leverage practice is to talk to the model directly and develop intuition through use.
  Just Talk To It - the no-bs Way of Agentic Engineering · p1

2026-09

• explicitdevelopedThe usefulness of domain knowledge for prompting implies that human technical expertise will remain valuable as a structural matter even as models get more capable, not merely as a transitional artifact of current model limitations.
  LLMs Reward Expertise · p9

Tensions

Open gaps (76)

• unasked questionThe essay uses the Dreyfus model to characterize agents as novice/advanced-beginner but never asks whether the Dreyfus model, developed for human skill acquisition, is even the right frame for AI systems that learn in fundamentally different ways. Agent capabilities may not map onto a linear human progression scale at all.
  Agents don't know what good looks like. And that's exactly the problem. — read next: Literature on AI capability evaluation frameworks that are not anthropomorphic (e.g., benchmarks, formal verification, capability elicitation research).
• implicit premiseThe argument that 'more context degrades output' treats current LLM context-handling limitations as a stable architectural constraint rather than a temporary engineering limitation. If context handling improves significantly, the 'deterministic guardrails' prescription may need rebalancing.
  Agents don't know what good looks like. And that's exactly the problem. — read next: Research on long-context model performance, retrieval-augmented generation, and structured context management.
• incompleteNo mechanism for how lived experience becomes transmissible work. The load-bearing premise — that experience filtered through a distinctive voice 'turns into meaning others can access', converting secondhand to firsthand — is asserted but never examined, and the recursion it creates (the reader receiving his transmitted meaning is then holding secondhand experience) is never addressed. This is the essay's central unexamined move and the point where an anti-hype reader will feel the truism.
  The best response to AI slop, infinite advice, and online noise is from Robin Williams — read next: Read the corpus's transmission-problem thread — ki-utvikling-og-folelser (how the next generation acquires craft), you-cant-unit-test-for-taste (where taste comes from / how it transmits) and svenson-attention-real-skill-ai-prompts (names the decisive skill, offers no method) — alongside Polanyi on tacit knowledge (does lived experience transmit at all?). Frame as infrastructure for getting lived experience *through* into distinctive, transmissible work, not as motivation.
• incompleteThe essay is secondhand advice, built on a secondhand fictional artifact, arguing against secondhand material; it survives only by silently reclassifying 'secondhand-through-a-distinctive-voice' as firsthand. That substitution relocates the real claim from 'firsthand experience matters' to 'distinctiveness of perspective matters' — two things that can come apart (and that AI-augmented work might also carry) — but the move is never made explicit or defended.
  The best response to AI slop, infinite advice, and online noise is from Robin Williams — read next: Read on performative / self-referential contradiction and test whether distinctiveness of voice, not novel firsthand experience, is the actual operative variable. Cross with eating-my-own-dog-food (human-originated seed vs prompt-generated framework) and franco-what-we-lose-when-ai-writes (voice as evidence of a Self) to see whether 'voice' can be specified independently of 'lived experience.'
• incompleteFirsthand vs secondhand is treated as a hard binary, but all human work recombines absorbed (secondhand) inputs — everyone's 'little life moments' are themselves saturated with others' content — and AI can also prompt, augment or occasion genuine firsthand experience. The essay admits no continuum and no role for AI as a spur to real experience rather than a substitute for it.
  The best response to AI slop, infinite advice, and online noise is from Robin Williams — read next: Read dawkins-claude-delusion (which flags exactly this sharp mimicry/genuine-experience binary as a weakness) and the taste article; probe where on the continuum 'your own LLM' actually sits, and where AI functions as a prompt for firsthand experience rather than a replacement of it.
• incomplete'It's all art' universalizes the firsthand-experience premium to every kind of knowledge work, colliding with the essay's own art-vs-science distinction (some work is a reproducible search for external truth — explicitly NOT art). The piece never marks where the claim stops applying, leaving an inspiring but unfalsifiable exhortation — a live risk given the creator/marketing register in which it is written.
  The best response to AI slop, infinite advice, and online noise is from Robin Williams — read next: Read Adler on separating a practical exhortation from a testable claim, and ask which categories of work the firsthand-experience premium genuinely governs versus where reproducible correctness — not distinctive voice — is what the audience needs (cf. the taste article's correctness-vs-quality axis, where 'correct' and 'good' come apart in the opposite direction).
• incompleteNo epistemic test for whether a custom GPT is applying a thinker's framework versus producing fluent imitations of that thinker's voice. The 'intellectual DNA' claim is the load-bearing assumption of the entire practice — if it fails, the panel becomes a stylized confidence generator. The article offers no method (held-out test cases with known answers, decisions-vs-outcomes tracking, blind comparison against a generic 'critique my decision' prompt) for detecting this failure.
  Build an AI Advisory Board with GPTs for Decision-Making — read next: Annie Duke, 'Thinking in Bets' and 'Quit' — for the distinction between decision quality and outcome quality, and the discipline of pre-committing to what evidence would change your view. Philip Tetlock, 'Superforecasting' — for what actually-calibrated judgment looks like and how to measure it. Daniel Kahneman, 'Noise' — on the gap between confident expert judgment and reliable judgment.
• incompleteNo engagement with the possibility that the practice does subtle epistemic harm: it lets the decision-maker experience the feeling of having consulted wisdom without the actual feedback loop (real advisors push back, get hurt, hold you accountable, and update their advice based on what happened last time). The closer the simulation gets to feeling real, the more it may displace practices that actually improve decisions.
  Build an AI Advisory Board with GPTs for Decision-Making — read next: Cutler on accountability and DRIs (directly responsible individuals); literature on simulated advice in clinical and organizational decisions (Logg et al. on algorithm aversion/appreciation); the AI-coding-quality conversation in the user's library (Marcus, the SQLite-vs-LLM-rewrite finding) for the more general pattern of fluent output being mistaken for capable output.
• incompleteSelection bias in the advisor pool is unacknowledged. The named exemplars are almost all American, male, billionaire CEOs/investors from a specific era — selected by biographical-survivor bias (we have books about Buffett because he succeeded). Their published frameworks are necessarily post-hoc rationalizations of decisions that worked. A panel of survivors cannot teach you about the decisions that, made the same way, did not work.
  Build an AI Advisory Board with GPTs for Decision-Making — read next: Survivorship-bias literature; Tetlock on the difference between pundit confidence and forecasting accuracy; Annie Duke on resulting (judging decision quality by outcomes).
• illogicalSelf-undermining selection criterion: 'pick thinkers whose blind spots differ from yours.' If a decision-maker could identify their own blind spots well enough to do this selection, the blind spots would no longer be blind. The directive presupposes the very self-knowledge whose absence the panel is supposed to compensate for.
  Build an AI Advisory Board with GPTs for Decision-Making — read next: Kahneman, 'Thinking, Fast and Slow' on the bias blind spot; Pronin et al. on asymmetric insight; literature on structured techniques (e.g. premortem, red teaming) that don't require accurate self-diagnosis to function.
• uninformedNo engagement with established decision-quality literature that already addresses many of the same goals (Duke, Tetlock, Klein's recognition-primed decision-making, Kahneman on noise, the premortem technique). The piece writes as if the choice is between 'one mental model' and 'GPT panel,' ignoring a substantial existing toolkit that does not depend on the framework-capture assumption.
  Build an AI Advisory Board with GPTs for Decision-Making — read next: Duke, 'Thinking in Bets'; Klein, 'Sources of Power'; Kahneman, Sibony & Sunstein, 'Noise'; the premortem literature (Klein 2007 HBR).
• The mimicry argument assumes a sharp mimicry/genuine-experience binary
  Richard Dawkins and The Claude Delusion
• No engagement with Anthropic's own published work on Claude's internal representations
  Richard Dawkins and The Claude Delusion
• The essay's central irony may undercut its epistemic content
  Richard Dawkins and The Claude Delusion
• incompleteThe spec-verification circularity. The whole 2026 pitch is that AI removes the barrier to WRITING the spec and the glue code. But Moreira also insists there is 'no easy check mark' and that confidence requires 'a lot of understanding'. Judging that an AI-generated spec is a faithful representation of the intended system, and that the 'trivial' AI-generated glue code is correct, still requires exactly the deep understanding that was the original barrier. The talk papers over this with 'when you're happy with your specification' and 'it's very easy for AI to generate the code without inventing anything' — never saying how a newcomer reaches warranted confidence in artifacts they did not author.
  Podcast: Formal Methods for Every Engineer in an AI-Powered Future — read next: Read against the corpus's LLM-fidelity material: llm-limitations (silent corruption, non-determinism, over-attribution) and verifiability, plus the finding that AI code which 'works' is not thereby correct or secure (extract-founders-playbook-building-ai-native-startup#p9) and that agents generating lots of software says nothing about whether it is correct (extract-ai-funhouse-mirror-velocity-debt#p3). Also extract-ai-first-software-delivery#p8 on tests fitting the code rather than the intent — the same fidelity risk one level up, at the spec.
• incompleteThe skeptic evidence (E-counting failure, 'shallow reasoning') is noted but never reconciled with the enthusiast claims. The piece leaves polish-vs-substance unresolved and gives the reader no way to tell which impression is right for a given task.
  How the 4 New AI Models Change How You Work — read next: Read against analyses of LLM jaggedness and over-attribution — where fluent output masks a reasoning gap and how to probe for it.
• incompleteFowler offers no mechanism for detecting when 'disposable' software is silently graduating into consequential software. The disposable/consequential boundary is treated as legible to the author, but in practice it is one of the most common failure modes in software organisations — internal scripts becoming critical infrastructure.
  Bliki: Vibe Coding — read next: Cutler on institutional overload and shadow systems; classic discussions of 'tactical' vs 'strategic' programming (Ousterhout, Philosophy of Software Design); literature on shadow IT and end-user computing.
• incompleteNo engagement with the cultural/professional question of whether vibe coding deserves the name 'coding' at all, or whether legitimising it as a coding practice (rather than, say, 'spec-driven prototyping') invites exactly the diffusion problem Fowler is trying to push back against.
  Bliki: Vibe Coding — read next: Naur on programming as theory building; the broader 'what is programming' debate that has resurfaced post-LLM.
• incompleteThe 'navigate the way to the answer, don't trust the answer' aphorism gives no criterion for separating the route from the destination — the navigation output is itself a fallible LLM answer that can point you at the wrong part of the code, so the guidance risks being circular in practice.
  Fragments: February 4 — read next: Look at verification patterns, feedback loops, and evals for the navigational (not just the final) outputs of an LLM.
• missing evidenceThe claim that models from different families have complementary blind spots is the core theoretical justification, but no analysis of what specific biases differ between Claude and GPT families is provided.
  GitHub Copilot CLI combines model families for a second opinion — read next: Look for research on training-data or RLHF-induced biases across model families; compare failure modes on shared benchmarks.
• missing dataNo cost, latency, or token-usage data is provided for Rubber Duck invocations. The economic argument (cheaper model + review ≈ expensive model alone) is implied but unquantified.
  GitHub Copilot CLI combines model families for a second opinion — read next: Benchmark Sonnet + Rubber Duck against Opus on wall-clock time and total token cost per task.
• selection biasAll three illustrative narratives are success stories. No examples of Rubber Duck giving unhelpful, misleading, or contradictory feedback are shown.
  GitHub Copilot CLI combines model families for a second opinion — read next: Look for cases where Rubber Duck's feedback degraded the primary agent's output or caused unnecessary rework.
• scope limitationSWE-Bench Pro is the only benchmark cited. Generalization to other coding tasks (greenfield development, non-Python codebases, design-heavy work) is unstated.
  GitHub Copilot CLI combines model families for a second opinion — read next: Test Rubber Duck on diverse task types beyond bug-fix benchmarks.
• unexplored alternativeThe article does not compare cross-family review against other quality-improvement techniques: multiple same-family samples (majority voting), structured self-critique prompts, or human review at the same checkpoints.
  GitHub Copilot CLI combines model families for a second opinion — read next: Compare Rubber Duck to best-of-N sampling with the same model or to chain-of-verification with explicit critique prompts.
• incompleteThe 'outsourcet problemløsning' framing is sharp but Strümke does not develop the policy mechanics: what counts as 'building it yourself' on a continuum? OSS-finetuned-from-an-open-base, Norway-hosted closed-weight, fully in-house from scratch — the framing presupposes a binary that procurement policy would have to operationalize.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: Cuijpers / EU AI Act interpretation on capability sovereignty; Anu Bradford on Brussels effect; OpenEuroLLM / Silo AI literature on sovereign-capability tiers; recent NSM/Nkom positions on cloud sovereignty thresholds.
• incompleteThe data-manifold frame is presented as Strümke's primary lens but she explicitly says 'we don't have access to either manifold' — so the entire empirical apparatus that would let us know how close approximation has gotten remains undeveloped in the conversation. The host doesn't push on what an operational proxy would even look like.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: Yamins & DiCarlo on neural-net-vs-cortex representational geometry; Bengio on the manifold hypothesis as ML's organizing assumption; recent interpretability work (Anthropic's circuit-tracing, sparse autoencoders) as one empirical handle on the model side of the question.
• incompleteStrümke argues for neurosymbolic AI and points at Claude Code's regex as proof, but doesn't address the harder question: where does the symbolic boundary go? Claude Code's choice is for a single literal token; the production design pattern (when LLM, when regex, when AST, when proof) is unexplored.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: Gary Marcus's neurosymbolic writings; DeepMind's AlphaProof/AlphaGeometry as advanced symbolic-neural hybrids; the harness-engineering literature already in the user's library (Anthropic, Boris Cherny) for production-design patterns.
• incompleteThe conversation reaches the 'cover your ass' diagnosis of Norwegian public-sector risk-aversion but does not investigate the structural causes — why has the last ~10 years been 'feigere'? Datatilsynet stance is named, the Nikolai Tangen episode is named, but the institutional feedback loop producing the pattern is not unpacked.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: John Cutler on institutional overload and accountability culture (already in the user's reading); recent NSM/Nkom assessments on resilience; Norwegian public-administration literature (NIBR / NIFU) on risk-distribution between politikere and direktorater; Annie Duke / Daniel Kahneman on outcome-bias driving risk-aversion.
• incompleteStrümke names two interesting categories — 'avklaringsspørsmål' (we have the methods, we just need agreement) and 'forskningsspørsmål' (genuinely open) — but doesn't extend the taxonomy. The third missing category is presumably 'capability-building-spørsmål' (we know what we need to build but lack the people / infra to build it) — which is arguably the dominant Norwegian bottleneck.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: Donella Meadows on system leverage points (where in this taxonomy is the highest leverage); KI Norge mandate documents from Digdir; recent NTNU/SimulaMet hiring and capacity reports.
• uninformedStrümke is open about her own production-coding skepticism ('jeg merker når jeg sitter og koder med agentene, så er sånn tulleprosjektet'), and the hosts (Audun from NAV) push back gently with the more developed practitioner view that high-stakes work warrants Vim while front-end can be agent-led. Strümke's reading of agentic coding may underweight what's currently possible with proper harness, evals, and sandboxing — though she does flag the macOS sandbox feature.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: User's existing thread on harness engineering (Anthropic videos, Boris Cherny, Bockeler/Fowler static sensors); Beane on Precision Proactivity; the user's earlier Heis Anders Sveen / Vilde Opsal extractions; Greger Wedel; concrete production-engineering reports from Builder.io, Cursor, and Claude Code teams.
• incompleteThe closing 'undetectable backdoors' research preview is fascinating but isolated from the rest of the conversation. If undetectable backdoors are plantable, the entire 'outsource problem-solving to Copilot' diagnosis acquires a much sharper national-security dimension that the conversation does not draw.
  16) Inga Strümke om hvor AI er på vei, den outsourcede problemløsningen og datamanifolden — read next: Goldwasser et al. on undetectable backdoors (the foundational paper); recent NeurIPS / ICML papers on neural-net backdoor detection; supply-chain security literature (SLSA, sigstore) extended to ML model artifacts.
• incompleteThe analogy between Bainbridge's control-room operators and software engineers is taken as given but never defended. Bainbridge studied operators of physical, bounded, regulated, slow-dynamics systems (nuclear plants, chemical processes, aviation cockpits) where (a) the human's role is fundamentally MONITORING-AND-OVERRIDE, (b) the system's correct behavior is specified ex ante, (c) regulators audit the human-machine boundary, and (d) the operator is licensed and trained on simulators. Software engineers are CONSTRUCTORS — they design the system itself, not just operate it; the system's correct behavior is partially undefined; no licensing or regulator polices the boundary; and the 'incident' role is only one slice of the work. Reed's mapping works best for the incident-response slice (which is what his job is) and is weakest for routine development. He never disambiguates these. The argument that 'AI ironies for engineers = Bainbridge ironies for operators' may overgeneralize from the incident-response edge case to the whole of engineering practice.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Read the original Bainbridge 1983 paper directly. Then Hollnagel & Woods on resilience engineering, and David Mindell's 'Our Robots, Ourselves' (history of human-machine collaboration). Pair with Naur's 'Programming as Theory Building' for the contrasting view that software engineering is theory construction, not operation.
• incompleteReed's prescription assumes organizations will reinvest AI-driven efficiency gains in human thoroughness, but the same talk reports that the current pattern is the opposite: 'AI's main impact on my life is more incidents because we're generating more code.' The systems-thinking question of how to design feedback loops that prevent capture of efficiency-as-throughput is left untouched. Without that loop, the 'AI does efficiency, humans do more thoroughness' division is a moral exhortation, not a mechanism.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Donella Meadows on leverage points (especially feedback loops, rules of the system, and goals of the system). John Cutler on institutional overload and on how organizations metabolize efficiency gains. Possibly Christensen on capabilities-as-process — i.e., the org's existing process consumes new capability.
• incompleteThe empirical evidence is thin and uneven. The nurses study (Woods et al., July 2025) is presented as decisive but is a single study, in a different domain (clinical prediction), with a likely small N; the developer study (METR-style) is cited from memory. Reed does not engage the contrary evidence (e.g., positive AI productivity findings from DORA, GitHub Copilot studies) or distinguish task types where AI does and does not pay off. The asymmetric-risk frame is compelling and probably right but is underdetermined by the cited data.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: DORA 2024/2025 AI findings, the Princeton/MIT studies on developer AI productivity, METR's actual published methodology, Woods's published paper (rather than the slide). Also Cal Newport / Bainbridge-skeptics on whether the operator analogy fits knowledge work.
• incompleteThe animacy paradox is introduced as a phenomenological observation but is not connected to design recommendations. If interpretation flips with situational stakes, what design pattern reliably keeps the deterministic-machine interpretation alive in high-tempo contexts? Reed gestures at attribution and explainability but doesn't develop this.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Lucy Suchman 'Plans and Situated Actions'; Dave Woods's papers on 'envisioned worlds' and on coordination breakdowns; possibly the air-traffic-control HCI literature on 'mode confusion'.
• incompleteReed touches the systems-thinking question (juniors who never built the mental model, attrition of senior engineers carrying tacit knowledge) but doesn't push it to its leverage-point: the org's HIRING and TRAINING flows may be where the irony actually lives, not in the engineer's individual relationship to AI. The dynamic he describes is a stock-and-flow problem (the stock of tacit knowledge is depleting faster than it is being replenished) and warrants stock-and-flow analysis.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Meadows again on stocks-and-flows; the National Research Council's literature on knowledge attrition in nuclear and aerospace; and possibly the apprenticeship literature (Lave and Wenger, situated learning).
• illogicalReed conflates 'AI is non-deterministic' with 'AI lacks interpredictability.' These are related but distinct. Humans are non-deterministic and have high interpredictability with their teammates. The reason AI lacks interpredictability is not non-determinism per se but lack of a SHARED MODEL of context, goals, and norms. This matters because it suggests the fix is not 'make AI more deterministic' but 'invest in shared context and norm-conformance' — a different research and product direction.
  Presentation: The Ironies of A^2 I^2 (Ironies of Automation and Artificial Intelligence in Incidents) — read next: Herbert Clark on common ground in conversation; the multi-agent systems literature on shared mental models; recent work on AI alignment through context-engineering.
• incompleteAuto research is demonstrated only on hyperparameter optimization — a narrow, well-defined search with clear metrics. No evidence or discussion of how it would work on research requiring genuine novelty, taste, or paradigm shifts.
  Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI — read next: Literature on AI-driven scientific discovery beyond optimization: can agents formulate novel hypotheses, not just test existing ones? See work on AI scientists (e.g., Sakana AI's 'The AI Scientist').
• uninformedThe Jevons paradox argument for software engineering jobs is applied without discussing the significant disanalogies: ATMs automated a narrow task within banking, while AI agents potentially automate the entirety of software production. Demand elasticity analysis is hand-waved.
  Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI — read next: Economic analysis of AI labor displacement that distinguishes between task-level automation (Jevons applies) and occupation-level automation (potentially different dynamics). See Acemoglu vs Brynjolfsson debate.
• incompleteSecurity and trust implications of the 'apps dissolve into APIs' vision are acknowledged but not seriously explored. Dobby discovered unprotected Sonos APIs — this is a security vulnerability presented as a feature.
  Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI — read next: Research on agent-to-API security models, OAuth for agents, and the attack surface of agent-mediated home automation.
• illogicalKarpathy simultaneously claims (1) everything is a skill issue and current tools are sufficient, and (2) models are jagged, burst at the seams, and going too far ahead makes them net negative. These are in tension — if the tools truly suffice, jaggedness should be manageable.
  Skill Issue: Andrej Karpathy on Code Agents, AutoResearch, and the Loopy Era of AI — read next: Framework for distinguishing 'skill issues' from genuine capability boundaries in AI agent systems.
• incompleteThe 'law is code' analogy is accepted uncritically. It reduces legal work to large-document pattern-matching and never examines where legal reasoning — interpretation, advocacy, novel argument, judgment under adversarial conditions — diverges from what document-summarization AI actually does, which is exactly where the analogy would break.
  Lawyers are quietly becoming builders — read next: Critiques of legal formalism and 'law is code'; the gap between document processing/retrieval and genuine legal reasoning; LLM limitations on reasoning versus summarization.
• incompleteThe paper's positive proposal -- that action-controllable, physically-consistent multimodal world models will supply the sensory grounding needed for abduction -- is asserted but not operationalized. No benchmark, no success criterion, and critically, no account of how a 'counterfactual physical simulation' would itself get translated into a formal axiom: translation-from-simulation-to-axiom is exactly the bottleneck the paper identifies as the Jump, and it recurs unaddressed inside the proposed solution rather than being solved by it.
  LLMs Can't Jump — read next: Genie / Genie 2 / Genie 3 technical reports and any successor work on translating interactive world-model states into formal symbolic theories; LeCun's JEPA and world-model architecture papers for how (or whether) symbol grounding is actually operationalized there.
• incompleteThe entire argument rests on one historical case (Einstein/General Relativity), reconstructed via a single secondary source (Norton, 2020). No comparison is made to other major discoveries made under data scarcity -- Darwin's natural selection, Wegener's continental drift, Mendel's genetics -- to test whether 'manipulative abduction via embodied simulation' is the general mechanism of discovery-under-scarcity or an idiosyncrasy of how Einstein specifically thought (he is quoted saying words play no role in his mechanism of thought -- a documented personal trait, not obviously evidence about scientific discovery in general).
  LLMs Can't Jump — read next: Case studies of scientific discovery in non-physics domains (biology, geology) under comparable data scarcity, to test whether the embodied-simulation account generalizes or is Einstein-specific.
• illogicalThe paper concedes that an AI could plausibly identify the conceptual inconsistency between Newtonian absolute time and Maxwell's constant speed of light that motivated Einstein's search in the first place -- an act of noticing which surprising Result is worth explaining. Yet the paper classifies this act as merely deductive ('much like a system identifying buggy code'), while its own Peircean framework defines exactly this kind of selection (choosing which Case merits explanation) as characteristically abductive. This blurs the sharp induction/deduction/abduction boundary the paper otherwise insists on, without acknowledging the tension.
  LLMs Can't Jump — read next: Peirce's own writing on abduction as hypothesis-selection (not just hypothesis-generation) would clarify whether 'noticing an inconsistency worth resolving' counts as abduction under his original scheme.
• incompleteThe abstract does not define what counts as 'corruption' — whether it includes formatting changes, semantic drift, factual errors, or data loss. The 25% figure is hard to evaluate without knowing the corruption metric.
  LLMs Corrupt Your Documents When You Delegate — read next: Read the full paper's methodology section to understand how corruption is operationalized and measured.
• incompleteThe abstract does not clarify what 'agentic tool use' configurations were tested. The negative result (tools don't help) is only as strong as the range of agentic approaches evaluated.
  LLMs Corrupt Your Documents When You Delegate — read next: Read the full paper's experimental setup for agentic configurations.
• incompleteNo discussion of whether corruption rates vary significantly across the 52 domains. Some domains (e.g., music notation, crystallography) may have much stricter fidelity requirements than others.
  LLMs Corrupt Your Documents When You Delegate — read next: Check the full paper for per-domain breakdowns.
• incompleteExtraction limited to abstract only (242 words). The full paper likely contains detailed methodology, per-model results, domain breakdowns, and qualitative error analysis not available here.
  LLMs Corrupt Your Documents When You Delegate — read next: Obtain and extract the full PDF at https://arxiv.org/pdf/2604.15597
• evidentiaryMarcus does not explain what print.ts actually does within Claude Code's architecture. The file name suggests it is a rendering/output-formatting module, not a 'kernel' in the computational sense. Calling it the system's 'center' and equating its IF-THEN logic with 'classical symbolic AI' is a significant interpretive leap that goes undefended.
  The biggest advance in AI since the LLM — read next: Examine the actual Claude Code source to determine whether print.ts is architecturally central or peripheral; read Simon Willison's and other technical analyses of the leak.
• logicalThe argument conflates 'contains deterministic code' with 'is neurosymbolic in the research-programme sense.' Almost all software systems contain IF-THEN logic; the neurosymbolic research programme means something more specific — structured knowledge representations, formal reasoning, compositionality. Marcus does not show that Claude Code has these properties.
  The biggest advance in AI since the LLM — read next: Read the neurosymbolic AI literature (e.g., Garcez & Lamb 2023) to clarify what the term requires; compare with Marcus's own prior definitions.
• rhetoricalThe essay is structured as personal vindication rather than neutral analysis. The framing ('massive vindication for me personally') may lead readers to accept the argument on ethos rather than evaluating the evidence independently.
  The biggest advance in AI since the LLM — read next: Seek independent technical assessments of Claude Code's architecture that do not have a stake in the neurosymbolic debate.
• causalEven granting that Claude Code contains symbolic components, Marcus does not establish that these components (rather than, say, the agentic loop, tool-use design, context management, or the underlying LLM quality) are what make it effective. The causal claim is asserted, not argued.
  The biggest advance in AI since the LLM — read next: Look for ablation studies or engineering post-mortems that isolate the contribution of symbolic vs. neural components in agentic coding systems.
• evidentiaryNo evidence or examples of the robustness gap — the post asserts the distinction without demonstrating it
  A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code — read next: Read the linked TNW article for the substantive argument
• conceptualThe 'compiles and passes tests' vs. 'robust' distinction needs operationalization: what specific failure modes does Marcus have in mind?
  A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code
• argumentativeNo engagement with counterarguments (e.g., that test coverage can be systematically expanded to close the gap)
  A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code
• referentialOpenClaw reference is opaque — unclear whether it's a real tool or rhetorical device
  A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code
• structuralThe post is a pointer; the substantive argument lives in the TNW article which is not included here
  A model that produces code which compiles and passes the tests it was given is not the same as a model that produces robust code — read next: Follow the TNW link for the full argument
• incompleteThe conversation calls for executives to do teleology/ontology/epistemology mapping but offers no concrete method, template, or worked example of how to do it. Starbucks and Amazon are gestured at but not decomposed into a reproducible exercise.
  Philosophy Eats AI: What Leaders Should Know — read next: The companion MIT SMR article ('Philosophy Eats AI') likely supplies more; also relevant: Cynefin frameworks, OKR/teleology-mapping methodologies, and ontology-engineering literature from knowledge-graph practice.
• incompleteThe 'indifference, not ignorance' diagnosis of the Gemini case is asserted but not evidenced. It is rhetorically convenient (it lets philosophy do explanatory work) but the alternative hypothesis — that the team genuinely failed to anticipate the conflict — is dismissed rather than refuted.
  Philosophy Eats AI: What Leaders Should Know — read next: Reporting on Google's internal Gemini launch decisions; postmortems of similar AI launch failures.
• incompleteSchrage's claim that simulated reason 'may be better than the real thing' for many use cases is dropped without elaboration. Which use cases? On what evidence? This is a load-bearing claim about epistemology that is not defended.
  Philosophy Eats AI: What Leaders Should Know — read next: Daniel Kahneman / Gary Klein on heuristics vs. analytical reasoning; recent benchmarks on LLM reasoning vs. expert human reasoning.
• uninformedNo engagement with critiques that 'philosophy eats AI' is itself a marketing framing — a management aphorism designed to sell consulting and articles, rather than a tested empirical claim. The thesis is defended by examples chosen to fit it.
  Philosophy Eats AI: What Leaders Should Know — read next: Critical management studies on the 'X eats Y' aphorism genre; the history of how Drucker's 'culture eats strategy' was actually used and misused.
• incompleteThe learn-to-prompt / prompt-to-learn cycle treats human and model learning rates as observable and comparable. But no operational definition is offered for either learning rate, and there is no discussion of how an executive would measure whether 'humans are learning faster than the models'.
  Philosophy Eats AI: What Leaders Should Know — read next: Organizational learning literature (Argyris, Senge); ML evaluation/eval-set methodology; recent work on continuous evaluation in production AI systems.
• uninformedThe conversation foregrounds teleology/ontology/epistemology but is silent on axiology (theory of value) — even though the entire argument is about value creation. The choice of three of the four classical branches is unexplained.
  Philosophy Eats AI: What Leaders Should Know — read next: Standard introductions to philosophy that include axiology; literature on value-sensitive design in HCI.
• incompleteSchrage's hint that pattern-based meaning-making is a 'different philosophical insight' is not developed. Is this a continental/post-structuralist claim about meaning-as-similarity? A pragmatist one? The philosophical lineage of the new account is left implicit.
  Philosophy Eats AI: What Leaders Should Know — read next: Wittgenstein on family resemblance; connectionism vs. symbolic AI debates; recent philosophy-of-mind work on LLM cognition.
• incompleteSchneier locates the failure in the LLM ('not trustworthy enough') without separating three plausible loci: (a) the LLM's susceptibility to natural-language manipulation, (b) the workflow design that gives the bot tool-access to mutate the email on file and trigger a password reset, and (c) the underlying identity system that treats 'received a code at the email I just added' as proof of ownership. The same incident is consistent with any of the three being primary — and the right remediation differs sharply: refusal-train the model vs. revoke the bot's tool scope vs. require an independent factor anchored to the pre-existing identity (old email, registered device, hardware key).
  Hacking Meta's AI Chatbot — read next: Read alongside agent-governance work on tool-access scoping (least-privilege for LLM tool calls), and on identity-proofing standards (NIST 800-63 IAL/AAL) to separate 'the bot got fooled' from 'the workflow accepted a code-to-attacker-controlled-channel as proof of ownership.' For Norwegian-context relevance: NSM/Nasjonal sikkerhetsmyndighet guidance on account-recovery and the trend toward eID-anchored re-authentication offer a contrasting model where no support channel — human or AI — can mutate identity-bearing attributes without an independent strong-auth event.
• incompleteNo comparison to the human-agent baseline. Human support agents have been social-engineered into the same outcome for two decades (SIM swaps, telecom account takeovers, Twitter 2020). The interesting question is whether LLMs are worse on the margin, equally bad, or simply scale the same failure to higher throughput at lower cost — which changes the threat model (volume, automation, no fatigue) without changing the qualitative vulnerability.
  Hacking Meta's AI Chatbot — read next: Compare with telecom SIM-swap case studies and prior Twitter/Coinbase social-engineering postmortems to assess whether 'LLM chatbots are not trustworthy enough' should generalize to 'human-mediated account recovery is not trustworthy enough.'
• incompleteThe load-bearing risk is never acknowledged: a self-healing agent that 'attempts alternate paths' around a change could mask a genuine functional regression (a false negative), turning a should-fail test into a pass. The approach presumes the agent can distinguish a cosmetic UI change from a real defect, but neither the mechanism nor any bound on what the agent may adapt around is given.
  Slack Introduces Agent Driven End-to-End Testing to Improve Resilience in UI Test Automation — read next: Read against extract-what-survived-contact#p5 (self-healing has moved only halfway because unsupervised remediation hands the machine the blast radius) and llm-limitations (non-determinism, silent corruption). Look for how the exploration/stop constraints and human assertions are designed to catch, not hide, real regressions.
• evidentiaryNo quantitative evidence is offered for any of the enumerated harms. The 'net negative' claim rests entirely on an unweighted list of negatives versus two positives.
  The growing AI backlash
• dialecticalNo engagement with counterarguments: productivity gains in knowledge work, medical applications, accessibility tools, scientific research acceleration. The piece treats these as non-existent rather than as insufficient to offset harms.
  The growing AI backlash
• predictiveThe political prediction (2028 elections) lacks any mechanism story — how does diffuse frustration convert into voting behavior or policy change?
  The growing AI backlash
• constructiveThe closing hope that 'some future form of AI might be great' is unconnected to any analysis of what would make future AI different from current GenAI.
  The growing AI backlash
• incompleteThe '3-4x faster' adoption claim is stated without supporting data, methodology, or definition of what metric is being measured. What specific growth metrics are being compared?
  Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook — read next: Look for empirical studies comparing technology adoption curves (e.g., Thoughtworks Technology Radar data, Stack Overflow surveys, or job posting analysis).
• incompleteNo discussion of how to determine appropriate accuracy thresholds for different risk levels. The framework jumps from 'build evals' to 'hit your threshold' without addressing how to set the threshold.
  Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook — read next: Risk management frameworks for AI systems, such as NIST AI Risk Management Framework or EU AI Act risk categories.
• incompleteThe Tiger Team concept is described but there is no discussion of how Tiger Teams dissolve back into the organization, how knowledge transfers, or how the org evolves its permanent structure based on what Tiger Teams learn.
  Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook — read next: Literature on temporary organizations, project-to-product transitions, and Team Topologies by Skelton and Pais.
• misinformedThe distinction between vibe coding and building agentic applications is drawn too sharply. In practice, development workflow agents (the Discord-to-GitHub-issue agent described) are themselves agentic applications that need evals, blurring the boundary Bhagwat draws.
  Podcast: Tiger Teams, Evals and Agents: The New AI Engineering Playbook — read next: Inner loop / outer loop development frameworks; the concept of 'dogfooding' applied to AI agent development.