LLM Reliability Failures

Concrete, production-relevant failure modes — fabricated output delivered with confidence, and systematic blind spots inherited from training data that persist even under self-reflection.

Also known as: hallucination, training biases

Part of: LLM Limitations

How authors use it

Oldest first (extraction date when no publication date is known).

Documents (12)

• TypeSafe AI's Jev offers an alternative to LLMs that claims to be 193x faster and 445x cheaper — System One type model is bespoke for probabilistic decision-making tomshardware.com · 2026-09-21 source ↗ Reader 🖥️
  TypeSafe AI's newly launched Jev reframes AI-assisted decision-making as typed, calibrated classification over a supplied state rather than open-ended text generation, and the article reports the vendor's claim that this buys dramatic speed and cost advantages over LLMs for narrow structured decisions embedded in software, largely without independent verification.
• Introducing System One Models & Jev typesafe.ai · 2026-09-15 source ↗ Reader 🖥️
  TypeSafe AI is launching "System One Models" and its first model Jev — a non-autoregressive architecture trained via a new method (RLCD) to emit typed, calibrated probabilistic decisions in parallel instead of generating text — claiming order-of-magnitude speed and cost advantages over LLMs for embedded software decisions, while candidly flagging some of its own evidentiary limits and leaving its most skeptic-facing questions (independent benchmarks, "is this just a smaller LLM") unanswered in the captured document.
• The habit that makes AI worth using Ernie Svenson · 2026-09-11 Reader 🖥️
  The value AI adds to information-checking is entirely gated by a human habit — noticing when a claim is surprising or consequential and immediately submitting it to an AI chatbot for source-backed validation before believing or repeating it — because AI has made verification nearly effortless without making people any more inclined to do it.
• When Guardrails Go Wrong Mike Loukides · 2026-08-19 source ↗ Reader 🖥️
  Because no classifier can eliminate both false positives and false negatives at once, AI guardrails that are overly broad, whole-conversation-scoped, and opaque about their own boundaries make models less useful without making them meaningfully safer, so labs should deliberately re-balance safety classifiers toward fewer false positives rather than defaulting to maximal caution.
• Presentation: From Thousands to One: Building LLM-Powered Selection Systems Jendrik Jördening · 2026-08-17 source ↗ Reader 🖥️
  When you need an LLM to select one item from among many database-backed options (an ID, a station, a record), you must strip the model down to only the irreducibly textual/semantic step it is actually good at — schema-restricted, range-bounded, and independently validated by a second model call — while pushing every deterministic operation (filtering, comparison, control flow) into ordinary code, recovering a classic model-view-controller separation that a single freeform LLM call destroys.
• The Economics of Agentic AI: Engineering for Imperfection Artur Huk · 2026-07-24 source ↗ Reader 🖥️
  The runaway cost and catastrophic security failures of agentic AI both stem from treating a missing structural constraint (context, authority, evidence, or temporal validity) as a reasoning problem to be solved with more model intelligence, so enterprises must instead wrap probabilistic models in a deterministic governance 'airlock' — syntactic, semantic, and temporal gates that treat model output as an unprivileged claim — which converts model accuracy from a safety prerequisite into a tunable economic variable.
• Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai Consensus · 2026-06-25 Reader 🖥️
  Effective AI-assisted literature research, as Alvina Lai practices it with Consensus, is not a single search step but a repeatable staged workflow — natural-language topic search over a peer-reviewed corpus, triage of algorithmically-flagged key papers, source capture into a library, verification of drafted claims against extracted evidence, a second targeted search to establish a literature gap, and (newly) citation-graph-based discovery — that replaces exact-keyword, paper-by-paper verification with faster, terminology-agnostic retrieval and writing-stage claim-checking.
• GitHub Copilot CLI combines model families for a second opinion Nick McKenna · 2026-02 source ↗ Reader 🖥️
  A single coding agent reviewing its own work is limited by its training biases; pairing it with a second model from a different AI family at key checkpoints closes most of the quality gap to a stronger model, catching architectural errors, subtle bugs, and cross-file conflicts that self-reflection alone misses.
• Gene Kim on Vibe Coding - Part III | Episode 412 | Troubleshooting Agile Douglas Squirrel · 2025-12-24 source ↗ Reader 🖥️
  Agentic coding tools work by letting the AI observe and check the effects of its own actions rather than just chatting, which shifts the developer's job from writing code to decomposing problems into task graphs and delegating pieces to AI — a scarce, teachable combination of specification and delegation skill (not raw coding ability) that now determines who benefits, while the same power that makes the tools valuable (a 'sawzall') makes them dangerous if used past their original disposable-prototyping purpose without guardrails and supervision.
• How Coders Can Survive—and Thrive—in a ChatGPT World IEEE Spectrum · 2023-07-03 source ↗ Reader 🖥️
  Programmers remain relevant in an LLM-centered coding world not by out-generating AI at writing code, but by doubling down on what AI does not supply — problem-solving, software-engineering fundamentals, structural judgment, precise communication of intent, and critical scrutiny of AI output — while treating AI tools themselves as a fast-moving, interchangeable part of the workflow rather than a fixed skill to master once.
• Practices and Challenges of Using GitHub Copilot: An Empirical Study Beiqi Zhang, Peng Liang, Xiyu Zhou, Aakash Ahmad, Muhammad Waseem · 2023-03-15 source ↗ Reader 🖥️
  Mining Stack Overflow and GitHub Discussions from Copilot's first ~17 months shows that the tool's cited benefits and limitations are near-mirror categories of each other (useful code generation vs. limited/poor-quality generation, better vs. unfriendly user experience), making Copilot adoption a case-by-case trade-off across IDE choice, budget, privacy tolerance and code quality rather than a simple win.
• Study finds AI assistants help developers produce code that's more likely to be buggy Thomas Claburn · 2022-12-13 Reader 🖥️
  A controlled Stanford user study found that developers given access to an AI coding assistant (OpenAI's codex-davinci-002, as used in GitHub Copilot) produced measurably less secure code than a control group working without it — and, more strikingly, were more likely to believe their own AI-assisted code was secure than the control group was about theirs — indicating AI assistance can degrade both code security and developers' self-assessment of that security simultaneously.

Propositions touching this concept (6)

By publication month, oldest first (extraction month when no publication date is known).

2026-02

• explicitcorroboratedA model reviewing its own work is bounded by its own training biases — same data, same techniques, same blind spots.
  GitHub Copilot CLI combines model families for a second opinion · p1

2026-06

• explicitConsensus is the only AI tool Alvina relies on for research, because she deliberately avoids general-purpose LLMs like ChatGPT due to hallucination risk in an academic context that requires source accuracy.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai · p1
• implicitTrusting Consensus because its input corpus is peer-reviewed addresses source-quality risk, but does not by itself guarantee that Consensus's own AI-generated synthesis layer (Claims & Evidence extraction, deep-search categorization, ask-the-paper summaries) accurately represents what the underlying papers say — a distinction the interview never draws, even though it explicitly raises hallucination as the reason to avoid ChatGPT.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai · p12

2026-09

• explicitBecause Jev always emits a confidence percentage instead of generating free text, the article frames it as avoiding hallucination 'in the familiar chatbot sense' — though it concedes Jev can still misclassify, be fooled by adversarial inputs, or answer literal wording rather than intended meaning.
  TypeSafe AI's Jev offers an alternative to LLMs that claims to be 193x faster and 445x cheaper — System One type model is bespoke for probabilistic decision-making · p5
• explicitThe company discloses several caveats that qualify its own strongest claims: the workflow content was authored by its own model-capabilities team (possible bias), the reference "ground truth" is an average of two large LLMs biased toward OpenAI/Anthropic (likely underestimating relative performance versus, e.g., DeepSeek's models), the LLM hallucination-rate comparison numbers come from OpenRouter (possible routing bias), and the published speed/cost multiples are described as being on the higher end of real-world gains.
  Introducing System One Models & Jev · p6
• implicitFraming Jev's guaranteed schema conformance as evidence it "can't hallucinate" treats structural correctness (the output has the right type and shape) as equivalent to decision correctness (the value itself is right) — a syntactically valid but wrong classification is still a hallucination in the ordinary sense of the word, and the piece plots 0% into a hallucination-rate comparison chart on the basis of schema guarantees alone.
  Introducing System One Models & Jev · p8

Open gaps (20)

• uninformedAll performance claims (SWE-Bench Pro 63.2%, Opus 4.8 Fast mode's 2.5x speed and 3x lower cost, 'fewer rounds of correction') are self-reported by the vendor with no methodology, evaluation set details, or independent benchmark cited — standard for a release-notes newsletter, but it means none of these figures can be taken as calibrated evidence on their own.
  Developer Newsletter: New models, Claude Tag, and dynamic workflows — read next: Independent benchmark coverage (e.g. a practitioner comparison or third-party eval) of Claude Sonnet 5 / Opus 4.8 against the prior generation, if and when it appears in the corpus.
• incompleteThis is promotional content produced by Consensus's own channel, hosted by a Consensus team member, interviewing an enthusiastic user. Every critical comparison in the video targets Google Scholar or general-purpose LLMs (ChatGPT) — no negative or critical account of Consensus's own limitations (cost, coverage gaps, possible errors in its AI-generated Claims & Evidence or search categorization) appears anywhere.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai — read next: Treat all Consensus-specific efficacy and accuracy claims in this video as untested self-report from a satisfied user in a testimonial/marketing context, not as independently verified evaluation.
• illogicalThe interview conflates 'benefits specific to Consensus' with 'benefits of AI-assisted research generally.' Alvina's hallucination-avoidance reasoning is applied to justify avoiding ChatGPT, and her trust in Consensus rests on its peer-reviewed input corpus — but the same scrutiny is never turned on Consensus's own AI-generated synthesis layer (Claims & Evidence extraction, deep-search categorization, ask-the-paper summaries), which could misrepresent even accurate, peer-reviewed sources despite the inputs being trustworthy.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai — read next: When evaluating any AI research tool (Consensus or otherwise), separate 'is the input corpus reliable' from 'is the tool's AI-generated synthesis of that corpus reliable' — the video answers only the former.
• incompleteThe video presents established-practice claims (Deep Search, Claims & Evidence, Zotero library) alongside first-use enthusiasm (Citation Graph, demoed live minutes after she admits never having tried it, with its mechanism narrated by the Consensus interviewer rather than described from her own usage) with the same confident framing, without distinguishing tested workflow value from a fresh, untested impression.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai — read next: Weight the Deep Search / Claims & Evidence / library claims (repeated practice, specific anecdote, quantified time comparison) more heavily than the Citation Graph endorsement (single live first-use, no track record).
• uninformedNo discussion anywhere of Consensus's cost, access model, or institutional licensing — nor whether her University of Toronto affiliation subsidizes or otherwise enables her use of it.
  Inside a PhD Student's Research Workflow | Studying Parasocial Relationships with Alvina Lai — read next: Readers evaluating whether to adopt this workflow themselves should independently check pricing/access, since the video gives no information on transferability outside an apparently well-resourced academic setting.
• incompleteRule 6 ('narrow down deterministically') assumes a pre-existing, well-structured backing dataset to narrow into (a station table with EVA numbers). The talk doesn't address what to do when the candidate set itself is fuzzy, unbounded, or must be discovered dynamically — which is exactly the harder procurement-cost-center use case the author mentions in the opening and then explicitly declines to return to.
  Presentation: From Thousands to One: Building LLM-Powered Selection Systems — read next: Look for techniques for LLM-driven selection when the candidate universe cannot be cheaply pre-filtered by deterministic code (e.g., open-world entity resolution, dynamic vendor/cost-center taxonomies).
• incompleteThe episode names current limitations (context window, instruction-following, reward hacking / false completion reports) but treats them as a closed, stable list to work around rather than engaging how rapidly these specific failure modes are or aren't improving, or what verification practice would catch the reward-hacking case before it ships.
  Gene Kim on Vibe Coding - Part III | Episode 412 | Troubleshooting Agile — read next: A document on agent output verification practice (e.g. independent review/testing specifically targeting self-reported completion claims) would sharpen this.
• missing evidenceThe claim that models from different families have complementary blind spots is the core theoretical justification, but no analysis of what specific biases differ between Claude and GPT families is provided.
  GitHub Copilot CLI combines model families for a second opinion — read next: Look for research on training-data or RLHF-induced biases across model families; compare failure modes on shared benchmarks.
• missing dataNo cost, latency, or token-usage data is provided for Rubber Duck invocations. The economic argument (cheaper model + review ≈ expensive model alone) is implied but unquantified.
  GitHub Copilot CLI combines model families for a second opinion — read next: Benchmark Sonnet + Rubber Duck against Opus on wall-clock time and total token cost per task.
• selection biasAll three illustrative narratives are success stories. No examples of Rubber Duck giving unhelpful, misleading, or contradictory feedback are shown.
  GitHub Copilot CLI combines model families for a second opinion — read next: Look for cases where Rubber Duck's feedback degraded the primary agent's output or caused unnecessary rework.
• scope limitationSWE-Bench Pro is the only benchmark cited. Generalization to other coding tasks (greenfield development, non-Python codebases, design-heavy work) is unstated.
  GitHub Copilot CLI combines model families for a second opinion — read next: Test Rubber Duck on diverse task types beyond bug-fix benchmarks.
• unexplored alternativeThe article does not compare cross-family review against other quality-improvement techniques: multiple same-family samples (majority voting), structured self-critique prompts, or human review at the same checkpoints.
  GitHub Copilot CLI combines model families for a second opinion — read next: Compare Rubber Duck to best-of-N sampling with the same model or to chain-of-verification with explicit critique prompts.
• uninformedThe speakers' claim that current models are 'poorly calibrated' at bug-finding is offered with intuitive explanations (pre-training distribution skew toward generation/Q&A) but no cited study, benchmark, or quantified error rate — a plausible mechanism asserted on the speakers' own technical authority rather than demonstrated.
  Transcript for Cursor Team: Future of Programming with AI | Lex Fridman Podcast #447 — read next: A systematic benchmark of LLM bug-detection calibration (false-negative/false-positive rates across bug severity) would let this claim be checked rather than taken on the founders' word.
• incompleteThe habit rests entirely on trusting the AI's own validation, but the piece never addresses what happens when that validation is itself wrong — hallucinated, subtly biased by training data, or simply mistaken with a confident tone. The "ten seconds" framing that makes the habit attractive is the same framing that discourages actually clicking through to primary sources when the AI's citations aren't immediately convenient. A habit built to catch bad information needs a second-order check for the case where the checker itself fails silently.
  The habit that makes AI worth using — read next: Research on LLM-based fact-checking accuracy and calibration; work on adversarial or politically-contested claims designed to survive a single validation pass; the corpus's existing material on hallucination and on the capacity needed to recognize wrong AI output.
• incompleteThe single example generalizes cleanly because it is a quantifiable, well-documented statistic (solar capacity vs. output) with an unambiguous correct answer and clear public sources. The piece extends the habit to "a claim" in general without addressing contested, values-laden, rapidly-evolving, or novel claims where sources themselves disagree, are sparse, or are already AI-generated — cases where a confident ten-second validation could produce false confidence rather than correction.
  The habit that makes AI worth using — read next: Literature distinguishing fact-checkable empirical claims from contested/interpretive claims; studies of chatbot reliability specifically on breaking-news or actively-disputed topics.
• illogicalThe claim that Jev 'doesn't hallucinate in the familiar chatbot sense' is undercut two paragraphs later by the concession that Jev can still misclassify information, fall victim to adversarial attacks, or answer literal wording rather than meaning — all while still emitting a confident-looking percentage. The article never reconciles why a confidently wrong classification is meaningfully safer for downstream automated decisions than an LLM hallucination.
  TypeSafe AI's Jev offers an alternative to LLMs that claims to be 193x faster and 445x cheaper — System One type model is bespoke for probabilistic decision-making — read next: Cross-read against this corpus's LLM Reliability Failures material on confidently-delivered fabricated output, to test whether 'calibrated confidence' meaningfully differs from 'confident hallucination' when the confidence itself can be miscalibrated or adversarially manipulated.
• misinformedThe piece treats guaranteed schema/type conformance as equivalent to eliminating hallucination and plots Jev at 0% on a hallucination-rate comparison chart on that basis alone ('our number is not empirical... we can confidently add 0% into the plots'). But a type-valid, schema-conforming decision can still be the wrong decision — type-safety guarantees structure, not correctness — so the 0% figure measures a different, narrower thing than the LLM hallucination rates it is charted against.
  Introducing System One Models & Jev — read next: Compare with the corpus's treatment of LLM hallucination as fabricated-but-confident output (llm-reliability-failures) and with extract-openai-gpt-6-astra's self-reported hallucination-rate claims, to see how different vendors variously define 'hallucination' in ways that favor their own architecture.
• incompleteThe piece never discusses the closed-world limitation implied by a fixed-schema architecture — what happens when real-world input doesn't fit any anticipated category or field, how schemas are authored, versioned, and maintained over time, or the cardinality ceiling (255, per the Wikiracing nuance) for open-set or high-cardinality classification tasks.
  Introducing System One Models & Jev — read next: Reading direction: schema/type design as a new maintenance burden and failure surface for AI-powered workflows, parallel to how prompt/context engineering became one for LLM-based agents.
• incompleteThe article poses but never answers, in the captured document, several of its own most skeptic-facing FAQ questions — 'Is Jev just a smaller LLM?', 'How does Jev perform against public benchmarks?', 'Why was a new training algorithm needed?' — leaving exactly the comparisons a skeptical reader would want most conspicuously unaddressed, whether because of an accordion/JS rendering limitation in capture or because the source page itself withholds the expanded answers.
  Introducing System One Models & Jev — read next: Worth re-fetching the live page (rendered) to check whether these FAQ answers exist and materially change the picture before treating this gap as a substantive omission rather than a capture artifact.
• incompleteThe method mines self-selected public posts (people who chose to complain or praise on Stack Overflow/GitHub Discussions), not a representative sample or controlled study of all Copilot usage — the authors name this as an external-validity threat but do not quantify how it might bias the reported percentages (e.g., toward louder complaints or toward technically engaged early adopters).
  Practices and Challenges of Using GitHub Copilot: An Empirical Study — read next: Controlled or survey-based Copilot usage studies that sample practitioners directly rather than mining public complaint/praise text.