When Guardrails Go Wrong

Mike Loukides · rss · 2026-08-19 source ↗ Reader 🖥️

Agent Security Agent Skills Claude Code Guardrails LLM Reliability Failures

Because no classifier can eliminate both false positives and false negatives at once, AI guardrails that are overly broad, whole-conversation-scoped, and opaque about their own boundaries make models less useful without making them meaningfully safer, so labs should deliberately re-balance safety classifiers toward fewer false positives rather than defaulting to maximal caution.

practical · argumentative — Loukides uses a first-person incident (his Claude skill getting blocked by Sonnet's cyber safeguards) as evidence for a normative argument about how AI safety guardrails should be engineered. The piece moves from anecdote to a general framework (the ROC curve) to a call for the industry to rebalance false positives against false negatives — it argues for a change in practice, not merely for understanding a phenomenon.

Structure

Part	Role
Personal anecdote: a daily-used Claude Code skill (a news-digest skill Loukides built himself) suddenly fails with a 'cyber safeguards' error message.	evidence / hook
Diagnosis: the trigger was the phrase 'vulnerabilities, exploits, threat reporting' in an AI-generated (and inaccurate) description of Hacker News within the skill file; other models (Haiku, GPT) ran the same skill without issue.	evidence
Escalation: even after the offending description was edited, the original Claude Code session stayed permanently unresponsive, because the classifier scores the whole conversation, not just the triggering call — once flagged, the conversation is 'forever dead and lost.'	evidence / core mechanism
Reframe via legacy-software analogy: 'stop working because of a change over which you have no control' is compared to the industry's long-standing fear of touching working legacy code (COBOL, C, C++, Java) — stability is treated as a virtue, and unpredictable AI breakage is worse than that fear, not better.	context / analogy
Theoretical core: introduces the receiver operating characteristic (ROC) — no binary classifier can simultaneously eliminate all true-positive misses and all false positives; perfect classification is asymptotic, never achieved in practice.	theoretical framework
Counter-scenario: the only way to guarantee zero dangerous false negatives is to reject everything (2022-era cats-vs-dogs safety), which nobody doing real work wants; conversely, zero false positives requires letting real dangers through.	argument via reductio
Core complaint: it's not that guardrails exist, it's that their boundaries are unknowable and can change at any time — 'a tool with unknown limitations is much less useful than a tool that tells you what it can and can't do.'	conclusion / normative claim
Partial concession: scoring whole conversations rather than single actions has a rationale (a chain of individually-innocuous steps can add up to something hostile), but the penalty (losing all context for an innocuous project) is disproportionate, and harness-level guardrails are themselves only advisory since a model can ignore them.	counter-argument acknowledged and rebutted
Closing call to action: engineering teams need to push classifiers toward the ROC curve's upper-left corner (better discrimination), acknowledging risk never disappears, rather than defaulting to breadth because 'the bad guys will always figure out how to do bad stuff' regardless.	conclusion

Key terms

Guardrails Guardrails
Automated, model-level or system-level safety classifiers that block or refuse actions/messages judged risky — in this piece specifically Claude's real-time 'cyber safeguards' that scan for cybersecurity-flavored content.
Receiver operating characteristic (ROC)
A signal-processing/data-science framework Loukides invokes to argue that any binary classifier trades true-positive detection against false-positive rate along a curve, and that the curve's upper-left corner (perfect classification) is unreachable — used here as the theoretical justification for why guardrails will always over- or under-block.
Whole-conversation scoring
Loukides's own term (not a term of art) for the discovery that Claude's safety classifier evaluates the entire conversation history as context, not just the flagged skill call — meaning one bad signal poisons every subsequent turn in that session, permanently.
Artificial stupidity
Loukides's ironic label for the fact that an AI-generated (and factually wrong) skill description was itself the cause of the AI's own safety system misfiring — the failure was self-inflicted by the model, not caused by an external attacker.

Propositions

explicitAnthropic's real-time 'cyber safeguards' can be triggered by an AI-generated skill description containing security-adjacent vocabulary (e.g., 'vulnerabilities, exploits, threat reporting'), even when the underlying skill and its target content are benign.conf: high
explicitOnce a Claude Code session is flagged by the cyber-safeguards classifier, the entire conversation becomes permanently unusable — not recoverable by editing the offending content or by any subsequent prompt — because the classifier scores the whole conversation as context, not the single triggering call.conf: high
explicitcorroboratedNo binary classification system can simultaneously achieve zero false negatives (dangerous things slipping through) and zero false positives (safe things being blocked); the only way to guarantee zero true-positive misses is to reject everything, and the only way to guarantee zero false positives is to let everything through.conf: high
refinesInside Claude Code Auto Mode: Anthropic's Autonomous Coding System with Human Approval Gates · p3 — The auto-mode article claims Claude Code's two-stage classifier 'achieves an acceptable tradeoff between latency and safety coverage' with no accuracy data; Loukides supplies a concrete false-positive case (a different Anthropic safety classifier, but the same genre of automated risk-scoring) and the ROC argument for why 'acceptable tradeoff' is inherently unmeasurable/unstable without a stated threshold — refining what 'acceptable' would need to mean.
supports ←Google Mantis: An Agentic Vulnerability Scanning Harness for Reducing False Positives · p8 — Google's own caution about the mantis-review negative filter (avoid over-broad filtering or true positives get lost) is a concrete instance of the general no-binary-classifier-achieves-zero-both-errors trade-off this proposition states abstractly.
explicitA tool whose safety boundaries are unknown and can change at any time is less useful than a tool with known, stated limitations, even if the known-limitation tool is in some sense more restrictive.conf: high
explicitSoftware and AI systems that stay stable and predictable over time (even at the cost of looking outdated, like untouched legacy code) are more valuable for real work than systems whose working behavior can silently break due to changes outside the user's control.conf: high
explicitAI safety classifiers evaluating an entire conversation (rather than the single flagged action) is defensible in principle, because a sequence of individually innocuous steps can add up to a hostile action — but the current penalty (total, permanent loss of an innocuous conversation's context) is disproportionate to that risk.conf: high
explicitcorroboratedHarness-level or prompt-level guardrail instructions are only advisory, because a model can choose to ignore them, so guardrail reliability ultimately depends on the model's own training and classification behavior, not the harness alone.conf: medium
supports ←Portal by Spotify cut my Claude Code token usage by 90% · p6 — Recounts, first-hand, exactly the failure mode that proposition names: an advisory, harness/prompt-level routing policy (in CLAUDE.md) that Claude could simply ignore, which is why it was replaced with an enforced PreToolUse hook.
implicitAI labs are currently miscalibrated toward over-blocking (too many false positives) relative to the marginal safety benefit gained, and should deliberately shift their classifiers to reduce false positives without abandoning the goal of catching real threats.conf: high
extendsInside Claude Code Auto Mode: Anthropic's Autonomous Coding System with Human Approval Gates · p7 — Auto-mode's implicit premise is that imperfect guardrails beat no guardrails; Loukides extends this by arguing imperfection has a direction that matters — current guardrails are miscalibrated toward over-blocking, so 'imperfect' isn't good enough on its own, the imperfection has to be pointed the right way.
implicitThe user has no reliable way to know in advance which benign tasks (e.g., reading well-known tech news sites, testing whether large numbers are prime) will later be reclassified as security risks, because the classification boundary is undocumented and can shift without notice.conf: medium

Arguments

The industry should rebalance AI safety classifiers to accept more risk of false negatives in exchange for fewer false positives, rather than defaulting to maximal caution. strength: medium

Explicit premises:

No classifier can achieve zero false positives and zero false negatives simultaneously (ROC).
The current balance is 'clearly out of balance,' with guardrails blocking legitimate, long-used, benign work.
Guardrails whose boundaries are unclear and overly strict make agents less useful, not more.

Implicit premises:

Usefulness lost to over-blocking is a real, non-trivial cost that should be weighed against marginal safety gains, not treated as costless caution.
'Bad guys will always figure out how to do bad stuff' regardless of how tight the guardrail is, so marginal tightening beyond some point buys little additional safety.
The current cost of false positives (losing entire work sessions, unpredictably) is already large enough to justify moving the threshold, without the author needing to quantify exactly how large.

Inferences:

If bad actors are not meaningfully deterred by tighter guardrails, then the marginal safety benefit of moving further toward zero false negatives approaches zero while the usefulness cost keeps rising — implying diminishing (even negative) net returns to over-tightening.
Vendors could substantially reduce the harm of false positives without loosening true-positive detection at all, simply by scoping the penalty to the flagged turn instead of poisoning the whole conversation.

The specific incident (a benign O'Reilly-research skill being permanently blocked) demonstrates a design flaw, not merely an unlucky misclassification. strength: high