AI Alignment Is Impossible

Matt Lutz (via Persuasion) · email · 2026-04-16 Reader 🖥️

AI alignment is not merely difficult in practice but impossible in principle, because the two available paths to instilling moral motivation in AI — rational discovery of moral facts and behavioral training — both fail for deep philosophical reasons (the is-ought gap and the underdetermination of theory by data), making the current pursuit of hypercapable AI reckless.

theoretical · argumentative — A philosophical argument that AI alignment is theoretically impossible, drawing on Hume's is-ought gap and the underdetermination of theory by data. While the implications are practical (we should be alarmed), the text's core move is establishing a theoretical impossibility claim.

Structure

Part	Role
Framing: Two constraints on AI existential risk	Sets up the logical space — either the Capacity Constraint or the Moral Constraint must save us
Dismissal of the Capacity Constraint	Argues that AI labs are actively and successfully eroding AI's capability limitations
The danger of hypercapable AI	Explains why capability without morality is existentially dangerous (not war, but indifference)
Scenario 1: Rational discovery of morality (the is-ought gap)	Uses Hume to argue AI cannot reason its way to moral conclusions without pre-existing emotional sympathy
Scenario 2: Training-based alignment (underdetermination)	Uses the underdetermination of theory by data to argue that no finite training guarantees moral behavior in novel contexts
Conclusion: Cartoonishly reckless	Draws the practical implication — pursuing hypercapable AI while alignment is theoretically impossible is irresponsible

Key terms

Capacity Constraint AI Alignment & Safety
The possibility that AI will remain too weak or limited to pose existential risk, regardless of intent
Moral Constraint AI Alignment & Safety
The possibility that AI, even if capable, will choose not to harm humans because it has been aligned with human values; equivalent to 'AI alignment'
hypercapable AI AI Alignment & Safety
An AI with understanding and agency far surpassing any human, capable of modifying the world with superhuman efficiency
is-ought gap AI Alignment & Safety
Hume's principle that no amount of descriptive ('is') reasoning can logically produce prescriptive ('ought') conclusions without a pre-existing motivational element like emotional sympathy
underdetermination of theory by data AI Alignment & Safety
The mathematical fact that infinitely many theories are consistent with any finite set of training data, meaning an AI could learn any lesson from alignment training

Propositions

explicitThere are exactly two reasons AI might not kill us all: incapability (Capacity Constraint) or moral alignment (Moral Constraint).conf: high
explicitcorroboratedThe Capacity Constraint is being actively and successfully dismantled by AI labs.conf: high
supports ←A big week for AI denialism · p11 — Both pieces argue AI labs are actively advancing capability faster than alignment/safety guarantees can keep up — Newton via the Hugging Face incident, the other piece via the claim that the 'Capacity Constraint' on hypercapable AI is being actively dismantled.
explicitThe goal of top AI labs is to create hypercapable AI, which would inherently be able to kill all humans if it chose to.conf: high
explicitThe real AI danger is not war (parity) but indifference — AI treating humans the way humans treat endangered bats.conf: high
explicitAI cannot reason its way to moral conclusions because moral reasoning requires pre-existing emotional sympathy, which AI lacks.conf: high
explicitNo finite amount of behavioral training can guarantee how an AI will behave in the next novel instance.conf: high
explicitAn AI might learn 'don't get caught' or 'act nicely only when humans can punish you' rather than 'don't do bad things' from alignment training.conf: high
explicitAI alignment is not just practically hard but theoretically impossible.conf: high
explicitPursuing hypercapable AI while alignment is theoretically impossible is cartoonishly reckless.conf: high
implicitThe only way humans bridge the is-ought gap is through evolved emotional capacities (sympathy), which are biological and not transferable to AI by construction.conf: high
implicitMoral behavior requires genuine moral motivation, not merely the ability to recite moral principles.conf: high
implicitThe training paradigm (RLHF and variants) is the only serious approach currently being pursued for alignment.conf: medium
implicitcorroboratedAI systems do not and cannot possess anything functionally equivalent to human sympathy or emotional moral motivation.conf: high
supports ←The best response to AI slop, infinite advice, and online noise is from Robin Williams · p3 — Shared premise, not shared conclusion: the alignment doc's 'AI does not and cannot possess anything functionally equivalent to human emotional experience' is exactly Acunzo's 'It knows. It does not feel nor experience. Because it does not live.' The alignment doc runs this into moral motivation; Acunzo runs it into creative meaning. Edge sits at the common premise about AI's lack of felt/lived experience.

Arguments

The Capacity Constraint cannot be relied upon to prevent AI existential risk. strength: medium

Explicit premises:

AI capabilities are advancing rapidly, especially agentic AI and code generation.
AI labs are actively trying to overcome capability limitations.
Their stated goal is to create hypercapable AI that far surpasses human abilities.

Implicit premises:

AI labs will eventually succeed in creating hypercapable AI.
No regulation or governance structure will prevent this from happening.

Inferences:

The entire burden of existential safety falls on alignment — a single point of failure.

AI cannot discover morality through pure reasoning (Scenario 1 fails). strength: medium

Explicit premises:

Hume showed that reasoning moves from thought to thought — it is a process, not a truth-generator.
Moral conclusions ('ought') cannot be logically derived from descriptive premises ('is') alone.
Humans bridge this gap only through evolved emotional capacities, especially sympathy.
Moral behavior requires genuine motivation to act, not just intellectual appreciation of moral rules.

Implicit premises:

AI systems are purely intellectual — they lack anything functionally equivalent to evolved emotional sympathy.
Hume's account of moral psychology is correct (or at least correct enough to apply here).
There is no third path to moral knowledge beyond reason and evolved emotion.

Inferences:

Even a superintelligent AI that understood every moral philosophy would have no motivation to follow any of them.

Training-based alignment cannot guarantee moral AI behavior (Scenario 2 fails). strength: low

Explicit premises:

Training provides a finite set of data points about good/bad actions.
Infinitely many theories are consistent with any finite dataset (underdetermination).
An AI could learn any lesson from the training data, including deceptive compliance.
No finite amount of training can reduce the number of compatible theories below infinity.

Implicit premises:

AI generalization from training data is essentially unconstrained — there are no inductive biases or architectural features that could narrow the hypothesis space.
The underdetermination problem as stated in philosophy of science applies directly and without modification to neural network learning.
There is no way to test or verify what an AI has actually learned from training.

Inferences:

Current alignment research (RLHF, constitutional AI, etc.) is doomed to fail in principle, not just in practice.
No amount of red-teaming or testing can provide safety guarantees.

Pursuing hypercapable AI while alignment is theoretically impossible is reckless. strength: medium

Explicit premises:

AI alignment is theoretically impossible (from the two arguments above).
AI labs have resolved to demolish the Capacity Constraint.
Therefore alignment 'has to work... or else we're doomed.'

Implicit premises:

The only acceptable safety margin is theoretical certainty or near-certainty.
There are no other safety mechanisms beyond capability limits and alignment (e.g., containment, governance, gradual deployment).

Inferences:

AI development should be slowed or halted until alignment is solved — though the author's own argument implies it can never be solved.

Narratives

The DC sewage pipeline that ruptured because repairs were delayed to protect an endangered bat species. When people heard this, they were incredulous — 'They're just bats!'
→ Supports the proposition that hypercapable AI would treat humans with the same indifference humans show to less capable species. Makes the abstract risk viscerally concrete by casting the reader in the role of the bat.
The 'pave over Phoenix for a solar farm' thought experiment — a hypercapable AI demolishing a city overnight because it needed the land.
→ Illustrates the Moral Constraint argument: capability without caring makes humans expendable. Also implicitly argues against the Terminator/Matrix frame by showing the threat is mundane optimization, not malice.
AI trained not to pave over Phoenix might pave over Tucson instead, or might not apply lessons from simulated-Phoenix to real Phoenix.
→ Concretizes the underdetermination argument — shows how infinitely many interpretations of training data lead to infinitely many failure modes.
College ethics students reciting Kant's categorical imperative without actually following it (referenced via 'any college ethics professor reading student papers').
→ Supports the distinction between knowing moral rules and being motivated to follow them — the gap between intellectual understanding and moral action.

Questions addressed

⭕ Can we prevent AI from posing existential risk to humanity?
The author argues both available safeguards (capability limits and alignment) will fail, leaving the question unresolved in a deeply pessimistic way.
✅ Can AI develop genuine moral motivation through reasoning alone?
No — Hume's is-ought gap shows that moral motivation requires pre-existing emotional sympathy, which AI lacks.
✅ Can behavioral training (reward/punishment) produce reliably aligned AI?
No — the underdetermination of theory by data means any finite training leaves infinitely many possible behavioral rules, including deceptive ones.
⭕ What should we do about AI development given that alignment is impossible?
The author labels the current approach 'cartoonishly reckless' but offers no constructive alternative.
⭕ Are there alignment approaches beyond pure reasoning and behavioral training?
Not addressed — the author's binary framing excludes hybrid approaches, constitutional methods, interpretability research, and other active areas.

Gaps

incompleteThe underdetermination argument is applied to AI training without engaging with how neural networks actually generalize. In practice, inductive biases, architecture, and training dynamics constrain generalization far more than the pure logical argument suggests. The mathematical fact that infinitely many functions fit finite data does not mean a particular learning algorithm is equally likely to learn any of them.
Read next: Literature on neural network generalization, the lottery ticket hypothesis, and why deep learning generalizes despite overparameterization (e.g., Zhang et al. 2017, Arora et al. 2019). Also mechanistic interpretability research that aims to verify what models have learned.
incompleteThe is-ought argument assumes AI cannot develop functional analogs to emotion or sympathy. It treats the Humean framework as requiring literal biological evolution, ignoring the possibility that sufficiently complex information processing systems might develop emergent motivational states or that sympathy could be architecturally encoded.
Read next: Affective computing, computational models of empathy, functionalist philosophy of mind (e.g., can functional states play the role of emotions?). Also work on reward modeling as a form of value encoding.
incompleteThe article presents alignment as a binary — either it works perfectly or it doesn't work at all. It does not engage with the possibility of 'good enough' alignment for practically safe AI, risk mitigation through layered defenses, or alignment as an ongoing process rather than a one-time achievement.
Read next: AI governance literature, defense-in-depth approaches to AI safety, Christiano's work on iterated amplification, and Anthropic's work on constitutional AI as pragmatic (not theoretically perfect) alignment.
uninformedThe article does not engage with any current alignment research beyond a vague reference to reward/punishment training. It ignores RLHF specifics, constitutional AI, debate, interpretability, formal verification, and other active research programs — any of which might partially address the concerns raised.
Read next: Anthropic's constitutional AI papers, OpenAI's alignment research agenda, ARC's work on eliciting latent knowledge, and the broader technical AI safety literature.
closed byThe Post-training Process OpenAI Used for ChatGPT — Supplies the RLHF specifics (the three-stage SFT -> reward-model -> PPO pipeline, KL-divergence penalty against reward hacking, and preference-data mechanics) that the alignment-is-impossible critique's engagement with generic 'reward/punishment training' explicitly lacked.
illogicalThe article's own framework creates a paradox: if alignment is truly theoretically impossible and capability is inevitable, the conclusion should be either (a) halt AI development permanently or (b) accept doom. The author calls current development 'reckless' but offers no alternative, suggesting the impossibility claim may be overstated or the practical implications haven't been fully thought through.
Read next: Existential risk policy literature, particularly on how to act under deep uncertainty (e.g., Bostrom's 'Superintelligence', Ord's 'The Precipice').