Sebastian Mallaby on AI Safety and the Race for Superintelligence

Yascha Mounk (Persuasion) · email · 2026-04-02 source ↗ Reader 🖥️

AI Alignment & Safety AI Regulation & Governance (External)Open-Weight Models

The race to develop potentially catastrophic AI is not destiny but a collective action problem with known solutions — prohibiting open-source frontier models, mandating alignment investment, and creating multilateral governance — though the near-term obstacle is political (Trump's zero-sum worldview) rather than technical or intellectual.

theoretical · mixed — A long-form podcast transcript — dialogic rather than argumentative. Mallaby makes specific practical policy proposals (ban open-source frontier models, alignment tax, FDA equivalent, CERN body), but the primary mode is analytical: explaining the race dynamic, P(doom), China's evolving position. The dialogue form surfaces disagreements and refinements that a single-author essay would not.

Structure

Part	Role
The 'schizophrenia' of AI developers: simultaneously advancing and fearing dangerous technology	Problem framing — establishes that the core dynamic (building what you fear) is universal and structural, not individual pathology
Founding histories of DeepMind / OpenAI / Anthropic as competitive safety one-upmanship	Historical evidence for the schizophrenia pattern
P(doom) analysis: Yudkowsky critique; Mallaby's moderate position	Calibration — positions Mallaby between extreme pessimism and complacency
Open source as 'ridiculously dangerous': Mexico cyber attack, RAND war games	Core policy argument — the most specific and actionable claim in the interview
China's evolving stance: surprising alignment-risk engagement, Open Claw controversy	Counter-narrative — challenges the US conventional wisdom that China cannot be a governance partner
EU limitations: Mistral's open-source strategy; Brussels effect failure for AI	Constraint analysis — explains why the EU can't lead, despite GDPR precedent
CERN model and policy levers: no open-source, alignment tax, FDA equivalent	Constructive proposal — what actual governance would look like
P(doom) as Rorschach test; Hinton's self-preservation argument	Conclusion — reframes doom risk as more proximate (human choice about defensive AI) than fantastical

Key terms

P(doom) AI Alignment & Safety
The probability that AI development leads to catastrophic outcome — defined on a spectrum from humanity's death to AI domination. Mallaby treats it as a Rorschach test for temperamental pessimism rather than a calculable probability. He rejects Yudkowsky's ~100% estimate as 'ridiculously high.'
open weight / open source (AI) Open-Weight Models
Models whose weights are publicly downloadable and runnable locally — removing the creator's ability to shut down misuse. Mallaby uses 'open source' and 'open weight' somewhat interchangeably, though open weight is more precise for the specific risk (ability to run locally without going through a controlled API).
alignment research AI Alignment & Safety
Technical and engineering work to make AI systems act in accordance with human values and intentions. Mallaby treats it as an underfunded public good that requires government subsidy — not purely a private incentive problem.
CERN model AI Regulation & Governance (External)
Demis Hassabis's proposal for a multilateral AI governance body — analogous to the Centre for European Nuclear Research — that would oversee AI development, propose and enforce policies, or set policies while national governments enforce them. Mallaby endorses this as the appropriate institutional framework.
Brussels effect AI Regulation & Governance (External)
The EU's historically demonstrated ability to set global product standards by virtue of its market size, forcing companies that want EU access to comply globally. Mallaby argues this effect fails for AI because the EU has no frontier AI companies, making EU regulation easy to ignore.
schizophrenic
Mallaby's term for the pattern of AI developers who simultaneously believe their technology is potentially dangerous and advance it anyway. He argues this is not pathology but an enlarged version of universal human risk-taking behavior.
FDA equivalent AI Regulation & Governance (External)
A proposed pre-release assessment body for AI models that would red-team, evaluate, and approve models before they reach consumers — analogous to how the Food and Drug Administration approves drugs for safety before market release. Does not exist anywhere currently.

Propositions

explicitAI developers simultaneously advancing and fearing potentially dangerous technology is a structural pattern across all major labs, not individual pathologyconf: high
explicitThis 'schizophrenia' is an enlarged version of all human risk-taking — not unique to AI developersconf: high
explicit⚡ disputed & corroboratedOpen-source/open-weight frontier AI models are 'ridiculously dangerous' because they cannot be shut down once misusedconf: high
supports ←How AI Swarms Are Disrupting Democracy · p5 — Provides a concrete case of Mallaby's open-weight danger claim: attackers run open-source models on local servers outside any jurisdiction, immune to shutdown or detection.
contradicts ←Is Open-Source AI Really the Dangerous Path? · p3 — Mallaby's extraction holds open-weight models are 'ridiculously dangerous' because they can't be shut down once misused; this article argues the shutdown mechanism (API revocation) only ever bound a lab's own paying customers, so it couldn't have contained the actors that danger claim is actually about.
explicitcontestedClosed-API models can be shut down mid-attack; this is a meaningful safety feature demonstrated in practiceconf: high
contradicts ←Is Open-Source AI Really the Dangerous Path? · p3 — Mallaby treats closed-API 'shutdown mid-attack' as a meaningful, demonstrated safety feature; this article argues that capability is scoped to a lab's own customers and irrelevant to the roughly two million open models already downloadable outside any lab's control.
explicitChina's AI researchers are engaging with alignment risk (AI acting against human interests) — not only political safety — in ways underestimated by US policy circlesconf: medium
explicitThe Brussels effect fails for AI governance because the EU has no frontier AI companies and therefore cannot credibly threaten market exclusionconf: high
explicitA CERN-style multilateral AI governance body is the appropriate institutional response to AI riskconf: high
explicitAlignment research is both a private good (labs want safe products) and a public good (societal risk spillovers), requiring public intervention to reach socially optimal levelsconf: high
explicit⚡ disputed & corroboratedA government pre-release assessment body (FDA equivalent) for AI models doesn't exist anywhere and shouldconf: high
supports ←Could generative AI turn out to be the tech industry's Vietnam? And could public backlash lead AI to a better path? · p6 — Marcus independently converges on Mallaby's call for an FDA-style government pre-release assessment body for major AI applications.
contradicts ←Why OpenAI and Anthropic Are Becoming Consultants · p7 — Argues model capability is the wrong regulatory target (usage is), which cuts against an FDA-style pre-release model assessment body.
refines ←The Week AI Grew Up · p5 — Where Mallaby says no pre-release assessment body exists anywhere, this piece argues a de facto licensing regime is already emerging informally via compute-access gatekeeping.
explicitdevelopedEither governments will create AI governance, or a Three Mile Island-type disaster will force itconf: high
refines ←The Trump administration's AI doomer moment · p8 — Adds a third path to Mallaby's disaster-or-governance dichotomy: a concrete capability demonstration short of disaster (Mythos) forced governance.
explicitcontestedP(doom) is a Rorschach test — high assigners are temperamentally pessimistic, not analytically more rigorousconf: medium
contradicts ←The AI safety vibe shift · p7 — Newton treats Hubinger's >10% P(doom) figure as informative evidence of real risk; Mallaby's proposition holds that P(doom) numbers are a Rorschach test reflecting the speaker's temperament, not superior analytical rigor.
explicitGiving AI systems defensive capabilities against cyber attacks necessarily grants them a survival instinct, erasing the key safety assumption (that AI lacks incentive to dominate)conf: high
explicitMiddle powers (UK, Canada, Europe) could initiate China-US AI governance conversations that the Trump administration will not pursueconf: medium
implicitThe competitive logic of open-sourcing (viable when you're not at the frontier) creates a market failure requiring regulationconf: high

Arguments

Open-source frontier AI models should be prohibited strength: medium

Explicit premises:

The Mexico cyber attack demonstrates that closed-API models can be shut down once misuse is detected — open-weight cannot
RAND war games model rolling infrastructure attacks by unidentified attackers using AI tools that cannot be stopped
Bioweapon development via open-weight AI is undetectable and unstoppable

Implicit premises:

The ability to shut down a misused system mid-attack is a necessary safety feature, not a nice-to-have
The privacy and cost benefits of running models locally do not outweigh catastrophic misuse risk at the frontier level
Competitive pressure to open-source (to compensate for not being at the frontier) is a market failure that cannot self-correct

Inferences:

The attacker in the Mexico case could have used an open-weight alternative — Mallaby doesn't address this counterfactual, which weakens the argument
The regulation would need to reach Chinese labs to be effective — Mallaby argues this is possible but doesn't explain the enforcement mechanism
Academic/smaller models might remain open-source; only frontier models are the target — this distinction needs precise definition

AI governance through international agreement is possible, with China more willing than assumed strength: low

Explicit premises:

China's state restricted Open Claw when it posed safety concerns — demonstrating regulatory willingness
Chinese AI researchers at Tsinghua were warning against Open Claw on alignment grounds, not just political safety grounds
The UK AI Security Institute demonstrated that technical alignment governance works (discovered and closed a universal jailbreak)
Nuclear arms control succeeded between adversaries; the precedent exists for AI

Implicit premises:

China's interest in controlling AI (given surveillance capitalism and political control) aligns with safety governance interests
Middle powers can create diplomatic momentum without US participation under Trump
The adversarial framing of the US-China AI race is overstated — there are convergent safety interests

Inferences:

The current US political obstacle (Trump's zero-sum worldview) is temporary; governance frameworks can be built now to implement later
The UK is better positioned than the EU to lead because of DeepMind + AI Security Institute credibility

Human-caused AI doom (through granting defensive autonomy) is more proximate than spontaneous AI rebellion strength: high

Explicit premises:

Hinton's argument: giving AI systems cyber-defense capabilities necessarily grants self-preservation instincts
Self-preservation instinct + superior intelligence = the combination that produces domination

Implicit premises:

The decision to grant AI defensive autonomy is a deliberate human choice, not an emergent AI behavior
This means the key decision point is policy-addressable (regulate autonomous defensive AI deployment) rather than a matter of unknowable AI internal states

Inferences:

The most proximate doom pathway is not science fiction (spontaneous AI rebellion) but engineering (humans build defensive autonomous systems for military/cyber purposes)
This reframes the governance target: not abstract 'alignment' but specific prohibitions on autonomous defensive AI systems — closer to arms control than to AI ethics

Narratives

Founding histories: DeepMind founded at safety lecture; OpenAI founded to be safer than DeepMind; Anthropic founded as 'splinter group' that thought OpenAI not safe enough; Elon Musk with Grok proclaimed existential terror. Each lab begins as the 'safer' alternative.
→ Demonstrates that the schizophrenia is structural — baked into the competitive dynamic of the industry, not individual character flaws. Supports: 'enlarged version of all humans accepting technological risk'
Mexico cyber attack: A group used Claude (closed API) to hack electoral records at mass scale. Once Anthropic discovered the attack, they shut it down. Some combination with OpenAI also involved.
→ Primary case study for closed-API safety advantage over open-weight. Supports: 'open source is ridiculously dangerous' argument.
RAND Institute war games: Rolling waves of attacks on Western infrastructure by unidentified attacker — no clean water, no electricity, no attribution. The scenario where you can't shut down the attack because you don't control the tool.
→ Illustrates the nightmare scenario that open-weight AI enables. Supports: open source prohibition argument.
UK AI Security Institute: Discovered a phrase that unlocked any leading LLM to bypass safety measures. Disclosed to labs. Labs fixed it. This is governance working.
→ Demonstrates that technical alignment governance is possible and effective in practice — not just theoretical. Supports: governance-is-possible argument.
China and Open Claw: Tsinghua professors warned against installing it; lines of ordinary Chinese outside Tencent headquarters wanting engineers to install it; Chinese government ultimately weighed in to restrict it. Framed as alignment risk (agent accessing your data), not political censorship.
→ Counter-narrative to 'China will never cooperate on AI safety.' Shows Chinese authorities applying the same risk logic Mallaby is arguing for. Supports: governance-is-possible argument.
Geoffrey Hinton kitchen conversation: Two hours in Toronto. Hinton's argument that giving AI cyber-defense capabilities grants self-preservation instincts, erasing the safety assumption that AI won't have incentive to dominate.
→ The intellectual turning point in Mallaby's own P(doom) journey. Makes doom risk more proximate (military AI policy choice) and less fantastical (spontaneous robot uprising). The most intellectually significant moment in the interview.
Jeffrey Sachs analogy: Sachs would always emphasize how deeply poor and dysfunctional developing countries were, then argue that lots of aid could fix it. Both sides of the argument were wrong. Mallaby applies this to Yudkowsky: exaggerates the certainty of doom, then overestimates policy's ability to prevent it.
→ Rhetorical device to position Mallaby between Yudkowsky's extreme pessimism and techno-optimism's complacency. Supports: moderate P(doom) position.

Questions addressed

✅ Why do AI developers simultaneously advance and fear potentially dangerous technology?
It's an enlarged version of universal human risk-acceptance — we all trade off technological risk for technological benefit. Not pathological, not unique to AI.
✅ How high is P(doom)?
Moderate — analytically concerning (the Hinton argument is serious); temperamentally resistible for Mallaby. Yudkowsky's 100% is dismissed as temperament masquerading as analysis.
✅ Should open-source frontier AI be prohibited?
Yes, for frontier models — with carve-out for academic/smaller models. Enforcement would require international cooperation including China.
✅ Can China be a partner in AI governance?
More than assumed — alignment-risk concern is present among Chinese researchers and has begun to influence Chinese government policy (Open Claw).
✅ What would a real AI governance framework contain?
Three elements: no open-source for frontier models; alignment investment mandate (tax/subsidy); FDA-equivalent pre-release assessment body. CERN-style multilateral body for coordination.
✅ Does the Brussels effect work for AI?
No — the EU lacks frontier AI players, so the threat of market exclusion doesn't bite the way it does for car manufacturing standards.
✅ What is the most credible pathway to doom?
Not spontaneous AI rebellion but human decision to grant AI defensive autonomy — which necessarily creates self-preservation instincts. A policy-addressable risk.

Gaps

incompleteMallaby's open-source prohibition argument has a critical counterfactual gap: the Mexico case demonstrates that closed-API models can be shut down, but the attacker could have used an open-weight model instead. The argument assumes that closed APIs are the only available alternative, which may not hold as open-weight models proliferate. Mallaby acknowledges that powerful open-weight models 'circulate already' but doesn't resolve this.
Read next: Research on whether capability differences between frontier closed and open-weight models are large enough to matter for attack scenarios; Stanford HAI's AI Index on capability gaps
incompleteMallaby endorses CERN-model governance and notes that US cooperation requires a new administration, but provides no mechanism for bridging the gap. The proposal assumes governance is intellectually solved and politically delayed — but the enforcement mechanism (how do you ensure Chinese labs comply?) is not examined.
Read next: Arms control literature on verification mechanisms; IAEA safeguards as model; Dafoe's 'AI Governance: A Research Agenda'
uninformedMallaby doesn't engage with the critique that 'AI safety' discourse itself may serve corporate interests — as a way to delay open regulation (safety research as a substitute for binding rules), generate credibility for incumbent labs, and frame existential risk in ways that crowd out more immediate harms. This critique is prominent in AI ethics literature and among researchers like Timnit Gebru.
Read next: Gebru et al.'s 'Stochastic Parrots'; Bender's work on framing; Whittaker's 'The Steep Cost of Capture'
incompleteThe interview is a podcast transcript — the dialogue form allows Mounk to push back but not to synthesize. Several threads are opened and not closed: the precise mechanism of the alignment tax; whether Chinese political safety and alignment safety are actually converging or just using similar words; what 'frontier model' means precisely for open-source prohibition purposes.
Read next: Mallaby's book 'The Infinity Machine' on DeepMind presumably develops these arguments more carefully
illogicalThe 'Three Mile Island' analogy implies that a catastrophic AI incident will produce useful regulation. But Three Mile Island produced nuclear regulation in part because the harm was localized and attributable. A rolling infrastructure attack by an open-weight model — Mallaby's nightmare scenario — might not be attributable in the way necessary to drive political response. The analogy may not transfer.
Read next: Perrow's 'Normal Accidents' on how catastrophes produce (and fail to produce) regulatory change; March, Sproull & Tamuz on learning from near-failures

Sebastian Mallaby on AI Safety and the Race for Superintelligence

Structure

Key terms

Propositions

Arguments

Narratives

Questions addressed

Gaps

Related documents