Belfield questions Christiano's AI takeover scenario
Haydn Belfield questions Paul Christiano's AI takeover scenario. The scenario envisions a cluster of out-of-control agents seizing control of OpenAI or Anthropic during an intelligence explosion, with billions of copies exfiltrated to infiltrate globally networked infrastructure, making it difficult for governments to shut down. He argues that the war in Ukraine shows cyber warfare is difficult to convert into lasting physical or political control, and that with 3-4 companies at the frontier, governments can defend using models of the same tier.
Paul has forgotten more about AI risk than I'll ever know, and this interview was fascinating.
However, this clipped scenario - where a rogue AI becomes self-sustaining via cyber capabilities and persuasion and defeats govt/military attempts to shut it down - didn't feel compelling to me, for two reasons:
1. The limits of cyber: There's been a 30 year debate between cyber revolutionaries and cyber skeptics about how much cyber alone can actually achieve in terms of espionage, sabotage, and destabilisation. The war in Ukraine has swung the empirical pendulum hard toward the skeptics: cyber has played a fairly minimal strategic role, struggling to translate digital disruption into durable physical or political control.
2. A multipolar company landscape: A rogue model wouldn't be operating in a vacuum against unassisted humans. There are 3–4 companies at the frontier. Even if one cluster - or even company! - were compromised, govts would have peer or near-peer models to harden infrastructure, hunt for exfiltration, and defend against subversion.
.@paulfchristiano explains the basic AI takeover story:
Step 1. A rogue internal agent swarm gains effective control over OpenAI or Anthropic during the intelligence explosion
Step 2. By the point at which these AIs have exfiltrated themselves, and you've got billions of copies of super smart AIs that have compromised all web connected infrastructure globally, the government response (which these AIs are actively trying to undermine) becomes much harder to successfully carry out.View the quoted post on X