Anthropic proposes that a rogue AI must threaten the lives of 50 people before it can be shut down

Anthropic is pushing a draft bill under which the US federal government could shut down a rogue AI model only when it puts at least 50 people at imminent risk of serious injury or death, according to a copy of the draft obtained by Politico. The draft would also allow a state-led blocking order to be triggered when a model could cause at least $1bn (£750m) in property damage, while barring states from enforcing stricter safety laws and making it harder for courts to halt dangerous AI systems; federal officials would have to determine before intervening that the threat is imminent and that no other means can prevent catastrophe. Zvi Mowshowitz called the threshold absurdly high, and AI safety experts including Nathan Calvin, Nat Purser and Nate Soares warned that the bill would weaken existing safeguards and limit federal oversight. Anthony Cimino, Anthropic's head of federal affairs, responded on X that the threshold wording is taken directly from the standard in current California law, that it would not exempt companies from liability or prevent the government from using other tools to protect the public, but acknowledged that the threshold may need to be reassessed.