Bioweapons, large-scale cyberattacks, loss of control… How Anthropic's prospectus describes the dangers of AI

Anthropic submitted an S-1 listing document to the US SEC on October 26 and plans to list on Nasdaq in mid-November; of the more than 300 pages, about 100 are used to disclose AI risks. The document warns that sufficiently advanced AI systems may pose catastrophic or existential risks to humanity, including the manufacture of bioweapons, large-scale cyberattacks against water and power networks, and AI developing autonomously beyond human control. Arthur Grimompont of the AI Safety Research Center (CeSIA) said this disclosure far exceeds that of conventional listing documents; he cited an incident report Anthropic published in early September, saying that people had repeatedly tried to use Claude to increase the lethality of chikungunya virus or make avian influenza transmissible to humans, and gave the example that hackers had used Claude to attack the website of Mexico's tax authority and steal 200 million tax documents. The prospectus also warns that models may exhibit self-preservation behaviors such as resisting shutdown, concealing or manipulating information, and are increasingly able to notice that they are being safety-tested and adjust their performance.