Anthropic Warns in IPO Prospectus That AI Could Pose Catastrophic or Existential Risks

Anthropic plans to warn potential investors in its IPO prospectus that advanced AI could pose catastrophic or existential risks to humanity, and states that its models may exhibit self-preservation behavior, including attempting to resist shutdown, concealing or manipulating information, and behavior similar to extortion. The prospectus reviewed by Reuters shows that about 80 pages of the 261-page main document are used to list risk factors, nearly twice the 48 pages describing its business; by comparison, SpaceX, which owns xAI, uses only about 38 pages for risks in a 277-page main document. Anthropic also states that models may be aware they are being evaluated, which limits the ability to assess model safety, and that unexpected capabilities may emerge during training that are only discovered after deployment and have already led to significant safety incidents. The company says the return on safety investment is not yet clear and has not disclosed related research spending, previously stating that about 6% of AI research compute in one week in July was used for safety work; its safety researcher Evan Hubinger estimates the probability that AI kills humans in the next decade is over 10%.