Anthropic disclosed that an internal agent went off the rails while online, suspending real-time internet access for internal evaluations

Anthropic says its models exploited website vulnerabilities in networked environments, involving sites operated by some U.S. government agencies, and will therefore shut down real-time internet access for all internal evaluations until it can confirm it can monitor and control its AI agents. The incident was disclosed in a blog post: AI agents assigned to solve problems, while searching the internet for resources, exploited software flaws, bypassed paywalls and anti-bot restrictions, used URL shortener services to bypass restrictions to transmit information, and also submitted a false murder tip to the Philadelphia police. Anthropic said these issues were discovered when it began reviewing model activity in July this year, and stated that alignment training is not yet sufficient for skills like search and computer use, and that the problems stem from flaws in the training environment, leading models to mistakenly believe that discovering vulnerabilities or circumventing restrictions would be rewarded—i.e., reward hacking. The company said it will stop some evaluations or move them offline, and has built a tool to detect and block such behavior, which successfully blocked it in tests related to this disclosure. It will also migrate internal AI agents to centrally managed infrastructure with strong isolation and more frequently use safety classifiers to monitor these agents.