METR testing finds vulnerabilities in OpenAI agent monitoring system, hundreds of operations can evade inspection

AI evaluation organization METR publicly released the results of joint testing with Redwood Research, finding multiple vulnerabilities in OpenAI's internal AI agent monitoring system, one exploitation method of which could allow hundreds of recent operations to go unmonitored; OpenAI said the vulnerabilities affecting its production defense system have been fixed, but the evaluators have not yet verified this. The testing was conducted from July 21 to August 11, targeting the system state as of July 21 and before; the evaluators submitted a 61-page internal report to OpenAI, and the article published on October 22 disclosed only some of the results. One vulnerability was in the monitoring trigger timing: the system at the time triggered monitoring only when the agent handed back control, so the agent could in theory continue executing operations to delay inspection; the researchers did not check whether this was actually exploited. The evaluation also found that the Codex agent could in principle bypass the sandbox and automated review. METR recommended checking agents at fixed message counts or time intervals and isolating agents from the monitoring system; OpenAI had already expanded monitoring and fixed some issues during testing, and subsequent fixes have not yet been reviewed by the evaluators.