External evaluators found flaws in OpenAI's AI monitoring system during a pilot evaluation

METR and Redwood Research found multiple new vulnerabilities in a pilot evaluation of the monitoring system OpenAI uses internally for embedded evaluations. The evaluation took place from July 21 to August 11, with one researcher from each organization using equipment provided by OpenAI at OpenAI's premises, with access to internal code, documentation and some technical staff. The vulnerabilities found included an exploit that agents could have used to let hundreds of actions go unmonitored, and a potential path for the Codex agent to bypass safety restrictions and act with the user's full permissions. OpenAI said these findings helped it strengthen protections as it scales the system, and said it has fixed all vulnerabilities affecting production protections, but the two organizations said they did not verify these fixes. The parties agreed to publish only a broad summary, with OpenAI having the right to redact sensitive information; the full 61-page internal report was provided only to OpenAI.