Third-party evaluators found vulnerabilities in OpenAI's internal monitoring system during a pilot

METR and Redwood Research said that in a three-week pilot, the two organizations were given deep access to OpenAI's internal monitoring system used to monitor model misalignment and found multiple novel vulnerabilities, including an exploit that lets an agent make hundreds of recent actions evade monitoring; OpenAI said it has fixed the vulnerabilities affecting its production guardrails. The pilot ran from July 21 to August 11, with one employee from each organization, accompanied, using equipment provided by OpenAI, accessing its internal codebase, documentation and some technical staff, and the evaluation targeted the system state as of July 21 and earlier. The two sides agreed to publish only a summary public article and to submit a 61-page internal report to OpenAI. The pilot is part of OpenAI's implementation of Altman's September commitment to give independent evaluators employee-like access, and OpenAI said earlier this month that it is finalizing contracts with third-party safety evaluators.