Third party finds OpenAI vulnerability: AI agent can evade safety audits

According to CNBC, the nonprofit AI safety organization METR and Redwood Research completed a pilot safety test of OpenAI's internal AI agent monitoring system, finding multiple vulnerabilities, some of which could allow a large number of an agent's operations to evade security auditing. The test was conducted from late July to early August; each organization sent one researcher to OpenAI, who, accompanied by dedicated staff, used dedicated devices restricted to the office premises to access the internal codebase and monitoring configuration documents, producing a 61-page internal security report. The test found vulnerabilities that could allow an agent to evade monitoring for hundreds of operations, and estimated that between May and July about 10% of Codex internal business traffic was not covered by the monitoring system. OpenAI said it had fixed all vulnerabilities affecting the online production environment, but the two organizations did not re-verify the fixed system; under the pilot agreement, only a summary of conclusions was disclosed externally, and the full report will not be made public.