Simulated control evaluations (SCEs): using an LLM simulator to bring control evaluations closer to real deployment
Simulated Control Evaluations
The authors release a paper on simulated control evaluations (SCEs), proposing to use an LLM simulator to fabricate tool returns, testing without actually executing an attack whether monitors that developers use to supervise agents can detect models that deliberately insert vulnerabilities, steal credentials or sabotage experiments. The method starts from an agent's work trajectories in a real codebase, with gpt-6-astra simulating tool outputs (with gpt-5.6-sol as fallback), and gpt-5.6-sol simultaneously serving as the attack policy and the judge of attack success. The paper reports that, validated against real execution on LinuxArena, the gap between simulated and real is comparable to the gap between two independent real runs; under a 5% audit budget, the safety prediction RMSE is 3.6 percentage points, and for BashArena it is 10.1 percentage points; SCE explains 95% of the expected safety differences between side-task categories and 78% between environments, and sampling tasks at 4.6% of deployment scale already exceeds BashArena's full prediction.
The paper uses real execution on LinuxArena as a control to quantify the error and sample efficiency of simulated control evaluations, providing a reusable method for control evaluations that are closer to deployment.