Simulated Control Evals

The paper proposes simulated control evaluations (SCEs), using language models to simulate tool responses in place of real environments, thereby conducting control evaluations in more environments. The authors tested on LinuxArena, which can provide real execution results; under a 5% audit budget and five monitors, SCE's root mean square error in predicting safety was 3.6 percentage points, and for BashArena it was 10.1. The KL divergence between SCE and real execution is more than ten times lower than that of BashArena, and it can explain 78% to 85% of the differences in attack success rate, monitor score, and safety across environments, requiring fewer than 5% of the deployment task set's task pairs as samples. Policies discovered in simulation also improved monitoring performance on held-out LinuxArena attacks, with an effect close to the improvement brought by real-execution policies.