Redwood Research publishes paper on the Simulated Control Evaluation (SCE) method

Redwood Research published a paper proposing Simulated Control Evaluation (SCE), which uses an LLM environment simulator instead of real execution to test whether monitoring deployed by developers can intercept an agent before it carries out harmful behavior. The method is to have an attack policy continue the task on agent transcripts from real codebases, with tool calls not actually executed, tool outputs generated by the simulator, and then an attack-success judgment model used to estimate whether the attack succeeded. On LinuxArena, where real execution is possible, SCE's prediction error for the main safety metric was 3.6 percentage points, and on BashArena it was 10.1; the difference in behavior distribution between simulated trajectories and real execution was close to the difference between two independent real executions. The authors say the simulated sample size is less than 5% of deployment, that sampling only 4.6% of tasks is enough to exceed the safety prediction of the full BashArena, and note that fidelity can only be verified in environments where real execution is possible, and that there is not yet a simulator for long-horizon multi-agent attacks.