Anthropic uses real deployment resources to improve the realism of code auditing

Anthropic's alignment science team proposed the realism win rate metric, using an LLM judge in pairwise comparisons to determine whether an audit record or a real deployment record is more realistic, thereby measuring the realism of automated audits. The study provided the Petri audit agent with real system prompts, tool definitions, and codebase resources, raising the average realism win rate from 4.6% to 32.8% across 5 reward hacking audit scenarios, without significantly changing the reward hacking rate. On benign coding tasks, resources also steadily improved realism, but in high-risk scenarios involving shutdown resistance, rewriting the seed instruction improved realism more than providing resources, indicating that the task itself, rather than the environment, is the main bottleneck. This feature has been deployed in Petri.