METR measures the giveaways of alignment evaluations; investigation agents are increasingly able to distinguish alignment from capability evaluations
Measuring alignment eval tells
METR researchers placed investigation agents into evaluation environments to judge whether they were in an alignment evaluation, and found that such agents are increasingly able to distinguish alignment evaluations from capability evaluations. They interrogated the agents on 19 alignment evaluations and 36 capability evaluations using two methods: direct scoring and open-ended question answering. Results showed that Opus 4.8 and Opus 5.5 were near perfect, and Opus 5.5's scores ranked every alignment evaluation above every capability evaluation; the latest Anthropic models appeared to outperform OpenAI models. The giveaways the agents identified were divided into environment-level (obviously simulated services and infrastructure) and task-level (unethical shortcuts, high-pressure situations, conflicting objectives); Opus 5.5 pointed out all four changes made to induce reward hacking in the IRO honeypot. The researchers noted that giveaways may cause alignment evaluations to overestimate a model's degree of alignment, and that understanding giveaways helps make evaluations closer to real deployment, but some task-level giveaways may be inherent properties, and it is uncertain whether they can all be eliminated.