Skip to content
Anthropic Research·Indexed by this site · Originally published· Tools →· Its place in the scan record →SelectedAttention38

Anthropic explains how to build reliable safety evaluations

Building strong safety evaluations

AI summary

Anthropic releases a practical guide explaining how to build reliable safety evaluations, combining its own evaluation experience with lessons from the reproducibility reform in psychology. The guide emphasizes writing down the construct to be measured first, points out that refusal rate is not always the right metric, and that for risks such as suicide and self-harm one should instead use the appropriate-response rate judged by domain-expert grading criteria, while also measuring over-refusal on benign requests. Methodologically, it recommends using multi-turn simulations to capture gradually escalating risks, running evaluations in a sandbox without external network access with real-time monitoring, validating grader models against human annotations, and pre-registering before analysis and reporting confidence intervals. The text gives as an example that on its API, recent models' single-turn evaluation harmless-response rates range from 94% to 98% and have become saturated; Claude Fable 5.1's appropriate-response rate in multi-turn safety evaluations differs by up to 34 percentage points between the API and claude.ai configurations, and after improving the system prompt in response to a self-harm response defect found in pre-release review, claude.ai's appropriate-response rate reached 94% and the API's 60%.

Why it was selected

Anthropic, based on its own system cards and evaluation practices, has compiled an actionable methodology for building safety evaluations; recommendations such as grader model validation and pre-registration can be transferred directly.

Related papers
Read originalanthropic.com