Understanding annotators' safety policies with interpretability

The researchers propose Annotator Policy Models (APMs), interpretable models that learn annotators' internal safety policies solely from annotation behavior, making annotators' reasoning visible and comparable without additional annotation. The work points out that annotation disagreement can stem from operational errors, policy ambiguity, or value pluralism, which respectively require quality control, policy clarification, and deliberation over multiple perspectives; directly asking annotators is costly, and self-reported reasoning often does not match the actual decision process. Validation shows that APMs model annotators' safety policies with over 80% accuracy, faithfully predict responses to counterfactual edits, and recover known policy differences in controlled settings. Applying APMs to large language models (LLMs) and human annotation can reveal different interpretations of safety instructions among annotators to expose policy ambiguity, and identify systematic differences in safety priorities across demographic groups to surface value pluralism, supporting more targeted, transparent, and inclusive safety policy design.