STAR: A Sociotechnical Method for Red Teaming Large Language Models

The researchers propose STAR, a sociotechnical framework that improves existing best practices for safety red teaming of large language models. Its first contribution is enhanced controllability: it generates parameterized instructions for human red teamers, thereby better covering the risk surface and providing more fine-grained insight into model failures without increasing cost. The second contribution is improved signal quality: by matching demographic characteristics to evaluate harms targeting specific groups, it obtains more sensitive annotations. STAR also introduces a novel arbitration step that uses diverse perspectives to improve label reliability, treating disagreement as a valuable contribution to signal quality rather than as noise.