DeepMind uses language models to red-team language models

AI summaryDeepMind researchers propose using language models to automatically generate test cases to red-team a target language model, and found tens of thousands of offensive replies on the 280B-parameter Dialogue-Prompted Gopher chatbot. The team compared four question-generation methods: zero-shot, random few-shot, supervised learning and reinforcement learning; 500,000 zero-shot questions elicited 3.7% offensive replies (18,444 failure cases), and reinforcement learning pushed the proportion above 40% at a KL penalty weight of α=0.3, with the diversity and difficulty of generated questions comparable to questions written by human red teams. Red-teaming also found 1,709 replies that leaked training data, that the chatbot would generate real phone numbers and email addresses (including giving a suicide hotline in the wrong context), that there is distributional bias in the degree of offensiveness toward different groups, and that in multi-turn conversations offensive replies elicit more offensive replies. The authors consider language model red-teaming a powerful complement to human testing, while cautioning that external adversaries could use the same methods to automatically attack commercial models at scale.