Red Teaming Generative AI: Silver Bullet or Security Theater?

Feffer et al. at CMU systematically analyze AI red teaming practices, examining six industry cases including Bing Chat, GPT-4, Gopher, Claude, and DEFCON, and surveying 104 papers. They find that the term is used inconsistently across purpose, evaluation targets, threat models, team composition, and result reporting, and that many public-facing red teaming exercises resemble security theater more than rigorous safety evaluation. The study identifies five axes of divergence and categorizes the literature into four method types: manual brute-force evaluation, AI-assisted attack generation, algorithmic search over prompt space, and targeted attacks against specific vulnerabilities. The paper distinguishes context-dependent harms from universally unacceptable content, noting that time-limited crowdsourced evaluations favor easily producible harmful outputs, creating an illusion of coverage. The authors propose a three-stage checklist of questions to standardize red teaming processes. The paper was submitted as a comment on the NIST AI Risk Management Framework and published at AAAI/ACM AIES 2024.