AART: AI-assisted red-teaming and diverse data generation for new LLM applications

The researchers propose AART (AI-assisted Red-Teaming), an AI-assisted red-teaming method that automatically generates adversarial evaluation datasets, as an automated alternative to human red-teaming, for testing the safety of content generated by LLMs in new downstream applications. The method provides reusable, customizable data generation and augmentation recipes, significantly reduces human effort, and supports introducing adversarial testing early in new product development. AART uses AI-assisted recipes to define, scope, and order diversity in the application context, generating highly diverse evaluation data covering sensitive and harmful concepts, a wide range of cultural and geographic regions, and application scenarios, and then scales evaluation prioritization through a structured LLM generation process. The authors state that, compared with some existing tools, AART shows encouraging results in concept coverage and data quality.