Jailbreak-Zero: A Path Toward Pareto-Optimal Red Teaming for Large Language Models

The researchers propose the Jailbreak-Zero red teaming method, which shifts large language model safety evaluation from a constrained example-based approach to a policy-based framework. The method first has an attack model generate a large number of diverse adversarial prompts, then fine-tunes it on a preference dataset, achieving Pareto optimality on three objectives: policy coverage, attack strategy diversity, and prompts resembling real user inputs. The paper's empirical results show that the method's attack success rate on open-source and proprietary models such as GPT-4o and Claude 3.5 is significantly higher than existing state-of-the-art techniques, and that the generated adversarial prompts are readable and effective and require almost no human intervention, providing a more scalable approach for discovering and mitigating large language model safety vulnerabilities.