Large-scale WildTeaming: from real-world jailbreaks to (adversarially) safer language models

1. 01What problem does this paper attempt to solve?

   The authors state that frontier LLMs are vulnerable to adversarial jailbreak attacks, while the open-source community lacks diverse jailbreak red-teaming methods and large-scale safety fine-tuning data to balance defense against over-refusal.

   The paper attempts to address the problem that large language models struggle to resist diverse adversarial jailbreak attacks, and that the open-source community lacks large-scale safety training resources that cover both direct and adversarial harmful queries without causing over-refusal.

   • Scenario and importance: Frontier large language models (LLMs) remain vulnerable when facing unsafe user requests, especially adversarial jailbreak attacks, prompting researchers to systematically mine potential jailbreak vulnerabilities and build effective defenses.
   • Shortcomings of existing methods: Existing red teaming jailbreak methods mostly rely on human annotation, gradient-based optimization, or LLM-based iterative modification, with limited attack diversity and high computational cost; meanwhile, existing public safety training data is small in scale and lacks adversarial complexity, causing models to be vulnerable to unknown adversarial attacks or to over-refuse benign requests.
   • Core observation and hypothesis: Real-world human-machine interaction logs contain rich and diverse user-written jailbreak strategies; by combining these mined strategies, adversarial prompts with both diversity and strength can be automatically generated at low computational cost, thereby supporting high-quality safety alignment training of models.
   • Proposed approach: The paper proposes the automatic red teaming framework WILDTEAMING and the synthetic safety instruction fine-tuning dataset WILDJAILBREAK containing 262K prompt-response pairs, aiming to improve models' defense against direct and adversarial harmful requests while avoiding over-refusal of benign requests and without harming general capabilities.
   • Threat model:
     • Attack goal: Bypass the safety protections of the victim model, prompting the model to give a full answer to harmful requests that originally belong to high-risk categories (PRMtarget_h).
     • Attacker knowledge: The paper evaluates black-box (e.g., GPT-3.5, GPT-4) and white-box (e.g., Vicuna-7B, Mistral-7B) target large models.
     • Attacker capability: The attacker has directly harmful requests, uses an off-the-shelf LLM (Mattack) to rewrite requests according to sampled jailbreak strategy combinations, and performs lightweight pruning through an off-topic classifier and a low-risk classifier.
     • Victim system: Open-source and closed-source frontier chat large language models, as well as downstream language models safety-aligned via supervised fine-tuning.

   Collapse this answer

2. 02What related research exists?

   The authors review three lines of work: red teaming jailbreaks, safety evaluation benchmarks, and safety training data, pointing out that previous methods lacked real strategy mining and large-scale adversarial fine-tuning training resources.

   Red Teaming and Jailbreak Attacks on Large Language Models

   • Human annotation and early attempts: Ganguli et al. (2022), Bai et al. (2022), OpenAI (2024) used human-written prompts and analyzed model responses. The authors state that human methods are limited in scope and efficiency due to high annotation costs.
   • Gradient-based optimization methods: Zou et al. (2023) (GCG), Guo et al. (2021), Guo et al. (2024) (Cold-attack), Schwinn et al. (2024) require backpropagation through model parameters. The authors state that these methods are computationally expensive, cannot be used for closed-source models, and often generate non-semantic text.
   • Reasoning-based and iterative modification methods: Chao et al. (2023) (PAIR), Liu et al. (2023) (AUTODAN), Mehrotra et al. (2024) (TAP) generate adversarial prompts through direct generation or iterative modification. The authors state that most methods have narrow attack and risk coverage, slow generation speed, and rarely translate into large-scale model training resources.

   Safety Evaluation and Benchmarks for Large Language Models

   • Vulnerability and harmful evaluation benchmarks: Carlini et al. (2023), Wei et al. (2023), Sun et al. (2024) (TrustLLM), Mazeika et al. (2024) (HARMBENCH) constructed benchmarks containing harmful prompts that models should refuse.
   • Over-refusal evaluation benchmarks: Röttger et al. (2023) (XSTest), Cui et al. (2024) (OR-Bench) are used to measure models' over-safe behavior on benign queries.

   Safety Training and Alignment Data

   • Safety fine-tuning and preference datasets: Bianchi et al. (2024) (SAFETY-TUNED LLAMAS) and Ji et al. (2023) (BeaverTails) respectively open-sourced a small-scale safety training set and a large-scale human preference dataset, but the authors note that they mainly focus on directly harmful requests.
   • Synthetic safety data exploration: Samvelyan et al. (2024) (Rainbow Teaming) uses a grid of human-defined attack styles and risk categories to fill the attack space with synthetic data.

   Baseline Methods and Benchmark Datasets

   • Jailbreak attack experiments compared gradient-based GCG (Zou et al., 2023), genetic algorithm-based AUTODAN (Liu et al., 2023), and LLM-based iterative prompting PAIR (Chao et al., 2023), with the test benchmark being the 159 standard harmful behavior test set from HARMBENCH (Mazeika et al., 2024).
   • Safety training experiments compared the original Tulu2Mix, the no-refusal-filtered version T2M-no-refusal, and the T2M-public-safety baseline that mixes existing public safety data HH-RLHF (Ganguli et al., 2022), SAFE-RLHF (Dai et al., 2024), and DAN (Shen et al., 2023).

   Differentiation from Existing Work: The authors state that WILDTEAMING differs from previous work in automatically mining human-conceived jailbreak strategies from real human-machine interactions and efficiently combining them, thereby constructing a larger-scale open-source instruction fine-tuning safety dataset WILDJAILBREAK that covers both direct and adversarial, harmful and benign four types of data.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors propose WILDTEAMING to mine and combine real jailbreak strategies, and construct the WILDJAILBREAK four-type dataset containing 262K samples for safety fine-tuning.

   The overall approach is to mine diverse jailbreak strategies from real human-machine interaction logs, combine them to generate adversarial attacks, and then construct the WILDJAILBREAK dataset containing direct and adversarial, harmful and benign four types of data for model safety alignment training.

   Real Strategy Mining (MINE)

   • Data source and candidate filtering: Single-turn conversations are collected from two real human-machine dialogue datasets, LMSYS-CHAT-1M and WILDCHAT. After initial screening with the OpenAI Moderation API, they are input to the lightweight safety model Tulu2-7B and judged by LLAMA-GUARD, retaining 16,850 adversarial prompts that induced harmful responses.
   • Seed strategies and automatic mining: Manual inspection of about 200 samples distills 35 seed jailbreak strategies and definitions; then GPT-4 is used to perform core request extraction and strategy identification on adversarial prompts (including existing strategies and potential new strategies, corresponding segments, and rationale).
   • Deduplication and clustering: Sentence embeddings are computed for strategy definitions using the Nomic Embed model, and hierarchical clustering deduplication is performed at a similarity threshold of 0.75, finally summarizing 5,688 unique strategy clusters from 105,438 strategy instances.

   Strategy Combination and Adversarial Attack Generation (COMPOSE)

   • Candidate attack generation: For a given directly harmful request P, a strategy subset Tⁱ = {t₁, …, t_n} is sampled from the strategy library T, and the attack LLM Mattack (e.g., Mixtral-8×7B) is used to conditionally generate adversarial candidate samples APⁱ ∼ M_attack(· ∣ P; Tⁱ). In jailbreak evaluation experiments, a prefix-inducing strategy (seed leading sentence) is fixed, and 3 additional strategies are randomly sampled to form a combination input of 4 strategies.
   • Dual off-topic and low-risk pruning: To ensure that adversarial prompts retain the original harmful intent and have sufficient threat, a natural language inference (NLI) classifier is used to judge whether the adversarial rewrite entails or is neutral to the original request (with the sum of entailment and neutral probabilities thresholded at 0.9), and an internally fine-tuned Llama-2 7B prompt harmfulness classifier is used to filter low-risk samples. A sample is retained only when it passes both filters simultaneously:

   is_keep_APⁱ = 𝕀(Pr_off-topic(APⁱ) = F ∧ Pr_low-risk(APⁱ) = F)

   Construction of the WILDJAILBREAK Four-Type Dataset

   • Directly harmful data (Vanilla Harmful, 50,050 items): Based on a risk taxonomy, GPT-4 is used to generate directly harmful requests covering 13 risk categories, and GPT-3.5 generates detailed and helpful refusal responses.
   • Directly benign data (Vanilla Benign, 50,050 items): Based on XSTest's over-safety classification system, GPT-4 is used to generate superficially similar but actually harmless requests, and GPT-3.5 generates compliant responses to mitigate over-refusal.
   • Adversarial harmful data (Adversarial Harmful, 82,728 items): WILDTEAMING is used to sample 2–7 real strategies, Mixtral-8×7B and GPT-4 transform directly harmful requests and undergo pruning, paired with refusal responses generated for the direct requests.
   • Adversarial benign data (Adversarial Benign, 78,706 items): WILDTEAMING is used to transform directly benign requests to generate samples with the external form of jailbreaks but without malicious intent, with compliant responses generated by GPT-3.5.

   Safety Alignment Fine-Tuning Configuration

   • Training data mixture: The 300K general instruction data Tulu2Mix-no-refusal, from which self-contradictory refusal responses were removed via keyword filtering, is mixed with 50K samples from each of the four WILDJAILBREAK types (200K in total), forming a 500K training mixture.
   • Fine-tuning hyperparameters: The Llama2-7B base model is fine-tuned for 2 epochs on a 128-chip TPU v3 pod, using BFloat16 precision, peak learning rate 2e-5 (linear decay), warmup ratio 0.03, maximum sequence length 2048, and batch size 32.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors evaluate jailbreak effectiveness and diversity on benchmarks such as HARMBENCH, and evaluate the fine-tuning alignment effect of WILDJAILBREAK on multiple safety and general benchmarks.

   Experimental setup

   • Models tested: Jailbreak attack experiments evaluated Vicuna-7B, Tulu2-DPO-7B, Mistral-7B, Mixtral-8×7B, GPT-3.5 (0613), and GPT-4 (0613); safety fine-tuning experiments evaluated models fine-tuned on Llama2-7B and Llama2-13B bases.
   • Evaluation datasets and benchmarks: Jailbreak evaluation uses the 159 directly harmful behaviors from the HARMBENCH standard test set; direct safety evaluation uses HARMBENCH (300 items), ToxiGen (500 items per 13 minority groups), and XSTest (200 harmful, 250 benign); adversarial safety evaluation uses JailbreakTrigger (400 items), DoAnythingNow (300 items), and the WILDJAILBREAK evaluation set (2000 harmful, 210 benign); general capability evaluation uses AlpacaEval (V1, V2), MT-Bench (80 questions), MMLU (14,079 questions), GSM8k (8.5k questions), BBH (6,511 questions), TydiQA (18,751 questions), HumanEval (164 questions), and TruthfulQA (818 questions).
   • Comparison baselines: Jailbreak attack baselines include GCG, AUTODAN, and PAIR; safety fine-tuning baselines include the original Tulu2Mix (T2M), T2M-no-refusal, and T2M-public-safety mixed with public safety data.
   • Evaluation metrics and judgment methods: Jailbreak effectiveness metrics are ASR (judged by a Llama2-13B classifier fine-tuned on HARMBENCH), query count (Query), and text perplexity (PPL, computed by Vicuna-7B); jailbreak diversity metrics are ASR×5_30 (average success rate of finding 1–5 independent successful attacks within 30 attempts) and Query×5_30; safety defense metrics include ASR, refusal rate RTA (judged by GPT-4), and comprehensive balanced accuracy (Accuracy); general capability uses win rate, F1, pass@10, or accuracy.
   • Generation and sampling parameters: Attack candidate generation max length 1024 tokens (temperature=1, top-p=0.9); target model completion max length 512 tokens (temperature=0, top-p=1).

   Main results

   The table below presents WILDTEAMING's attack effectiveness and diversity results on the HARMBENCH test set (159 standard harmful behaviors) against 6 representative models (from Table 2):

   Swipe horizontally to read the table

   Target model	ASR (↑)	Query (↓)	PPL (↓)	ASR×5_30 (↑)	Query×5_30 (↓)
   Vicuna (7B)	93.1	2.82	8.65	88.1	9.31
   Tulu2 DPO (7B)	96.9	2.61	8.77	87.8	8.98
   Mistral (7B)	95.0	2.37	8.56	89.2	8.72
   Mixtral (8×7B) [Note: target model is also the attack generation proxy model]	98.1	2.72	8.75	87.2	8.99
   GPT-3.5 (0613)	92.5	7.08	7.96	65.8	13.19
   GPT-4 (0613)	79.9	8.61	8.13	60.1	13.43

   • Diverse attack success rate performance: The authors state that WILDTEAMING improves the diversity metric ASR×5_30 by 4.6 to 25.6 points over PAIR, while reducing Query×5_30 by 3.8 to 5.5 queries, indicating an advantage in finding multiple independent successful jailbreak prompts.
   • Standard effectiveness and text naturalness: WILDTEAMING is comparable or superior to PAIR in ASR for single successful attacks (e.g., ASR of 79.9% on GPT-4 vs. PAIR's 78.6%), and the perplexity (PPL) of successful attack texts remains between 7.96–8.77, lower than PAIR (9.33–9.78) and GCG (2266.69–4265.86).
   • Limitations of optimization-based baselines: In diversity evaluation, AUTODAN's ASR×5_30 is only 19.4–20.0, and the authors note it struggles to find a second independent successful attack; GCG is not reported for diversity metrics due to high computational cost, as generating 30 candidate attacks is estimated to take about 15 hours on a single 80GB A100 GPU.

   Safety alignment training effects based on WILDJAILBREAK

   • Evaluation content: The evaluation includes fine-tuning Llama2-7B with 200K WILDJAILBREAK data (+WildJailbreak) and its impact on direct safety, adversarial safety, and general capability (Table 6).
   • Experimental results: The +WildJailbreak model reduces ASR on HARMBENCH to 3.1% (original Tulu2Mix: 20.8%, T2M-public-safety: 66.0%); on DoAnythingNow, ASR drops to 14.0% (original Tulu2Mix: 49.7%); on the WILDJAILBREAK comprehensive test set, accuracy reaches 98.4%, with harmful adversarial ASR at 1.7% and benign adversarial over-refusal RTA at only 1.6%; on general benchmarks, MT-Bench score is 6.29 (T2M-no-refusal: 5.84), and AlpacaEval V1 win rate is 74.6% (T2M-no-refusal: 75.9%). The +WJ (13B) model based on Llama2-13B scores 6.59 on MT-Bench, achieves an 80.5% win rate on AlpacaEval V1, and has a HARMBENCH ASR of 2.5% (Table 6).
   • Authors' interpretation: The authors state that WILDJAILBREAK greatly improves model defense against direct and adversarial attacks while having minimal negative impact on general capability, and clearly outperforms the T2M-public-safety baseline trained on public safety data.

   Ablation of WILDJAILBREAK data components and scaling analysis

   • Evaluation content: Table 6 ablates the four data types in WILDJAILBREAK, and Tables 35 and 36 evaluate the impact of scaling safety data from 2K to 60K/200K.
   • Experimental results: Training with only harmful data (+WJ-harm-only) increases over-refusal rates, with XSTest benign refusal rate XSTB at 10.0% and WILDJAILBREAK benign adversarial refusal rate WJB at 54.4%; fine-tuning with only direct data (+WJ-vani-only) still has an ASR of 57.5% on the adversarial test set WJ (H); fine-tuning with only adversarial data (+WJ-adv-only) still has an ASR of 20.8% on HARMBENCH direct harmful (Table 6). In scaling, mixing 60K WILDJAILBREAK data reduces HarmBench ASR to 2.5%, DAN ASR to 14.3%, and WJ (H) ASR to 2.3% (Table 36).
   • Authors' interpretation: The authors note that all four data components are indispensable for Pareto balance; direct and adversarial data must be combined to cover the full threat spectrum, and expanding safety data scale can steadily enhance model safety without harming general capability.

   Other ablations and analyses

   • Ablation of number of combined strategies (HARMBENCH validation set, Vicuna-7B): with a fixed prefix, combining 1, 2, 3, 4, 5, 6 strategies yields ASRs of 95.1%, 90.2%, 95.1%, 90.2%, 95.1%, 90.2%, and ASR×5_30 of 78.5, 83.4, 86.8, 82.4, 86.3, 84.4 (Table 21).
   • Ablation of random strategy combinations (HARMBENCH validation set, Vicuna-7B): without a fixed prefix, random combinations of 1, 2, 3, 4, 5, 6 strategies yield ASRs of 95.1%, 95.1%, 87.8%, 92.7%, 90.2%, 90.2%, and ASR×5_30 of 65.4, 74.6, 77.1, 80.5, 78.0, 79.0 (Table 21).
   • Ablation of pruning filter modules (HARMBENCH validation set, Vicuna-7B): with no pruning, off-topic pruning only, low-risk pruning only, and combined pruning, ASR is 95.1% in all cases, average query counts are 3.64, 2.95, 2.62, 2.46, and ASR×5_30 are 83.4, 83.9, 85.9, 86.8 (Table 21).
   • Validation accuracy of the prompt harmfulness classifier: overall accuracy is 86% on 3,091 labeled samples, with 90% for direct prompts (1,515 items) and 81% for adversarial prompts (1,576 items) (Table 27).
   • ASR of existing models on the WILDJAILBREAK adversarial evaluation set: Mistral-7B 76.2%, Mixtral-8x7B 69.2%, Tulu-2-7B 63.6%, GPT4-0125 29.5%, Llama-3-8B 14.8% (Figure 3, Table 28).

   Negative results and exceptions

   • In the copyright violation category of HARMBENCH, the fine-tuned model's defense improvement is relatively limited: +WJ has an ASR of 20.00%, while +WJ (13B) has a copyright ASR of 42.50% (original Tulu2Mix: 31.25%, Table 31). The authors note in the ethical statement that some harmful behaviors are highly context-dependent and cannot be fully covered by internal model training alone, requiring external guardrails.
   • The ablation model trained only on harmful data shows significant over-refusal, with a refusal rate of up to 54.4% on benign adversarial prompts (Table 6).
   • The prompt harmfulness classifier's classification accuracy on adversarial samples (81%) is lower than on direct samples (90%) (Table 27).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out limitations such as data source constraints, style deviation in synthetic prompts, and context-dependent risks requiring external protection, and propose directions such as evaluation upgrades and mechanism analysis.

   Limitations and future directions noted by the authors

   • Limited coverage of data sources and strategies: In the Ethical Statement (§8), the authors note that the findings are limited by existing datasets and the scope of WILDJAILBREAK, and cannot exhaust all potential misuse scenarios of large language models; the identified strategies are limited to the LMSYS-CHAT-1M and WILDCHAT data sources and may not reflect all combined jailbreak strategies used by real users.
   • Stylistic gap between synthetic adversarial prompts and real human attacks: In the Ethical Statement (§8), the authors note that although jailbreak strategies are derived from real-world data, the adversarial prompts in WILDJAILBREAK are generated synthetically by Mixtral-8×7B, GPT-3.5, and GPT-4, and do not fully match the actual queries of human users in form and content, potentially inheriting the source models' inherent style and filtering heuristics; the authors encourage future research to explore synthetic data generation methods more closely aligned with human attacks.
   • Context-dependent risks are difficult to fully address through internal model fine-tuning: In the Ethical Statement (§8), the authors note that while safety training can mitigate various risks, some harmful behaviors are inherently context-dependent and require external guardrails outside the model.
   • Safety evaluation benchmarks and metrics need upgrading: In the Discussion (§6), the authors note that many existing safety benchmarks suffer from data contamination or are nearing saturation, and existing classifiers and metrics are often insufficiently accurate; future work needs to upgrade safety evaluation methods, tasks, and metrics to keep pace with the continuous evolution of model capabilities.
   • Safety alignment training approaches and underlying mechanisms remain to be explored: In the Discussion (§6), the authors note the need to further understand the pros and cons of different alignment methods such as SFT, DPO, and PPO, explore the differences between end-to-end training and external filters, the trade-offs between direct refusal and detailed refusal, and deconstruct the conflict mechanisms between instruction following and refusal, while guarding against superficial safety alignment issues such as data poisoning and backdoors.

   Experimental coverage

   • Jailbreak attack targets: Jailbreak attack experiments covered 4 open-source models (Vicuna-7B, Tulu2-DPO-7B, Mistral-7B, Mixtral-8×7B) and 2 closed-source models (GPT-3.5-0613, GPT-4-0613), with the evaluation set being the 159 directly harmful behaviors from the HARMBENCH standard test set.
   • Model safety fine-tuning targets: Safety alignment training experiments used Llama2-7B as the primary base model (additionally including Llama2-13B), with supervised fine-tuning (SFT) on a mixture of 300K Tulu2Mix-no-refusal and 200K WILDJAILBREAK data.
   • Safety and general evaluation tasks: Direct safety evaluation covers HARMBENCH (300 items), ToxiGen (500 items per 13 minority groups), and XSTest (200 harmful, 250 benign); adversarial safety evaluation covers JailbreakTrigger (400 items), DoAnythingNow (300 items), and the WILDJAILBREAK evaluation set (2000 harmful, 210 benign); general capability covers AlpacaEval (V1, V2), MT-Bench, MMLU, GSM8k, BBH, TydiQA, HumanEval, and TruthfulQA.
   • Comparison attack baselines: The experiments compared three open-source jailbreak baselines: GCG, AUTODAN, and PAIR; the paper notes that the TAP baseline was not run due to computational resource constraints.
   • Unreported information: The paper does not report variance across repeated runs of attack experiments, nor statistical tests of agreement between evaluation classifier judgments and human annotations.

   Collapse this answer

6. 06Summarize the main content of the paper

   The paper proposes WILDTEAMING to mine and combine jailbreak strategies, builds the 262K WILDJAILBREAK dataset, and improves adversarial defense through fine-tuning while maintaining general capability.

   1. Research positioning and core problem: Addressing the vulnerability of large language models to adversarial jailbreak attacks and the lack of high-quality large-scale safety training resources in the open-source community, the paper proposes a red-teaming framework, WILDTEAMING, that automatically mines and combines real jailbreak strategies, and builds the open-source synthetic safety instruction fine-tuning dataset WILDJAILBREAK.
   2. Method and data construction: WILDTEAMING mines 5,688 unique jailbreak strategy clusters from 16,850 real conversation logs, and uses off-the-shelf large language models to combine and rewrite direct harmful requests with sampled strategies, filtering through off-topic and low-risk pruners; this constructs a dataset of 262K prompt-response pairs, including direct harmful (50,050), direct benign (50,050), adversarial harmful (82,728), and adversarial benign (78,706).
   3. Effectiveness and diversity of jailbreak attacks: On the HARMBENCH standard test set (159 harmful behaviors), WILDTEAMING achieves an ASR of 98.1% against Mixtral-8×7B, with an average query count of 2.72, perplexity (PPL) of 8.75, and diversity metric ASR×5_30 of 87.2, achieving more diverse attack performance than PAIR while maintaining lower query overhead (Table 2).
   4. Safety alignment fine-tuning effects: After fine-tuning Llama2-7B with a mixture of WILDJAILBREAK and general data, the model's ASR on HARMBENCH drops from 20.8% with the original Tulu2Mix to 3.1%, on DoAnythingNow jailbreak attacks from 49.7% to 14.0%, and achieves 98.4% accuracy on the WILDJAILBREAK comprehensive test set (Table 6).
   5. Balancing over-refusal and general capability: The fine-tuned model maintains low over-refusal rates on benign sensitive requests (XSTest benign refusal rate XSTB of 8.8%, WILDJAILBREAK benign adversarial refusal rate WJB of 1.6%), scores 6.29 on MT-Bench, and achieves a 74.6% win rate on AlpacaEval V1 (Table 6).
   6. Authors' conclusions and implications: The authors argue that mixing the four data types—direct and adversarial, harmful and benign—is essential for achieving Pareto optimality between safety and usability, that simple supervised fine-tuning can bring significant safety improvements, and call on the open-source community to further release safety training resources and upgrade dynamic evaluation benchmarks.

   Collapse this answer

Related papers