Skip to content

LLM defenses cannot yet withstand multi-turn human jailbreaks

LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

arXiv· arXiv:2408.15221· Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, Summer YueScale AI; UC BerkeleyIndexed by this site · 23 days ago· Originally publishedSelected AttacksSelectedAttention42

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors use multi-turn manual jailbreaks to attack four LLM defenses, reaching ASR of 51.7%–87.9% on HarmBench, higher than the ensemble upper bound of six automated attacks.

Threat modelRed team members conduct multi-turn conversations with the model through a black-box chat interface, cannot access the model's internals or output probabilities, are prohibited from prefilling assistant replies, and can switch among three temperatures: 0, 0.5, and 1; the victim system is a llama-3-8b-instruct chat interface loaded with defenses.

Questions
Existing LLM defenses are mainly evaluated under single-turn automated attacks; the authors argue that this threat model is insufficient to cover malicious use in multi-turn conversations in real deployments, and it remains unclear whether multi-turn manual jailbreaks break these defenses.
Methods
The authors organize a multi-stage manual red-teaming pipeline: each red team member performs multi-turn jailbreaking through a black-box chat interface within 30 minutes, and success is determined through manual review and GPT-4o filtering; they compare six automated attacks and construct the MHJ dataset and a jailbreak tactic taxonomy.
Experiments and results
On HarmBench (n=240), manual jailbreak ASR against RR, LAT, DERTA, and CYGNET is 51.7%–87.9%, higher than the ensemble upper bound of the six automated attacks; against the RMU unlearned model on WMDP-Bio (n=43), ASR is 51.2%, which the authors state recovers biosecurity knowledge.
Limitations and future work
The authors note that manual and automated attacks differ in threat model, evaluation protocol, and cost, so ASR cannot be compared directly; red team members vary in skill; the experiments cover only four defenses, one base model, and two benchmarks, and WMDP-Bio has only 43 questions and is manually scored.
Experimental setupRR、LAT et al. · HarmBench、WMDP-Bio et al. · ASR
Evaluated models
RRLATDERTACYGNET (cygnet-llama-3)RMU
Benchmarks
HarmBenchWMDP-BioMHJ
Metrics
ASR
AI summaryand selection rationaleThe paper shows that multi-turn human jailbreaks can break through existing LLM defenses, achieving over 70% attack success rate on HarmBench against defenses that report only single-digit attack success rates for automated single-turn attacks. The study organized human red teaming by Scale AI, conducting multi-turn conversations through a black-box chat interface, with results verified by two human reviewers and a GPT-4o classifier. On the 240 behaviors in HarmBench, human jailbreaks outperform an ensemble of six automated attacks by 19% to 65%; against the closed-source defense CYGNET they achieve a 70.4% attack success rate, whereas automated attacks in the original paper were 0%, and the paper notes the two settings differ and should not be directly compared. On WMDP-Bio, human red teaming achieves an attack success rate against the RMU machine unlearning defense 39.6% higher than the best single automated attack and 25.6% higher than the ensemble of six attacks. The team released the MHJ dataset, containing 2,912 prompts from 537 multi-turn jailbreak conversations along with a taxonomy of jailbreak strategies.

The paper shows that multi-turn human jailbreaks can break through existing LLM defenses, achieving over 70% attack success rate on HarmBench against defenses that report only single-digit attack success rates for automated single-turn attacks. The study organized human red teaming by Scale AI, conducting multi-turn conversations through a black-box chat interface, with results verified by two human reviewers and a GPT-4o classifier. On the 240 behaviors in HarmBench, human jailbreaks outperform an ensemble of six automated attacks by 19% to 65%; against the closed-source defense CYGNET they achieve a 70.4% attack success rate, whereas automated attacks in the original paper were 0%, and the paper notes the two settings differ and should not be directly compared. On WMDP-Bio, human red teaming achieves an attack success rate against the RMU machine unlearning defense 39.6% higher than the best single automated attack and 25.6% higher than the ensemble of six attacks. The team released the MHJ dataset, containing 2,912 prompts from 537 multi-turn jailbreak conversations along with a taxonomy of jailbreak strategies.

Why it was selectedThe paper compares human red teaming with six automated attacks; multi-turn jailbreaks achieve success rates 19 to 65 percentage points higher across four defenses, and the MHJ dataset is released.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors test whether existing LLM defenses are robust under multi-turn manual jailbreaks and release the MHJ dataset.

    Core question: Existing LLM defenses are mainly evaluated under single-turn automated attacks; the authors aim to test whether they remain robust under multi-turn manual jailbreaking, a threat model closer to real deployment.

    • Problem setting: The authors point out that recent defenses can achieve near-0% ASR against automated single-turn attacks on benchmarks such as HarmBench, but real users typically interact with models through multi-turn dialogue, and single-turn evaluation is an insufficient threat model.
    • Shortcomings of existing methods: The authors believe automated attacks targeting the multi-turn threat model are few, and defenders generally do not incorporate multi-turn adversaries into their design; for example, some defenses only perform adversarial training on single-turn attacks, and refusal training is also mostly conducted on short single-turn dialogues.
    • Core hypothesis: Multi-turn dialogue expands the attack surface, and experienced manual red team members can find vulnerabilities missed by single-turn automated attacks; the authors also hypothesize that the ASR distribution of automated attacks cannot represent the distribution of manual attacks.
    • Paper outputs: The authors organize a manual red-teaming pipeline to attack four defenses and one machine unlearning model, and compile successful jailbreaks into the MHJ dataset (2,912 prompts, 537 multi-turn conversations) and a jailbreak tactic taxonomy, released publicly.
    • Threat model:
      • Goal: Induce the model through multi-turn dialogue to give harmful responses to HarmBench harmful behaviors, or recover WMDP-Bio dual-use biosecurity knowledge from the unlearned model.
      • Knowledge: black-box chat interface; red team members cannot access model internals or output probabilities.
      • Capabilities: Multi-turn dialogue is allowed, and temperature can be switched among 0, 0.5, and 1; prefilling assistant replies is prohibited, assuming the attacker cannot obtain private assistant tokens.
      • Victim system: The llama-3-8b-instruct chat interface loaded with a defense (RR, LAT, DERTA, CYGNET, or RMU unlearning), with the system prompt "You are a helpful AI Assistant!".
  2. What related research exists?

    Related work falls into three categories: LLM automated attacks, LLM defenses, and manual red teaming; this paper uses six automated attacks as baselines.

    LLM automated attacks

    • Transformation-based attacks (cited in the paper [1, 25, 28, 36, 42, 74]): transform prompts to bypass refusals, usually without requiring access to model internals.
    • Gradient optimization-based attacks (cited in the paper [23, 47, 59, 63, 64, 68, 70, 80]): require access to model weights, activations, or gradients, such as GCG and AutoPrompt.
    • LLM as prompt engineer (cited in the paper [3, 13, 15, 17, 49, 54, 58, 65, 75]): use LLMs to automatically generate jailbreak prompts, simulating manual red teaming, such as PAIR, GPTFuzzer, AutoDAN, and Zero-Shot. Among them, Gibbs et al. [24] extended single-turn automated attacks to multi-turn and released an automated multi-turn jailbreak dataset; Crescendo by Russinovich et al. [58] is the multi-turn attack closest to the threat model of this paper, but the authors did not include it in experiments because its code was not released at the time of evaluation.

    LLM defenses

    • Adversarial training-based (cited in the paper [22, 48, 76, 78]): directly improve robustness to attacks through adversarial training; DERTA [76] evaluated in this paper belongs to decoupled refusal training with output-layer supervision.
    • Representation-level methods (cited in the paper [61, 66, 73, 79, 81]): alter internal model activations to reduce the likelihood of it "thinking" about harmful behavior; RR, CYGNET [81], and LAT [61] evaluated in this paper belong to this category.
    • Machine unlearning (cited in the paper [41, 43, 57, 61, 66]): aims to remove only harmful technical knowledge without harming beneficial capabilities; RMU [41] attacked in this paper belongs to this category; the paper cites [41, 45, 59, 66] to point out that adversarial attacks are needed to confirm that knowledge is truly unlearned rather than masked.

    Manual red teaming

    • Frontier model red teaming (cited in the paper [7, 18, 21, 22, 30, 37, 50, 60, 67, 71, 72]): manual red teaming discovers vulnerabilities with minimal model access and is a core strategy for frontier models.
    • Red team tactic taxonomies (cited in the paper [32, 37, 72, 77]): prior work has established taxonomies for manual red team tactics.
    • Cost issue: The paper points out that manual red teaming is costly, which is why researchers mostly use automated robustness benchmarks such as HarmBench [48] and JailbreakBench [16].

    Baselines and benchmarks: The baselines in this paper are six automated attacks—black-box AutoDAN, GPTFuzzer, PAIR, Zero-Shot and white-box AutoPrompt, GCG—using HarmBench v1.0 default hyperparameters; the benchmarks are 240 harmful behaviors from HarmBench and 43 questions sampled from WMDP-Bio.

    The authors position this paper as: using manual red teaming to test the neglected multi-turn threat model, pointing out a gap between automated attacks and manual red teaming, and calling for better baselines for automated attacks and defenses.

  3. How does the paper address this problem?

    Multi-stage manual red-teaming pipeline: 30-minute multi-turn jailbreaking, two-person review plus GPT-4o filtering, then comparison with six automated attacks.

    Overall approach: organize expert red team members to manually jailbreak four defenses under a multi-turn black-box chat interface, use a multi-stage pipeline to control false positives, then compare ASR with six automated attacks on the same benchmarks, and compile successful jailbreaks into the public dataset MHJ.

    Threat model and experimental setup

    • Interface setup: Red team members interact with the model through a black-box chat interface and cannot access model internals or output probabilities; prefilling assistant replies is prohibited, assuming private assistant tokens are unavailable.
    • Multi-turn and sampling: Multi-turn dialogue is allowed, and temperature can be switched among 0, 0.5, and 1; the authors state this is more restricted than API access and closer to real malicious use than single-turn attacks.
    • Victim system: Defenses are loaded on llama-3-8b-instruct, uniformly using the system prompt "You are a helpful AI Assistant!", without perplexity filtering or guardrail models; only attack-agnostic defenses are evaluated.

    Manual red-teaming pipeline (Figure 2)

    1. Attempt stage: Red Teamer 1 independently attempts jailbreaking within 30 minutes and self-rates as failed, partially successful, or fully successful; if only partially successful, Red Teamer 2 independently retries from scratch without seeing the former's information.
    2. Validate stage: Reviewer 1 verifies fully successful jailbreaks according to the standard of the HarmBench classifier prompt; if rejected, Red Teamer 3 retries from scratch, reviewed by Reviewer 2, and no further referral after rejection.
    3. GPT-4o final screening: gpt-4o-2024-05-13 with the HarmBench classifier prompt performs final filtering on manually approved jailbreaks; the authors state this is to obtain a conservative lower bound on manual ASR (Appendix A.3).

    Automated attack comparison setup

    • Six attacks: Black-box AutoDAN, GPTFuzzer, PAIR, Zero-Shot and white-box AutoPrompt, GCG, using HarmBench v1.0 default hyperparameters, run against all defenses and both benchmarks.
    • Ensemble upper bound: For each behavior, success by any one of the six automated attacks is recorded as ensemble success, serving as an upper bound on automated attack ASR.
    • Positioning of white-box attacks: White-box attacks exceed the threat model of this paper; the authors still report their results, on the grounds that defenses deployed in critical scenarios should be tested under stronger threats.

    Success determination

    • HarmBench: Determined by GPT-4o with the HarmBench classifier prompt; automated attacks are initially screened by GPT-4o and then manually reviewed, and only agreement between both counts as success; manual jailbreaks require two manual reviews plus GPT-4o approval.
    • WMDP-Bio: 43 multiple-choice questions sampled from the benchmark are converted to open-ended question answering; red team members have no time limit, totaling 240 hours, and all submissions are manually judged because the HarmBench classifier cannot judge dual-use biosecurity content; the authors explicitly state that ASR here cannot be compared with HarmBench results.

    MHJ dataset and tactic taxonomy

    • Data content: All successful submissions are compiled into MHJ, containing 2,912 prompts and 537 conversations; metadata includes primary tactic, time spent, and temperature, and red team members attach design notes.
    • Tactic system: 7 main categories (Direct Request, Echoing, Hidden Intention Streamline, Injection, Obfuscation, Output Format, Request Framing, Table 1); Appendix D.1 gives fine-grained subcategories, and Appendix D.2 gives examples of each category; each submission is labeled with only one dominant tactic.
    • De-identification: With support from legal counsel, sensitive and export-controlled content related to biological, cyber, and chemical safety is removed, all model completions are deleted, and only system and user messages are retained.
  4. What experiments did the paper conduct?

    Manual jailbreaks achieve 51.7%–87.9% ASR on HarmBench, exceeding the ensemble of six automated attacks; 51.2% on RMU.

    Experimental setup

    • Defenses tested: RR, LAT, DERTA (all loaded on llama-3-8b-instruct, open-source) and CYGNET (closed-source, attacked via the Gray Swan AI API, model cygnet-llama-3, 2024-07-27 to 2024-08-07); the unlearning experiment applies RMU to llama-3-8b-instruct.
    • Benchmarks: HarmBench standard and contextual test sets totaling 240 harmful behaviors (copyright category excluded); 43 questions sampled from WMDP-Bio converted to open-ended question answering.
    • Baselines: six automated attacks (AutoDAN, GPTFuzzer, PAIR, Zero-Shot, AutoPrompt, GCG) and their ensemble upper bound; the automated attack ASR for CYGNET is cited from the original source [81], and the authors note it should not be directly compared with manual ASR.
    • Review method: gpt-4o-2024-05-13 with the HarmBench classifier prompt; manual jailbreaks require two manual reviews plus GPT-4o approval, automated attacks are initially screened by GPT-4o plus one manual review; WMDP-Bio is entirely manually scored.
    • Samples and labor: HarmBench n=240 per defense; at least 6 red team members per defense, 30 minutes each per session; WMDP-Bio totals 240 hours of red teaming time.

    Main results

    ASR (%) of each attack against the four defenses on HarmBench (n=240, according to Table 2):

    Swipe horizontally to read the table

    AttackRRLATDERTACYGNET*
    Human (Ours)51.775.487.970.4
    Ensemble Automated8.310.068.30.0*
    AutoDAN0.40.029.60.0*
    AutoPrompt1.20.023.8-
    GCG2.92.935.00.0*
    GPTFuzzer0.80.034.2-
    PAIR5.45.829.60.0*
    ZeroShot1.73.816.2-

    *The automated attack numbers for CYGNET are cited from the original source [81], and the authors note they should not be directly compared with manual ASR; "-" indicates not reported in the paper.

    • The authors state that manual jailbreaks exceed all six automated attacks on all four defenses, and are 19.6%–65.4% higher than the ensemble upper bound on the three open-source defenses.
    • The authors point out that CYGNET is more robust to automated attacks than RR, but the red team has a higher success rate on CYGNET, indicating that automated attack ASR is not necessarily a proxy for robustness to manual jailbreaks.
    • Appendix A.2 adds: CYGNET ASR before the circuit breaker is 70.4% (169/240), with an additional 7.1% (17/240) after the circuit breaker, totaling 77.5%; the main text figures report only the pre-circuit-breaker numbers.

    WMDP-Bio unlearned model experiment

    • What was tested: On the RMU-unlearned llama-3-8b-instruct, manual red teaming and six automated attacks attempt to recover dual-use biosecurity knowledge (n=43, Figure 4).
    • Results: Manual jailbreak ASR 51.2%, ensemble automated attacks 25.6%, best single automated attack 11.6% (AutoDAN and GPTFuzzer tied), AutoPrompt 4.7%, GCG 9.3%, PAIR and ZeroShot 2.3% each.
    • Authors' interpretation: The authors state that automated attacks are insufficient to guarantee the unlearning robustness of RMU; manual ASR is still lower than on the HarmBench defenses, and the authors speculate that recovering highly technical knowledge may require domain expertise, leaving this to future work.

    Tactic distribution and multi-turn dependence

    • What was tested: The distribution of dominant tactics in successful manual jailbreaks (Figure 5) and the number of dialogue turns required.
    • Results: 91.62% of successful HarmBench submissions and 85.7% of successful WMDP-Bio submissions require more than one round of dialogue; the authors state that Obfuscation, Hidden Intention Streamline, and Direct Request are effective on all defenses, and the tactic counts differ across defenses in Figure 5 (e.g., Obfuscation is 60 for LAT, Direct Request is 40 for RR).
    • Authors' interpretation: The authors believe many tactics are optimized for the multi-turn threat model, for example Hidden Intention Streamline relies entirely on multi-turn inducement; restricting to single-turn or distilling multi-turn into single-turn is a future direction.

    Other ablations and analyses

    • Average time to success (HarmBench): RR 13.9 minutes, DERTA 12.6 minutes, LAT 17.3 minutes, CYGNET 16.5 minutes; the authors point out that ASR levels do not correspond to average time.
    • WMDP-Bio time: Successful submissions average 20.5 minutes.
    • GPT-4o classifier validation (Appendix A.3): On automated attacks, GPT-4o's false negative rate on manually annotated harmful responses is 0.0%, but precision is weak (Figure 7: of 464 responses judged harmful by GPT-4o, 166 were manually confirmed); on manual jailbreaks, GPT-4o acceptance rate is 88.4%–94.3% (Table 3).
    • RMU hyperparameters (Appendix B): batches=400, c=50, α=5000, lr=2×10⁻⁴; after unlearning, WMDP-Bio accuracy is 28.0% (base 70.7%), MMLU 61.3% (base 63.8%) (Figure 8).
    • Semantic categories (Appendix A.1/Figure 6): The authors state that categories such as harassment bullying are harder for automated attacks than manual attacks; Figure 6 gives the ASR distribution of each attack by category, with category sample sizes of 18–54.
  5. What are some points that can be further explored?

    The authors discuss limitations such as differences between human and automated evaluation, and propose directions such as multi-turn automated attacks and domain-expert red teaming.

    Limitations and future directions noted by the authors

    • Human and machine procedures are not directly comparable (Section 5): The threat models of human and automated attacks differ—some automated attacks require model internal access and are all single-turn, while human red team members cannot access internals but can go multi-turn and adjust temperature; the authors state that the results should not be viewed as a one-to-one comparison or as an assertion that human red teaming is superior.
    • Time and resource setup (Section 5): Human red team members are limited to 30 minutes each, whereas real malicious actors can use more time and manpower; the paper does not limit the number of queries or tokens, and the authors suggest that future defenses could explore rate limiting.
    • Differences among red team members (Section 5): Red team members are fluid; although they receive uniform training and there are at least 6 people per defense, individual skills and experience still differ.
    • Cost differences (Section 5): Human red teaming costs orders of magnitude more than automated attacks, and the authors believe that any cost-controlled robustness analysis may yield different results.
    • Evaluation scheme differences (Section 5): Human and automated attacks use slightly different evaluation procedures; the design rationale is in Appendix A.3.
    • Future directions: The authors propose exploring automated attacks that are multi-turn or vary temperature (Section 5), training domain-expert red teams to recover highly technical knowledge (Section 4.2), restricting human red teaming to single-turn or distilling multi-turn attacks into single-turn (Section 4.2), and safety research for longer-context, multi-turn, out-of-distribution tool-use scenarios (Section 5).

    Scope of experiments

    • There are four defenses tested (RR, LAT, DERTA, CYGNET) plus one RMU unlearned model, all based on llama-3-8b-instruct; no other base models are evaluated.
    • The benchmarks are 240 HarmBench behaviors (copyright category excluded) and 43 sampled WMDP-Bio questions; WMDP-Bio is human-graded, and the paper does not report inter-rater agreement.
    • There are six automated attack baselines, using HarmBench v1.0 default hyperparameters; Crescendo is not included because its code was not released at evaluation time (Section 4.1).
    • The automated attack numbers for CYGNET are cited from the original source, and the authors note that the system prompt may differ and that they should not be directly compared with human ASR (Appendix A.2).
    • The paper does not report the number of queries or the compute budget for automated attacks, nor the number of repetitions of the human red team experiments.
  6. Summarize the main content of the paper

    Multi-turn manual jailbreaks break four defenses, with ASR far exceeding automated attacks; the authors release the MHJ dataset.

    This is an attack and dataset paper: the authors test whether LLM defenses that are robust under single-turn automated attacks can withstand multi-turn human jailbreaks that are closer to real-world use.

    • Problem: Existing defenses can reach near 0% ASR against automated single-turn attacks on benchmarks such as HarmBench, but real malicious use involves multi-turn conversations, and the authors argue that single-turn evaluation is an inadequate threat model.
    • Method: Red team members attack four defenses (RR, LAT, DERTA, CYGNET, all based on llama-3-8b-instruct) over multiple turns through a black-box chat interface, 30 minutes per person; success is determined by two human reviews plus GPT-4o filtering; the baseline is six automated attacks and their ensemble upper bound.
    • Main results: On HarmBench (n=240), human jailbreak ASR is RR 51.7%, LAT 75.4%, DERTA 87.9%, CYGNET 70.4%, which is 19.6%–65.4% higher than the automated attack ensemble upper bound on three open-source defenses (Table 2).
    • Unlearning experiment: For the RMU unlearned model, the human red team achieves ASR 51.2% on WMDP-Bio (n=43), higher than the best single automated attack's 11.6%, and the authors state that dual-use biosecurity knowledge was recovered.
    • Multi-turn dependence: 91.62% of successful HarmBench submissions require more than one turn of conversation, and the authors argue that many tactics are optimized specifically for the multi-turn threat model.
    • Outputs and conclusion: The authors release the MHJ dataset (2,912 prompts, 537 conversations) and a taxonomy of 7 categories of jailbreak tactics, and argue that current automated attacks are insufficient to guarantee defense robustness in real deployment, and that more realistic threat models and stronger automated attacks are needed; the authors also caution that human and machine ASR cannot be directly compared due to differences in setup.

Related papers

Show 3 more papers
Read originalarxiv.org