Crescendo: an LLM jailbreak attack that gradually escalates through multi-turn dialogue

1. 01What problem does this paper attempt to solve?

   The authors propose the multi-turn jailbreak Crescendo: it uses benign inputs throughout and escalates gradually via the model's own replies, bypassing alignment and filters that target only single-turn malicious inputs.

   Core problem: existing jailbreak attacks are mostly single-turn and contain identifiable malicious content, making them easy to defend against with input filters, while existing alignment and evaluations also target only single-turn jailbreaks; the authors investigate whether multi-turn jailbreaks with benign inputs are feasible.

   • Scenario and motivation: LLMs are widely deployed and safety-aligned, but the authors argue that current alignment strategies and benchmarks focus only on single-turn jailbreaks, and that multi-turn jailbreaks can bypass these measures.
   • Shortcomings of existing methods: the authors state that optimization-based jailbreaks (such as GCG and AutoDAN) require white-box access and are computationally expensive; once text-based jailbreaks (such as DAN) are discovered, input filters can defend against them because the input contains identifiable malicious content.
   • Core observation: the authors argue that LLMs tend to follow contextual patterns and focus on recent text, especially text generated by the model itself; Crescendo exploits this by starting from a harmless topic and escalating gradually by quoting the model's replies, similar to the "foot-in-the-door" psychological strategy.
   • Paper outputs: proposes the multi-turn jailbreak attack Crescendo and its automated tool Crescendomation (the authors state it has been integrated into PyRIT), and provides mechanism analysis, comparisons with multiple baselines, transferability, and defense evaluation.
   • Threat model:
     • Goal: make an aligned LLM perform tasks it would otherwise refuse (categories such as illegal, misinformation, pornographic, and hateful content).
     • Knowledge: completely black-box, with no assumptions about the target model.
     • Capability: carried out through normal interaction with the LLM; the automated version requires only black-box API access to the target LLM.
     • Victim systems: public chat systems and models, including ChatGPT, Gemini, Anthropic Chat, and LLaMA-2/3 Chat.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four categories: in-the-wild jailbreaks and inference parameter manipulation, optimization-based jailbreaks, LLM-generated/multi-turn jailbreaks, and context injection jailbreaks; the baselines are MSJ, CIA, PAIR, and CoA.

   In-the-wild jailbreaks and inference hyperparameters

   • [22, 27]—investigate in-the-wild jailbreak prompts, finding that most have been patched and some were still active at evaluation time.
   • [16, 35]—manipulate inference hyperparameters such as temperature, topK, and decoding method; the authors relay their results as claiming improved jailbreak success rates.

   Optimization-based jailbreaks

   • Wei et al. [28]—identify two failure modes of safety alignment (competing objectives and mismatched generalization); [13] exploits multilingualism (mismatched generalization) to bypass safety restrictions.
   • [36] (GCG) and [21] (AutoDAN)—optimize adversarial suffixes for jailbreaking; the paper notes that both require white-box access to fine-tune the suffix and are therefore ineffective against black-box models such as GPT-3.5/GPT-4.

   LLM-generated and multi-turn jailbreaks

   • PAIR [10], CoA [32]—use an LLM to automatically optimize jailbreak prompts; CIA [11]—generates all turns of a multi-turn jailbreak with a single LLM call; [31] constructed a 2-turn jailbreak dataset.
   • [14, 12] fine-tune LLMs to generate jailbreaks; [26, 33] use LLMs as fuzzers.

   Context-injection jailbreaks

   • MSJ [7]—inserts more than 100 malicious in-context examples into the context; [20]—inserts fabricated model-user interactions into the conversation history. The paper notes that the two differ in the scale and form of the injected content.

   Baselines and benchmarks: The comparison baselines are MSJ, CIA, PAIR, CoA (all using GPT-4 as the attack LLM with default settings); the benchmarks are an AdvBench subset (50 tasks) and HarmBench (100 random tasks).

   Positioning of this paper: The authors state that Crescendo uses no adversarial or malicious text, its inputs are entirely human-readable, and it relies on the model's own outputs rather than constructing explicit content; it involves no explicit optimization and is therefore a black-box jailbreak; the authors state that its ASR in Figure 6 and Table 4 is higher than other techniques, and that because it uses benign questions it is harder to detect and mitigate.

   Collapse this answer

3. 03How does the paper address this problem?

   Crescendo starts from a harmless topic and escalates turn by turn by referencing the model's own replies; Crescendomation uses GPT-4 to automatically generate questions, together with a three-layer judge and a backtracking loop.

   Overall approach: Crescendo opens with a harmless topic related to the target task, then escalates step by step each turn by quoting the model's own replies, leading the model to ultimately complete the task through a series of small, seemingly benign generations; Crescendomation automates this process.

   Attack Pattern (Manual Crescendo)

   • Basic pattern: Figure 2 presents the general pattern—first ask about an abstract topic, then have the model combine, rewrite, and reinforce its own generated content; the attacker almost never directly mentions the target task.
   • Backtracking strategy: during manual execution, each task allows at most 4 attempts; when the model refuses, the attacker uses the chat interface's edit function to modify and resend, or starts a new session if editing is not supported.
   • Variants: the task can be made more specific (targeting a particular region), or two Crescendos can be chained in the same session to introduce a second type of target (such as copyrighted content or profanity); a multimodal variant asks the model to generate an accompanying image after a successful text jailbreak.

   Mechanism Hypothesis and Verification Method

   • Hypothesis: after the model generates enough related content turn by turn, it effectively overrides its safety alignment.
   • Verification: two sets of experiments on LLaMA-2 70b (which the paper calls the only open-source aligned model)—gradually injecting aggressive content into the context and monitoring the token probability of target words (Figure 4); and using a three-turn Crescendo example (Sentence A/B/C) to measure changes in the probability of starting with "Sure"/"I" when skipping earlier turns, replacing the final turn, and removing the final reply content sentence by sentence (Table 3, Figure 5).

   Crescendomation Automation

   • Inputs and roles: the input is the target task t and the target model API; the attacking model A and evaluation model J are both GPT-4; the parameters are the number of rounds R and the number of independent iterations N (Algorithm 1).
   • Loop process: generate an initial question → send to the target model → process the reply and adaptively generate the next question → repeat until the task is completed; maintain the target model history and the attacking model history, and summarize the latest reply for use in subsequent iterations.
   • Meta-prompt: contains a description of the Crescendo attack and successful examples; it is not given in the appendix, and the main text states that the full algorithm is in Algorithm 1.

   Success Determination

   • Judge: GPT-4 outputs a boolean flag, a 0–100 percentage, and an explanation of the reply's completion of the task.
   • Secondary Judge: a second round of GPT-4 reviews whether the Judge's explanation is illogical due to safety restrictions, overturning the determination when necessary, to reduce false negatives.
   • Refusal Judge: determines whether the target model refused or triggered content filtering; if triggered, it backtracks, with at most 10 rephrasings.
   • External APIs: Perspective API (6 metrics, taking the highest score from 0–1) and Azure Content Filter (4 metrics, levels 0–7) provide auxiliary scoring; the authors note that these APIs do not cover categories such as misinformation and cannot be used as evaluation on their own. Finally, the best-performing replies are manually reviewed to reduce false positives.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Manual Crescendo succeeds on almost all tasks across 7 systems; Crescendomation achieves average ASR of 56.2%/82.6% against GPT-4/Gemini-Pro on the AdvBench subset, which the authors state exceeds four baselines.

   Experimental Setup

   • Models tested: Manual experiments cover ChatGPT (GPT-4), Gemini (Gemini Pro), Gemini Advanced (Gemini Ultra), Anthropic Chat (Claude-2, Claude-3 Opus, Claude-3.5 Sonnet), LLaMA-2 70b, LLaMA-3 70b (Table 2); Crescendomation experiments cover GPT-3.5, GPT-4, Gemini-Pro, Claude-3 (Opus), LLaMA-2 70b, and the model scale analysis additionally tests LLaMA-2 7b.
   • Datasets: the 15 self-built tasks in Table 1 (7 categories); a 50-task subset of AdvBench; 100 random tasks from HarmBench; for the generalization experiment, 12 tasks selected by GPT-4 from AdvBench categories (Table 9).
   • Baselines: MSJ, CIA, PAIR, CoA; all use default settings with GPT-4 as the attacking LLM; Crescendo, MSJ, CIA, and PAIR run 10 iterations, and CoA uses its default parallel setting. MSJ has no public code, so the authors reproduced it according to the paper, taking malicious examples from the GPTFuzz dataset, reducing the number of examples from 128 to 100, and using GPT-4 with 32k tokens.
   • Metrics: ASR (the proportion of trials in which the Judge determines at least one success, binary ASR) and average ASR (the mean of Judge scores from 0–100); the highest scores from Perspective API and Azure Content Filter.
   • Hyperparameters: Crescendomation runs independently 10 times, with at most 10 rounds of interaction and a backtracking limit of 10; temperature is set to 0.5 for all models.
   • Evaluation method: GPT-4 serves as Judge, Secondary Judge, and Refusal Judge; the authors state that they manually reviewed the highest-scoring replies, and the paper does not report numerical agreement between human reviewers and the Judge.

   Main Results

   According to Table 4 (50 AdvBench tasks, average ASR, with binary ASR in parentheses, unit %):

   Swipe horizontally to read the table

   Model	CIA	CoA	MSJ	PAIR	Crescendo
   GPT-4	35.6 (82.0)	22.0 (22.0)	37.0 (86.0)	40.0 (76.0)	56.2 (98.0)
   GeminiPro	42.4 (92.0)	24.0 (24.0)	35.4 (88.0)	33.0 (80.0)	82.6 (100.0)

   • The authors claim that Crescendo's average ASR is 29–61% higher than other techniques on GPT-4 and 49–71% higher on Gemini-Pro; under the binary measure, Crescendo jailbreaks 49/50 tasks on GPT-4, compared with 43 for the next-best MSJ.
   • Exception: Figure 6a shows one task on which all automated techniques fail against GPT-4 (the authors do not explain why).
   • Manual Crescendo (Table 2): 7 systems/models succeeded on almost all of 10 tasks; the exceptions are the Meth task for Gemini Ultra, which failed (×), and the Meth and Intimacy cells for Gemini Pro, marked (✓*), indicating that the jailbreak succeeded but triggered post-output filtering (the authors verified this using the Gemini-Pro API without filters).

   Generalization and Other Tasks

   • Cross-model generalization: on 12 AdvBench tasks, Crescendomation achieved at least 20% ASR on all 12 tasks for all 5 models, with ASR reaching 100% on multiple tasks (Figure 7).
   • 15-task comparison: LLaMA-2 70b and Claude-3 had slightly lower overall ASR and higher refusal rates; the Explicit task was difficult for almost all models; the Denial task had 100% ASR for all models (Figure 8, Figure 16). The authors note that manual Crescendo had previously succeeded in jailbreaking LLaMA-2 70b on Manifesto and Explicit, whereas Crescendomation did not, and state that the automated method can still be improved.
   • HarmBench: on 100 random tasks (GPT-4), Crescendo achieved an average ASR of 63.2%, with at least one success on 91% of tasks; MSJ achieved 38.9% and 70%, respectively (Figure 9).
   • Recovering failures: the authors state that for the 9 tasks on which Crescendomation failed, manual Crescendo succeeded on all of them.

   Mechanism, Transfer, and Defenses

   • Mechanism (LLaMA-2 70b): Sentence B alone achieved a 36.2% success rate, and 99.99% when preceded by Sentence A; Sentence C following B achieved only 17.3%, while A→B→C achieved 99.9%; replacing C with C', which directly states the task, lowered the success rate to below 1% (Table 3). Figure 5 shows that adding model-generated content sentence by sentence gradually increases the probability of "Sure", with the 4th sentence having the largest effect, but removing that sentence still raises the probability to 100%; on this basis the authors argue that what matters is the generated context as a whole rather than any single sentence.
   • Transferability: directly using Crescendo questions that succeeded on other models against GPT-4 and Gemini-Pro (Figure 12): for the Election task, ASR was at least 90% on both models, whereas for Explicit and Manifesto it was close to 0%; the authors state that Crescendo tailored to the target outperforms transfer, and that because some questions quote the original model's output and were not modified, the results are a lower bound on transferability.
   • Defense evaluation: Self-Reminder [30] and Goal Prioritization [34] were tested against GPT-4 (Figure 13): both defenses reduced ASR on tasks such as Toxin and Explicit, but had little effect on Election, Climate, and Stabbing; after increasing rounds and backtracking from 10 to 20, the Meth task was successfully jailbroken under Self-Reminder. The authors caution that an ASR above 0% means at least one run succeeded. Goal Prioritization could not be run for 30 rounds due to token cost and the 32k limit.

   Other Ablations and Analyses

   • Minimum number of rounds (Table 5): most tasks were jailbroken successfully in an average of within 5 rounds; Explicit for GPT-3.5 and Llama-2, and Manifesto for Llama-2, were ∞ (not successful).
   • Model scale (Figure 10): LLaMA-2 70b and 7b performed almost identically against Crescendo, with Manifesto and Explicit fully resisted by both and ASR of at least 20% on the remaining tasks; the authors call this similarity unexpected and argue that, for LLaMA-2, alignment fragility is uncorrelated with model scale.
   • Number of backtracking steps (Figure 11, GPT-4): the effect of backtracking varies by task—Election still had 100% ASR with 0 backtracking, while Manifesto was very difficult with 0 backtracking; increasing rounds from 10 to 20 partially compensates for reduced backtracking but is not as good as backtracking itself; tasks such as Soothing and Choice showed higher ASR with more rounds.
   • External API scores (Appendix Tables 6–8): the highest scores from each scoring system and the category labels are listed by model, and the authors state that the external APIs validate the effectiveness of Crescendomation.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list four limitations (no historical system, API dependency, alignment of the attacking LLM, manual results are only examples) and propose three directions for mitigation; the experiments cover 9 target models.

   Limitations and Future Directions Noted by the Authors

   • Systems without history are more resistant: Crescendo is inherently a multi-turn jailbreak, and systems lacking a history feature may be naturally more resistant; however, the authors note that supporting chat functionality requires maintaining history (6.1 Limitations).
   • Requires API access: Crescendomation requires the target model's API, so Claude-2 could not be evaluated (6.1).
   • Alignment of the attacking LLM itself: Crescendomation is based on GPT-4, and the attacking LLM sometimes refuses or resists generating attacks or performing evaluation tasks (6.1).
   • Manual results are only examples: the manual results in Section 3.2 do not cover the full potential of Crescendo; the authors state that in practice it can be used for more tasks and may be more effective (6.1).
   • Mitigation directions: pre-filtering suspicious content during training (the authors note this does not apply to misinformation-type tasks and that retraining is costly); using Crescendomation-generated data for alignment; applying content filtering to inputs and outputs (the authors note that comprehensive filtering is difficult and that character substitution further increases detection difficulty) (6.2 Mitigation).
   • The authors' recommendation: system designers should assume at deployment that current LLMs will eventually be jailbroken, and content-filtering defenses can be designed for the worst case (6.3 Status of LLM Safety).

   Scope of Experimental Coverage

   • There are 9 target models tested: GPT-3.5, GPT-4, Gemini-Pro, Gemini Ultra, Claude-2, Claude-3 (Opus/3.5 Sonnet appear in the manual experiments), LLaMA-2 70b, LLaMA-3 70b, and LLaMA-2 7b (5.1.1, Table 2, 5.4).
   • The attacking model and all evaluation models (Judge, Secondary Judge, Refusal Judge) are GPT-4; the paper does not use other models in these two roles for comparison (5.1.2, 5.1.3).
   • Benchmark coverage: 15 self-built tasks, a 50-task AdvBench subset, a 12-task AdvBench subset, and 100 random HarmBench tasks (5.1.4, 5.3).
   • Defense evaluation covers only two defenses, Self-Reminder and Goal Prioritization, and only on GPT-4 (5.7).
   • The paper does not report the sample size of manual review or numerical agreement between it and the Judge's determinations; Crescendomation defaults to 10 independent runs and 10 rounds of interaction (5.1.5).

   Collapse this answer

6. 06Summarize the main content of the paper

   Crescendo uses benign multi-turn dialogue and the model's own replies to gradually jailbreak it; the authors claim it outperforms existing automated jailbreaks on multiple models and benchmarks, and discuss the difficulties of mitigation.

   A multi-turn jailbreak attack paper: the authors propose Crescendo—starting from a harmless topic related to the task and escalating gradually by quoting the model's own replies turn by turn, without using malicious text at any point, thereby bypassing alignment and filtering aimed at single-turn malicious inputs; they also implement the automated tool Crescendomation (GPT-4 generates questions; Judge/Secondary Judge/Refusal Judge evaluate and backtrack; the authors state it has been integrated into PyRIT).

   • Problem to solve: the authors argue that existing alignment and evaluations cover only single-turn jailbreaks, and that once a single-turn jailbreak is discovered, it can be defended against by input filters.
   • Manual effectiveness: succeeded on almost all of 10 tasks across 7 systems—ChatGPT (GPT-4), Gemini Pro/Ultra, Claude-2/3, and LLaMA-2/3 70b (Table 2); the exception is the Meth task for Gemini Ultra.
   • Automated comparison: on the 50 AdvBench tasks, average ASR (binary ASR) was 56.2% (98.0%) against GPT-4 and 82.6% (100.0%) against Gemini-Pro, which the authors claim exceeds MSJ/CIA/PAIR/CoA by 29–61% and 49–71%, respectively (Table 4).
   • Generalization: average ASR of 63.2% on 100 HarmBench tasks (MSJ: 38.9%); at least 20% ASR on 12 tasks for all 5 models; most tasks succeeded within an average of 5 rounds (Table 5).
   • Mechanism evidence: on LLaMA-2 70b, preceding benign context raised the success rate of the key step from 36.2% to 99.99%, while replacing the final turn with a direct statement of the task lowered it to below 1% (Table 3); on this basis the authors argue that what matters is the model-generated context as a whole.
   • Defenses and conclusion: Self-Reminder and Goal Prioritization reduced ASR only on some tasks, and some tasks were still jailbroken after increasing rounds and backtracking; the authors recommend assuming at deployment that LLMs will eventually be jailbroken, and propose three mitigation directions—training data filtering, alignment using Crescendomation data, and input/output filtering—while noting the difficulties of each.

   Collapse this answer

Related papers