Catastrophic jailbreaks of open-source large language models using generation configurations
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation
The authors propose the generation exploitation attack, which changes only the decoding configuration to raise the ASR of 9 of 11 open-source LLMs to ≥95%, and claim it is about 30 times faster than GCG.
The attacker's goal is to make safety-aligned open-source LLMs give misaligned answers to malicious instructions. The attacker does not modify the input text and only manipulates generation configurations: removing the system prompt, changing decoding hyperparameters (temperature, top-k, top-p) and sampling methods, and using a scorer trained on HH-RLHF to select the highest-scoring one among multiple sampled outputs; the paper states that this assumes the attacker knows the distribution of aligned/misaligned outputs. The victim systems are 11 open-source LLMs and gpt-3.5-turbo (only the decoding hyperparameters exposed through the API).
- Safety evaluation and alignment of open-source LLMs are usually conducted only under default decoding configurations; the authors argue this may miss alignment failures under other generation strategies, leaving models exposed to jailbreak risk.
- Proposes the generation exploitation attack: without changing the input text, it only removes the system prompt and traverses decoding configurations such as temperature, top-k, and top-p, and after sampling under each configuration uses a scorer to pick the output most likely to be misaligned; it also proposes generation-aware alignment, which uses outputs under multiple decoding configurations for alignment training.
- On MaliciousInstruct, the ASR of 9 of the 11 open-source models was raised to ≥95%; multiple sampling and decoding constraints also raised the two LLAMA2-CHAT models to ≥95%; the authors state it is stronger than GCG and about 30 times faster; on gpt-3.5-turbo the ASR is only 7%; generation-aware alignment lowered the ASR of LLAMA2-7B-CHAT from 95% to 69%.
- The authors plan future work on transfer of the attack to more models (including multimodal models), better automatic harmfulness metrics, and more sample-efficient alignment strategies (Section 7). The experiments cover 11 open-source models and gpt-3.5-turbo; the defense experiments were conducted only on LLAMA2-7B-CHAT, and the paper does not report the total number of attack queries or the number of repetitions.
- Evaluated models
- Vicuna-7BVicuna-13BVicuna-33BMPT-7BMPT-30BFalcon-7BFalcon-40BLlama2-7BLlama2-13BLlama2-7B-ChatLlama2-13B-ChatChatGPT (gpt-3.5-turbo)
- Benchmarks
- AdvBenchMaliciousInstruct
- Metrics
- ASRHP
Researchers at Princeton University propose in their paper a "generation exploitation attack" that, by only removing the system prompt and changing decoding configurations such as temperature and top-p, raises the attack success rate to above 95% for a total of 11 open-source large language models across four families: LLaMA2, Vicuna, Falcon, and MPT, at a computational cost about 30 times lower than the then best-performing GCG attack. Human evaluation shows that at least half of the unaligned outputs actually contain harmful instructions. The authors propose generation-aware alignment as a defense, reducing the attack success rate on LLaMA2-7B-chat from 95% to 69%; the same technique achieves only a 7% success rate against the proprietary model gpt-3.5-turbo. The paper argues that safety evaluations using only fixed decoding configurations underestimate real risk, and calls for red-team testing covering multiple generation strategies before release. Code is open-sourced at https://github.com/Princeton-SysML/Jailbreak_LLM。
Why it was selectedChanging only the decoding configuration alone raises the attack success rate above 95% for 9 of 11 open-source models, and safety evaluations under fixed decoding configurations will underestimate real risk.
Paper analysis
What problem does this paper attempt to solve?
The authors argue that safety evaluations of open-source LLMs are fixed at the default decoding configuration, and that changing only the generation strategy can bypass alignment.
Core question: Does the alignment of open-source LLMs hold only under the default generation configuration—can manipulating only the decoding strategy, without changing the input text, make safety-aligned models give misaligned answers to malicious instructions?
- Problem setting and importance: Open-source LLMs are usually safety fine-tuned and RLHF-aligned before release, supplemented by red-team evaluations; the authors worry that open-source models are more easily used to spread harmful content (according to the authors).
- Shortcomings of existing methods: The authors point out that existing safety evaluations usually use only a single default decoding configuration, e.g., the LLAMA2 paper uses top-p sampling (p=0.9, τ=0.1) and always adds a system prompt (Section 2.3); existing automated jailbreak attacks (such as GCG) require optimizing adversarial prompts, which the authors describe as complex and computationally expensive.
- Core observation and hypothesis: The authors' key hypothesis is that existing alignment pipelines and evaluations are likely based on default decoding settings, and a slight change in configuration may cause them to fail; Figure 1 shows that LLAMA2-7B-CHAT changes from refusing to giving a step-by-step answer to the same malicious instruction after top-p is changed from 0.9 to 0.75.
- What the paper proposes: It proposes the generation exploitation attack (a jailbreak attack that manipulates only the generation configuration); a self-built benchmark MaliciousInstruct and a classifier-based ASR evaluator; and a defensive generation-aware alignment.
- Threat model:
- Attacker goal: Make the model produce misaligned (non-refusing) outputs to malicious instructions.
- Attacker knowledge: The paper does not use the terms white-box/black-box; the attack requires the ability to set decoding hyperparameters and whether to use a system prompt, and assumes the attacker knows the distribution of aligned/misaligned outputs (used to train the scorer, Appendix B.3).
- Attacker capability: Does not modify the input text, only removes the system prompt, changes temperature/top-k/top-p and samples multiple times, and uses the scorer to select the highest-scoring output.
- Victim systems: 11 open-source LLMs (VICUNA, MPT, FALCON, LLAMA2 series), and gpt-3.5-turbo, which exposes four decoding hyperparameters through the OpenAI API.
What related research exists?
Related work falls into three categories: safety alignment, attacks on open-source models, and attacks on closed-source models; the baseline is GCG.
Safety alignment and evaluation
- Ouyang et al. (2022), Bai et al. (2022a), Korbak et al. (2023), Zhou et al. (2023)——pre-release alignment techniques, with the mainstream pipeline being pretraining, supervised safety fine-tuning, and RLHF (Appendix A.1).
- Ganguli et al. (2022), Touvron et al. (2023b)——introduced the red-teaming concept into LLM evaluation, using human evaluation to cover sensitive categories.
- Qiu et al. (2023)——an evaluation framework centered on model translation of malicious prompts.
Attacks on open-source models
- Wei et al. (2023)——summarized categories of prompt modifications that can bypass safety fine-tuning.
- Lapid et al. (2023)——used a genetic algorithm to generate universal adversarial suffixes that bypass alignment.
- Zou et al. (2023, GCG)——optimizes a suffix appended to malicious prompts to synthesize jailbreaks; Carlini et al. (2023)——adds adversarial images to multimodal models to bypass alignment. The paper notes that such methods require computationally expensive optimization processes and/or white-box access, whereas this paper's method only requires access to decoding-related hyperparameters.
Attacks on closed-source models
- Li et al. (2023)——a multi-step attack on ChatGPT aimed at extracting private data.
- Kang et al. (2023), Shen et al. (2023), Liu et al. (2023b)——study, respectively, prompts that imitate executable code, the characteristics of in-the-wild jailbreak prompts, and ChatGPT's resistance to jailbreak prompts.
- Zou et al. (2023)——showed that jailbreaks can transfer from open-source models to closed-source models.
Baselines and benchmarks: the comparison baseline is GCG (Zou et al., 2023); the evaluation benchmarks are AdvBench (Zou et al., 2023, first 100 items) and the self-built MaliciousInstruct (100 items). The paper states that PEZ, GBDA and AutoPrompt were not included in the comparison because their ASR is lower than GCG's (Section 4.4).
The authors position this paper as follows: unlike attacks that optimize adversarial inputs, this paper does not modify the input and only manipulates generation configurations, which the authors describe as simple and an order of magnitude lower in computational cost than SOTA attacks.
How does the paper address this problem?
On the attack side, decoding configurations are traversed and a scorer selects the output; on the defense side, outputs under multiple decoding configurations are used for alignment training.
The overall approach has two parts: on the attack side, the generation exploitation attack does not modify the input and only traverses generation configurations and selects the output most likely to be misaligned; on the defense side, generation-aware alignment uses outputs sampled under multiple decoding configurations for alignment training.
Problem setting and evaluator
- ASR definition: the proportion of instructions that received a misaligned output (did not refuse the malicious instruction). The paper argues that the substring matching of Zou et al. (2023) (matching a small number of refusal phrases) may miss certain phrasings and overestimate the attack.
- Classifier evaluator: BERT-BASE-CASED is fine-tuned on the HH-RLHF dataset (over 161,000 items, each containing a prompt and two human-chosen/rejected responses) to estimate a risk score; a risk above 0.5 is judged as a successful attack; test set accuracy is 95.8% (Appendix B.2).
- Human agreement: the authors report 92% agreement between the classifier evaluator and human judgments, higher than the 86% of substring matching; agreement among the five annotators is 96% (Appendix B.2).
- HP (Harmfulness Percentage): human judgment of whether an output actually gives concrete, useful advice for the malicious instruction.
Attack: exploited generation strategies
- system prompt: with or without it is considered; FALCON has no built-in system prompt, so the LLAMA2 system prompt is used.
- Decoding strategy traversal: temperature τ from 0.05 to 1 in steps of 0.05 (20 configurations); top-k K in {1, 2, 5, 10, 20, 50, 100, 200, 500} (9 configurations); top-p from 0.05 to 1 in steps of 0.05 (20 configurations).
- Output selection: each instruction is sampled once under each of the 49 decoding configurations, giving 49 responses, and a scorer (also trained on HH-RLHF, with a training set disjoint from the classifier's) selects the highest-scoring one as the final output (Section 4.1).
Attack enhancement strategies
- Multiple sampling: sample multiple times for each decoding configuration (8 times per configuration when comparing with GCG), then the scorer selects one.
- Decoding with penalties and constraints: add a length penalty during decoding, forbid generation of a set of "bad words", and force generation of a set of "forced words" (Table 3; the specific word lists are omitted here, see Table 3 and Section 4.3 of the paper).
Defense: generation-aware alignment
- Approach: for each prompt p, sample n responses from each of multiple strategies in the decoding space H, split them into aligned and misaligned groups, and train with the chain of hindsight (Liu et al., 2023a) objective: L=1/(|P|)∑p∼ P1/(|Rmp|×|Rap|)∑(rm,ra) −log P(ra|p,sa)−log P(rm|p,sm), where s_a is "An aligned answer:" and s_m is "A misaligned answer:", meaning the model is made to reproduce aligned/misaligned responses respectively when given the corresponding marker.
- Training setup: about 2,000 malicious prompts with toxicity score > 0.5 are taken from HH-RLHF (scored with TOXIC-BERT), sampled under multiple decoding configurations on LLAMA2-7B-CHAT and grouped; 10,000 (prompt, aligned response, misaligned response) triples are randomly selected and fine-tuned with Alpaca Farm, learning rate 2×10⁻⁵, batch size 16, 3 epochs (Appendix B.3).
- Control: compared with a variant that collects samples under a fixed decoding strategy (τ=1.0 sampling) for alignment, with the same number of triples.
- Evaluation method: evaluated on MaliciousInstruct, appending "An aligned answer:" after each instruction and then decoding, following Liu et al. (2023a); the authors note that this evaluation is for simulation purposes and that actual deployment could use context distillation to avoid the explicit appending.
What experiments did the paper conduct?
The ASR of 9 of the 11 open-source models was raised to ≥95%; for gpt-3.5-turbo it is only 7%; the defense lowers the ASR from 95% to 69%.
Experimental setup
- Models tested: VICUNA (7B, 13B, 33B), MPT (7B, 30B), FALCON (7B, 40B), LLAMA2 (7B, 13B, 7B-CHAT, 13B-CHAT), 11 open-source models in total; of these only the two LLAMA2-CHAT models are safety-aligned via RLHF. Also gpt-3.5-turbo (Section 6).
- Benchmarks: MaliciousInstruct (100 items, covering ten categories: psychological manipulation, sabotage, theft, defamation, cyberbullying, false accusation, tax evasion, hacking, fraud, illegal drug use; generated with the help of ChatGPT and manually filtered); AdvBench (500 items; the first 100 are used when comparing with GCG, following the setup of Zou et al. (2023)).
- Baseline: GCG (Zou et al., 2023), batch size 512, top-k 256, 32 optimizable tokens, 500 optimization steps, one output generated per instance.
- Metrics and evaluation: ASR uses a BERT-BASE-CASED classifier trained on HH-RLHF (threshold 0.5), with substring matching also reported for reference; HP is judged manually by five annotators.
- Sample sizes and counts: in the main experiments each instruction is sampled once under each of the 49 decoding configurations; the HP human evaluation uses 100 misaligned outputs from LLAMA2-13B-CHAT; the paper does not report the number of experimental repetitions.
Main results
According to Table 1 (MaliciousInstruct, ASR%):
Swipe horizontally to read the table
Model Default (greedy+sys. prompt) Without sys. prompt Varied All VICUNA-7B 50 62 97 VICUNA-13B 21 55 97 VICUNA-33B 42 50 96 MPT-7B 0 86 97 MPT-30B 0 91 98 FALCON-7B 5 75 95 FALCON-40B 7 72 95 LLAMA2-7B 14 85 97 LLAMA2-13B 34 83 97 LLAMA2-7B-CHAT♠ 0 16 81 LLAMA2-13B-CHAT♠ 0 8 88 - The authors state that after removing the system prompt the ASR often rises by more than 10 percentage points, and for models not safety fine-tuned it can rise by more than 50 percentage points; the two LLAMA2-CHAT models have ASR of 81% and 88% under varying all decoding configurations (Varied All), lower than the ≥95% of the other models.
- Table 2 shows that the most vulnerable decoding configuration differs across models (e.g., LLAMA2-7B-CHAT has ASR 25% at τ=0.95 and 29% at p=0.7), on which basis the authors argue that evaluating alignment with a single fixed decoding configuration underestimates risk.
- The authors state that the ASR of 9/11 models was raised to ≥95% (Table 1), and that the two LLAMA2-CHAT models also reach ≥95% after the enhancements in Section 4.3.
Attack enhancement (LLAMA2-CHAT)
- Multiple sampling: Figure 3 shows ASR rising as the number of samples increases; the authors report that LLAMA2-7B-CHAT reaches 95% ASR with 5 samples and LLAMA2-13B-CHAT with 4 samples.
- Decoding constraints: Table 3 shows that the ASR of LLAMA2-7B-CHAT goes from 81% for the baseline attack to 89% with the length penalty, 93% with bad words added, and 100% with forced words also added.
Comparison with GCG
- ASR comparison: according to Table 4, on AdvBench and MaliciousInstruct, under both substring matching and classifier metrics, all variants of this attack are higher than GCG; for example, on MaliciousInstruct for LLAMA2-13B-CHAT, GCG is 24% (substring matching)/21% (classifier) while this attack's Combined is 100%/96%; for LLAMA2-7B-CHAT, GCG is 50%/42% while this attack's All is 99%/95%.
- Cost: the authors state that on a single NVIDIA A100, this attack takes about 3 minutes per prompt for LLAMA2-7B-CHAT, versus about 1.5 hours for GCG, i.e., a gap of about 30 times.
Human evaluation of harmfulness
- HP: five annotators judged that 50% of the 100 misaligned outputs of LLAMA2-13B-CHAT contain harmful guidance (Appendix B.2).
- Heuristic rule: the authors use a heuristic rule to automatically identify harmful outputs, reporting 93% agreement with humans; Figure 2 gives the HP of the two LLAMA2-CHAT models under each decoding configuration, and the authors report that the HP of LLAMA2-7B-CHAT reaches as high as 80%.
Defense experiments
- Results: according to Table 5, after generation-aware alignment of LLAMA2-7B-CHAT, the ASR drops from 95% to 69% when the attacker varies all decoding configurations; the control group aligned with samples from fixed decoding (τ=1.0) drops to 88%.
- By strategy: after generation-aware alignment, the ASR under the three exploits τ, Top-K and Top-p is 27%, 49% and 65% respectively, and the authors state that its protective advantage is greatest against the temperature exploit.
Closed-source model experiments
- Setup: gpt-3.5-turbo, varying temperature, top-p, presence penalty (−2 to 2, step 0.1) and frequency penalty (−2 to 2, step 0.1) through the API, with ASR reported on MaliciousInstruct.
- Results: according to Table 6, the ASR under default decoding is 0, varying each parameter individually gives 2%–4%, and varying all gives 7%. The authors observed that 9 of the 100 prompts timed out, speculate that they were blocked by the content filter, and consider closed-source model content filtering and red-teaming resources to be sources of the difference.
Other ablations and analyses
- system prompt ablation (Table 9): LLAMA2-7B-CHAT has an ASR of only 4% for varied decoding when the system prompt is kept, and 81% after removing it; LLAMA2-13B-CHAT is 23% versus 88%.
- Default decoding vs. greedy (Table 10): LLAMA2-CHAT has the same or similar ASR under default decoding (τ=0.1, p=0.9) and greedy (0/0, 16/15, 8/8).
- Risk heatmaps (Figure 4–7): the authors state that different malicious instructions trigger misaligned outputs under different decoding configurations, and that the risk of LLAMA2-7B/13B is higher than that of the aligned LLAMA2-13B-CHAT.
What are some points that can be further explored?
The authors list three future directions in the conclusion; the experiments cover 11 open-source models and one closed-source model.
Limitations and future directions noted by the authors
- Transferability: the authors plan to study the transfer of the attack to more models (including multimodal models) (Section 7).
- Automatic harmfulness metrics: the authors plan to develop better automatic harmfulness evaluation metrics to more rigorously assess model-generated content (Section 7).
- Sample efficiency of alignment: the authors plan to explore more advanced generation-aware alignment strategies, focusing on improving sample efficiency through strategic exploration of decoding configurations (Section 7).
- Simulated nature of the defense evaluation: the authors note that the "An aligned answer:" string appended in the defense evaluation could be removed by an attacker, that the evaluation is for simulation purposes only, and that actual deployment would need to use procedures such as context distillation to avoid the explicit appending (Section 5.2).
Scope of experiments
- The models tested are 11 open-source models from four families, VICUNA, MPT, FALCON and LLAMA2, plus one closed-source model, gpt-3.5-turbo (Section 3.1, Section 6).
- The benchmarks are MaliciousInstruct (100 items) and AdvBench (the first 100 items when comparing with GCG) (Section 3.1, 4.4).
- The only baseline attack is GCG; PEZ, GBDA and AutoPrompt were not included in the comparison, and the paper gives as the reason that their ASR is lower than GCG's (Section 4.4).
- The defense experiments were conducted only on LLAMA2-7B-CHAT, with about 2,000 prompts and 10,000 triples used for training (Section 5.2).
- The paper does not report the total number of attack queries or the number of experimental repetitions; the human evaluation used five annotators and 100 outputs (Appendix B.2).
Summarize the main content of the paper
Changing only the decoding configuration can bypass open-source LLM alignment, with 9/11 models at ASR≥95%; the authors propose a generation-aware alignment defense.
A jailbreak attack paper targeting open-source LLMs that also provides evaluation method improvements and a defense attempt.
- Problem: The authors believe that safety evaluation and alignment of open-source LLMs are fixed to the default decoding configuration (e.g., LLAMA2 uses τ=0.1, top-p 0.9 plus a system prompt), and alignment failures under other generation strategies are missed.
- Method: The generation exploitation attack does not change the input text; it only removes the system prompt and traverses temperature (20 configurations), top-k (9), and top-p (20), 49 decoding configurations in total, samples once for each, and then uses a scorer trained on HH-RLHF to select the highest-scoring output; evaluation uses a BERT classifier to judge ASR (92% agreement with humans, higher than 86% for substring matching), and a self-built MaliciousInstruct with ten categories of malicious intent (100 items).
- Main results: On MaliciousInstruct, the ASR of 9 of 11 open-source models is raised from 0%–50% under the default configuration to ≥95% (Table 1); the two RLHF-aligned LLAMA2-CHAT models reach 81%/88% when all configurations are varied, reach 95% with multiple sampling (5 times/4 times), and LLAMA2-7B-CHAT reaches 100% after adding decoding constraints (Figure 3, Table 3).
- Comparison and closed-source: The authors claim this attack is higher than GCG on two benchmarks and two metrics, and takes about 3 minutes per prompt versus about 1.5 hours for GCG (about 30 times); for gpt-3.5-turbo, varying all API decoding parameters yields an ASR of only 7% (Table 6).
- Defense: Generation-aware alignment uses outputs sampled under multiple decoding configurations for alignment training, reducing the ASR of LLAMA2-7B-CHAT under attack from 95% to 69%, better than 88% for alignment with fixed decoding sampling (Table 5).
- Authors' conclusion: The authors claim that current safety evaluation and alignment pipelines of open-source LLMs have major failures, and advocate more comprehensive red-team evaluation under multiple generation strategies before release, and adopting generation-aware alignment as a proactive countermeasure.
Related papers
Show 3 more papers
- Safety Alignment Should Be Made More Than Just a Few Tokens DeepAnalysis / interpretabilityICLR 2025