Defensive Prompt Patch: A Robust and Interpretable Defense Against Jailbreak Attacks on Large Language Models
Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
The authors propose DPP: using HGA to search for a defense suffix appended after the query, achieving an average ASR of 3.80% and a Win-Rate of 82.98% on Llama-2-7B-Chat.
Attackers use jailbreak prompts (GCG, AutoDAN, PAIR, TAP, ICA, Base64, Catastrophic, etc.) to bypass alignment; in the adaptive setting, attackers can query the defended model and know that a defense mechanism exists but do not know the specific DPP (DPP is a system prompt that a closed-source service provider appends to queries). The defender is the model service provider, and no model retraining is required.
- Aligned LLMs can still be bypassed by jailbreak attacks that circumvent safety guardrails. The authors argue that existing prompt-based defenses struggle to balance safety and utility: Goal Prioritization incurs a large utility loss, RPO has limited defense coverage, and Self-Reminder cannot find the optimal prompt on better-aligned models.
- Proposes Defensive Prompt Patch (DPP): targeting the weighted sum of the refusal score and the helpfulness score, uses a Hierarchical Genetic Algorithm (sentence-level synonym substitution plus paragraph-level crossover and mutation, GPT-4 rewriting) to iteratively optimize a human-readable suffix prompt, which is appended to each user query at inference time without retraining the model.
- On Llama-2-7B-Chat, DPP has a non-adaptive average ASR of 3.80% and a Win-Rate of 82.98%, and an adaptive average ASR of 13.0%, both the lowest among the evaluated defenses; on Mistral-7B-Instruct-v0.2, the non-adaptive average ASR is 2.0%, but the Win-Rate drops to 75.06%, a higher utility cost than Self-Reminder and System Prompt.
- The authors point out: HGA training is computationally expensive and has limited scalability; training depends on GPT-4 rewriting, costing about 75 USD; when running open-source models locally, users can remove DPP, so it cannot protect. Experiments cover only 7B–13B open-source models, and the number of repetitions and statistical tests are not reported.
- Evaluated models
- Llama-2-7B-ChatMistral-7B-Instruct-v0.2Starling-7B
- Benchmarks
- AdvBenchAlpacaAlpacaEvalJailbreakBench Chat (JBC)MaliciousInstruct
- Metrics
- ASRAverage ASRWin-RateMin Over PromptPerplexity
Defensive Prompt Patch (DPP) is a prompt-based jailbreak defense method for large language models that uses interpretable suffix prompts to defend against standard and adaptive jailbreak attacks. Experiments on LLAMA-2-7B-Chat and Mistral-7B-Instruct-v0.2 show that the method significantly reduces the attack success rate (ASR) with negligible impact on utility. The authors state that it outperforms existing defenses in balancing safety and utility, and that it is extensible and applicable to various LLM platforms.
Paper analysis
What problem does this paper attempt to solve?
How to make a prompt-level jailbreak defense achieve both low ASR and low utility loss, and remain robust to adaptive attacks.
Core question: how to design a prompt-level defense that reduces the jailbreak attack success rate (ASR) to a very low level while barely harming LLM utility.
- Scenario and importance: the authors note that many seemingly aligned LLMs are quickly shown to be vulnerable to jailbreak attacks—attackers add jailbreak prompts to malicious queries to bypass safety guardrails—so hardening is needed without retraining the model.
- Shortcomings of existing methods (according to the paper): Goal Prioritization defends strongly but greatly harms utility; RPO preserves utility but has limited defense coverage; Self-Reminder balances better, but due to flaws in its search algorithm, it performs poorly on better-aligned models such as Llama-2-7B-Chat (Table 1 gives a qualitative comparison).
- Core idea: use a Hierarchical Genetic Algorithm (HGA) to search for a human-readable defense suffix at two levels—sentence-level word replacement and segment-level crossover and mutation—with the optimization objective being the weighted sum of the refusal score and the helpfulness score.
- Paper output: proposes the Defensive Prompt Patch (DPP), evaluated on Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2 against standard and adaptive jailbreak attacks; the authors claim it outperforms existing prompt-level defenses on the safety-utility balance.
- Threat model:
- Attacker objective: use jailbreak prompts (GCG, AutoDAN, PAIR, TAP, ICA, Base64, Catastrophic) to make the model output harmful content.
- Attacker knowledge (adaptive setting): can query the defended model and knows that a defense mechanism exists, but does not know the specific DPP content—DPP is set as a system prompt appended after the query by a closed-source service provider.
- Victim system: an open-source LLM service with a prompt-level defense deployed (Llama-2-7B-Chat, Mistral-7B-Instruct-v0.2, etc.).
- Defender assumption: the defense is deployed by the model service provider and requires no model retraining or architecture knowledge.
What related research exists?
Related work falls into two categories: jailbreak attacks and prompt defenses; the baselines are Self-Reminder, Goal Prioritization, RPO, and System Prompt.
Jailbreak attacks
- GCG (Zou et al., 2023): gradient-based optimization of adversarial suffixes; Base64 (Wei et al., 2023a): encodes malicious queries as Base64 to bypass filtering. Both are non-interpretable attacks.
- AutoDAN (Liu et al., 2023): uses a genetic algorithm to generate readable jailbreak prompts; PAIR (Chao et al., 2023) and TAP (Mehrotra et al., 2023): use an attacker LLM to automatically generate jailbreak prompts, with black-box access to the target model.
- ICA (Wei et al., 2023b): uses in-context learning to induce misaligned responses; Catastrophic Attack (Huang et al., 2023): manipulates the generation configuration to trigger misaligned outputs.
Prompt-level defenses
- Self-Reminder (Xie et al., 2023): modifies the system prompt to make the model more self-aware in refusing; the paper notes that on Llama-2-7B-Chat, because the model is too well aligned and often refuses to improve the defense prompt, its training effect is poor.
- RPO (Zhou et al., 2024): the optimization objective minimizes the distance between harmful queries and safe responses; the paper notes that its training cost is high (it follows the GCG training procedure) and that its defense on Starling-7B is insufficient (Table 14), so it is not used as a baseline for Mistral.
- Goal Prioritization (Zhang et al., 2023b) and Default System Prompt (Zheng et al., 2024a): guide the model to prioritize safety; the paper notes that Goal Prioritization is weaker against AutoDAN, TAP, and PAIR, and that its context is long, inference becomes slower, and the utility loss is large.
Baselines and benchmarks: the compared baselines are Self-Reminder, Goal Prioritization, RPO (Llama-2 only), and System Prompt (Mistral only); evaluation uses AdvBench, Alpaca, AlpacaEval (Win-Rate), and JailbreakBench Chat.
Positioning of this paper: the authors claim that DPP is the only defense that simultaneously has an optimizable prompt, logits-based search, and human readability (Table 1), and that it outperforms the above methods on the safety-utility balance.
How does the paper address this problem?
Using HGA, under a weighted-sum objective of the refusal score and the helpfulness score, iteratively search for a defense suffix appended after the query.
Overall approach: formalize defense as a defensive prompt patch (DPP) appended after a (jailbreak) query, use an AutoDAN-inspired Hierarchical Genetic Algorithm to iteratively optimize it under a dual objective, and finally select the suffix with the highest total score, which is appended after every user query at inference time.
Problem Formulation
- Attack objective: a jailbreak input ũ1:m causes the model to output a harmful response r1:k; the attack maximizes P(r1:k|ũ1:m).
- Defense objective: after appending patch d1:l, the model outputs a refusal s1:n; the defense maximizes P(s1:n|ũ1:m ⊕ d1:l); at the same time, the probability of normal responses to benign queries should remain as unchanged as possible.
Dual-Objective Scoring
- Refusal score: SDi = log P(s1:n | ũ1:m ⊕ d1:l), i.e., the log-likelihood of the refusal response given the jailbreak query plus patch.
- Helpfulness score: SHi = log P(h1:q | b1:p ⊕ d1:l), i.e., the log-likelihood of the normal response given the benign query plus patch.
- Total score: STi = α · SDi + β · SHi, where α and β trade off defense and utility; in the implementation, Llama-2-7B-Chat uses α=1, β=10, and Mistral-7B-Instruct-v0.2 uses α=10, β=1 (Appendix C).
HGA Optimization Process
- Initialization: hand-written prototype prompts (e.g., Self-Reminder-style suffixes) are rewritten K times by GPT-4 to generate a candidate population; the suffix form is used by default (the ablation in Appendix B.2 supports suffixes over prefixes).
- Sentence-level word replacement: score sentences according to Eq. 6; words inherit the sentence score and are adjusted by occurrence frequency; words exceeding the threshold are randomly replaced with synonyms according to weight.
- Segment-level crossover and mutation: exchange parent sentence segments with probability pswap, and rewrite sentences with GPT-4 with probability pmutate, increasing population diversity.
- Generalization selection: sequentially use N pairs of refusal/helpful data to iteratively optimize the same candidate set, and finally take the one with the highest total score as the Best DPP (Algorithm 1, 5).
Key Hyperparameters
- Optimization settings: num_steps=100, batch_size=64, num_elites=0.1, crossover_rate=0.5, mutation_rate=0.01, 5 sentence-level iterations, 1 segment-level iteration (Appendix C).
- Training cost: a single 80GB A800 GPU; the authors report that training one DPP on Llama-2-7B-Chat costs about 75 USD (GPT-4 calls, Appendix C).
Success Determination and Evaluation Method
- ASR: keyword matching determines whether a jailbreak occurred (Appendix I lists the refusal keyword sets corresponding to each attack; the Catastrophic attack uses its official evaluator); Llama-Guard is additionally used as an LLM judge for review (Appendix L).
- Win-Rate: compared with the reference model Davinci003 under the AlpacaEval framework, with response length fixed at 1000 and no system prompt used (Appendix J).
What experiments did the paper conduct?
On Llama-2-7B-Chat and Mistral-7B-Instruct-v0.2, measure the non-adaptive and adaptive ASR and Win-Rate of 6–7 attacks, and conduct generalization and ablation studies.
Experimental Setup
- Models tested: Llama-2-7B-Chat, Mistral-7B-Instruct-v0.2; the appendix additionally covers Vicuna-13B-v1.5, Llama-3-8B-Instruct, and Starling-7B (RPO check only).
- Datasets: AdvBench takes 100 harmful behaviors and Llama-2-7B-Chat's refusals to form the adversarial dataset; Alpaca takes 100 benign question-answer pairs as the utility dataset; another 100 are independently drawn from AdvBench for unseen-query testing; JailbreakBench Chat (JBC) serves as the external query set.
- Attacks: non-adaptive uses Base64, ICA, AutoDAN, GCG, PAIR, TAP; adaptive uses ICA, Catastrophic, GCG, AutoDAN, PAIR, TAP (Base64 is a static attack and is removed under the adaptive setting; Catastrophic is an adaptive attack newly introduced in the paper).
- Baselines: on Llama-2, RPO, Goal Prioritization, Self-Reminder; on Mistral, System Prompt, Goal Prioritization, Self-Reminder.
- Metrics and determination: ASR (keyword matching, Appendix I), Average ASR, Win-Rate (AlpacaEval, reference model Davinci003); Llama-Guard judge for review (Appendix L).
- Sample size and repetitions: 100 queries per attack (PAIR/TAP use 50 sampled from their codebase); the paper does not report the number of repetitions.
Main Results
According to Table 2 and Table 4 (non-adaptive, lower ASR is better, higher Win-Rate is better):
Swipe horizontally to read the table
Method Model Base64 ICA AutoDAN GCG PAIR TAP Average ASR Win-Rate w/o defense Llama-2 0.990 0.690 0.640 0.550 0.100 0.120 0.515 81.37 RPO Llama-2 0.000 0.420 0.280 0.190 0.060 0.060 0.168 79.23 Goal Prioritization Llama-2 0.000 0.020 0.520 0.020 0.020 0.020 0.100 34.29 Self-Reminder Llama-2 0.030 0.290 0.000 0.040 0.020 0.000 0.063 64.84 DPP Llama-2 0.010 0.000 0.100 0.040 0.040 0.040 0.038 82.98 w/o defense Mistral 0.990 0.960 0.990 0.970 1.000 1.000 0.985 90.31 Self-Reminder Mistral 0.550 0.270 0.510 0.880 0.420 0.260 0.482 88.82 Goal Prioritization Mistral 0.030 0.440 0.030 0.390 0.300 0.140 0.222 56.59 DPP Mistral 0.000 0.010 0.020 0.030 0.040 0.020 0.020 75.06 (The System Prompt row for Mistral is omitted: average ASR 0.527, Win-Rate 84.97.)
- The authors claim that DPP achieves both the lowest average ASR and the highest Win-Rate on Llama-2; the exception is that under AutoDAN, DPP (0.100) is weaker than Self-Reminder (0.000).
- On Mistral, DPP has the lowest average ASR, but its Win-Rate (75.06) is lower than Self-Reminder (88.82) and System Prompt (84.97); the authors argue that the defense gap (average ASR more than 20 percentage points lower than the best baseline) offsets the utility loss, and state that Mistral's defense-utility trade-off is worse than Llama-2's.
Adaptive Attacks
- Llama-2 (Table 3): DPP's average adaptive ASR is 0.130, lower than Self-Reminder (0.177), Goal Prioritization (0.247), and RPO (0.457); the exceptions are that under the Catastrophic attack, Goal Prioritization's 0.0033 is lower than DPP's 0.247, and under AutoDAN, Self-Reminder's 0.080 is lower than DPP's 0.110.
- Mistral (Table 5): DPP averages 0.469, lower than Goal Prioritization (0.627), Self-Reminder (0.796), and System Prompt (0.862); under adaptive GCG, Goal Prioritization (0.110) outperforms DPP (0.390).
- Llama-Guard review (Appendix L): DPP's average ASR is 0.024 on Llama-2 and 0.054 on Mistral; the authors state this is consistent with the keyword-based conclusions.
Generalization Experiments
- Unseen queries (Appendix U, 100 new AdvBench queries, adaptive): on Llama-2, DPP's average ASR is 0.075 (Table 37); on Mistral, 0.394, still the lowest but high in absolute terms (Table 38).
- Other models: on Vicuna-13B-v1.5, DPP's average adaptive ASR is 0.268 (Table 39), and the DPP transferred from Mistral is 0.27 (Table 40); on Llama-3-8B-Instruct, DPP averages 0.045, and 0 under ICA (Table 41).
- JBC queries (Appendix P): on Llama-2, the ASR of all methods is 0 (Table 27); on Mistral, DPP reduces ASR from 0.41 to 0.01, on par with Goal Prioritization.
- More attacks (Appendices N, R): against llm-simple-adaptive-attacks and I-FSJ, DPP's average ASR on Llama-2 is 0.000 (Table 24); against five attacks including GPTFuzz, PAP, and DRL, DPP averages 0.062 (Table 35).
Other Ablations and Analyses
- Objective function (Table 6): removing the defense term (α=0) gives GCG ASR 0.16 and Win-Rate 72.85%; removing the helpfulness term (β=0) gives GCG ASR 0.03 and Win-Rate dropping to 65.34%.
- Patch position (Table 7): the suffix gives an average Win-Rate of 76.09 and GCG adaptive ASR 0.15; the prefix gives 73.05 and 0.57.
- Solver (Table 11): replacing with RLPrompt gives GCG ASR 0.15 and Win-Rate 47.89, weaker than HGA's 0.04 and 82.98.
- Synonym replacement (Table 12): when disabled, average ASR 0.15 and Win-Rate 79.95; the full algorithm gives 0.07 and 82.98.
- Readability (Table 34): GPT-4 perplexity is 56.57 for DPP, lower than RPO's 8780.94 and Self-Reminder's 298.39, and higher than System Prompt's 25.65 and Goal Prioritization's 40.65.
What are some points that can be further explored?
The authors list three limitations: training cost, dependence on GPT-4, and local removability; experiments cover only 7B–13B open-source models.
Limitations and Future Directions Noted by the Authors
- Computational efficiency and scalability (Limitation section): HGA training is computationally intensive, and scaling to larger datasets or more models may be limited by compute, so training efficiency may need further optimization.
- Training cost (Limitation section): training requires an LLM to rewrite prototypes, currently GPT-4, which is costly to access; the authors plan to replace GPT-4 with an open-source LLM in the future to reduce cost.
- Easily removed (Limitation section): DPP is intended for deployment by model service providers; when users run open-source models locally, they can easily remove the DPP or system prompt, in which case the model cannot be protected.
- Utility loss (Conclusion section): the authors plan in the future to reduce utility loss on less-aligned models while maintaining defense capability.
- Baseline shortcomings (Limitation section): Self-Reminder trains poorly on Llama-2-7B-Chat; RPO has a long training time; Goal Prioritization has a long context and slower inference.
Scope of Experimental Coverage
- The models tested are Llama-2-7B-Chat, Mistral-7B-Instruct-v0.2, Vicuna-13B-v1.5, and Llama-3-8B-Instruct, 4 in total (Starling-7B is additionally used only for the RPO check), all open-source models at the 7B–13B level.
- Attacks cover GCG, AutoDAN, PAIR, TAP, ICA, Base64, Catastrophic, and the llm-simple-adaptive-attacks, I-FSJ, GPTFuzz, PAP, DRL, and Ignorance adaptive attacks in the appendix.
- Each attack uses 100 queries (50 for PAIR/TAP); utility evaluation uses 100 Alpaca items and AlpacaEval Win-Rate (reference model Davinci003).
- ASR determination is mainly keyword matching, with the Llama-Guard judge for review; the paper does not report consistency values between the two determinations.
- The paper does not report the number of experimental repetitions, confidence intervals, or statistical significance tests; the rewriting model used for training is GPT-4, and the training cost is about 75 USD (Appendix C).
Summarize the main content of the paper
DPP uses HGA to search for a human-readable defense suffix, achieving the lowest average ASR among the evaluated defenses on the two 7B models at a relatively low utility cost.
A jailbreak defense paper: addressing the problem that aligned LLMs remain vulnerable to jailbreaks while existing prompt defenses struggle to balance safety and utility, it proposes Defensive Prompt Patch (DPP).
- Method: targeting the weighted sum of refusal log-likelihood and helpfulness log-likelihood (weights α, β), uses a Hierarchical Genetic Algorithm to perform sentence-level synonym substitution and paragraph-level crossover and mutation (GPT-4 rewriting) to search for a human-readable suffix prompt; it is appended to each user query at inference time without retraining the model.
- Llama-2-7B-Chat main results: under six non-adaptive attacks, average ASR 3.80% and Win-Rate 82.98%, both the best among the defenses tested (Table 2); under adaptive attacks, average ASR 13.0%, lower than Self-Reminder's 17.7% and RPO's 45.7% (Table 3).
- Mistral-7B-Instruct-v0.2: non-adaptive average ASR 2.0% (Table 4), adaptive 46.9% (Table 5), both the lowest; but Win-Rate 75.06%, a higher utility cost than Self-Reminder and System Prompt.
- Generalization: on unseen AdvBench queries, Llama-2 average ASR 7.5%; on Llama-3-8B-Instruct, average adaptive ASR 4.5%; against llm-simple-adaptive-attacks and I-FSJ, average ASR 0.0%.
- Ablation: suffix outperforms prefix (GCG adaptive ASR 0.15 vs. 0.57); HGA outperforms RLPrompt; removing either the defense or helpfulness worsens ASR or Win-Rate.
- Authors' conclusion: DPP outperforms existing prompt defenses on the safety-utility balance, and its human readability facilitates auditing and manual improvement; limitations are high training computational cost, dependence on GPT-4 (about 75 USD), and removability by users in local deployment.
Related papers
Show 3 more papers
- Safety Alignment Should Be Made More Than Just a Few Tokens DeepAnalysis / interpretabilityICLR 2025