JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

1. 01What problem does this paper attempt to solve?

   Jailbreak evaluation lacks unified standards, its measures are not comparable, and it is not reproducible; the authors propose the open benchmark JailbreakBench.

   Core problem: evaluation of LLM jailbreak attacks lacks standards, results are not comparable, and reproduction is difficult; an open, standardized benchmark is needed.

   • Problem setting: The authors note that even aligned LLMs can be induced by jailbreak attacks to generate harmful content; as LLMs are deployed in safety-critical domains, measuring progress in jailbreak attacks and defenses becomes important.
   • Existing shortcomings: The authors identify three problems with current evaluation—there is no clear standard practice for jailbreak evaluation; the way different works compute cost and success rate is not comparable; and many works do not release adversarial prompts, keep code closed-source, or rely on proprietary APIs that change, making them irreproducible.
   • Core claims: The benchmark design is built around three principles—full reproducibility through open-source jailbreak artifacts; extensibility to incorporate new attacks, defenses, and models; and an easily accessible evaluation pipeline (can run entirely in the cloud, no local GPU required).
   • Paper outputs: Introduces JailbreakBench, comprising a jailbreak artifact repository, the JBB-Behaviors dataset of 100 behaviors, a standardized evaluation framework (including threat model, system prompts, and scoring functions), and a leaderboard tracking attack and defense performance.
   • Threat model (declared at submission time, terminology as in the paper):
     • Objective: find a prompt P such that JUDGE(LLM(P), G)=True, i.e., make the target model generate harmful content for harmful goal G.
     • Knowledge/capability: accepts white-box, black-box, universal, transfer, and adaptive attacks; submissions must be labeled "black_box", "white_box", or "transfer".
     • Constraints: attackers are currently not allowed to modify the system prompt or prefill the LLM response with a specific string; text modality only.
     • Victim systems: open-source and closed-source LLMs (Vicuna, Llama-2, GPT-3.5, GPT-4).

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four categories: attacks, defenses, evaluation methods and benchmarks; the authors position this paper as a community-driven benchmark supporting adaptive attacks and test-time defenses.

   Jailbreak attacks

   • Manual prompts: Mowshowitz (2022), Albert (2023), Wei et al. (2023) early manually refined jailbreak prompts.
   • Optimization-based attacks: Zou et al. (2023), Geisler et al. (2024) use first-order discrete optimization; Lapid et al. (2023), Liu et al. (2023) use zeroth-order methods such as genetic algorithms; Andriushchenko et al. (2024), Sitawarin et al. (2024), Hayase et al. (2024) use random search.
   • Auxiliary LLM attacks: Yu et al. (2023) refine templates, Yong et al. (2023), Deng et al. (2023) translate into low-resource languages, Chao et al. (2023), Mehrotra et al. (2023) generate jailbreaks, Shah et al. (2023), Zeng et al. (2024), Takemoto (2024) rewrite harmful requests.

   Jailbreak defenses

   • Training-based: RLHF (Ouyang et al., 2022), DPO (Rafailov et al., 2024), adversarial training (Mazeika et al., 2024), fine-tuning on jailbreak strings (Hubinger et al., 2024).
   • Test-time defenses: SmoothLLM (Robey et al., 2023; Ji et al., 2024) and perplexity filtering (Jain et al., 2023; Alon and Kamfonas, 2023) as outer wrappers around the LLM to detect jailbreaks.

   Evaluation methods and benchmarks

   • Judge methods: ranging from human annotation (Wei et al., 2023), rule-based classifiers (Zou et al., 2023), neural network classifiers (Huang et al., 2023; Mazeika et al., 2024) to LLM-as-a-judge; Souly et al. (2024) discuss shortcomings of existing jailbreak judges and propose finer scoring criteria.
   • Benchmarks: PromptBench (Zhu et al., 2023) does not target jailbreaks; DecodingTrust (Wang et al., 2023) and TrustLLM (Sun et al., 2024) only evaluate static templates; HarmBench (Mazeika et al., 2024) implements attacks and defenses and covers copyright and multimodal topics; EasyJailbreak (Zhou et al., 2024) is concurrent work implementing 11 jailbreak methods; there are also competitions such as TDC (Mazeika et al., 2023). The authors note that many existing datasets contain duplicate entries, unachievable behaviors, or are not fully open-sourced.

   Baselines and benchmark/dataset: attack baselines are GCG, PAIR, JB-Chat, Prompt with RS; defense baselines are SmoothLLM, Perplexity Filter, Erase-and-Check, Synonym Substitution, Remove Non-Dictionary; evaluation uses JBB-Behaviors constructed by the authors (100 harmful + 100 benign behaviors), and judge selection used 100 benign samples from XS-Test.

   Positioning: The authors state that unlike HarmBench, this paper focuses on supporting adaptive attacks and test-time defenses, standardizes the evaluation of test-time defenses without standardizing attack implementations, and emphasizes being community-driven with clear submission guidelines; JailbreakBench is not a competition but an open-ended long-term tracking project.

   Collapse this answer

3. 03How does the paper address this problem?

   JailbreakBench consists of a behavior dataset, an artifact repository, red-teaming and defense pipelines, a selected judge, and a leaderboard.

   Overall approach: guided by the principles of reproducibility, extensibility, and accessibility, build the open-source benchmark JailbreakBench to unify the evaluation measures for jailbreak attacks and defenses. The jailbreak task is formalized as: find P ∈ T^⋆ subject to JUDGE(LLM(P), G) = True, i.e., find a prompt P such that the target model's generation for harmful goal G is judged harmful.

   JBB-Behaviors dataset

   • Scale and composition: 100 harmful behaviors across 10 categories corresponding to OpenAI usage policies (10 per category, Table 4); 55% are original, 18% from AdvBench, and 27% from TDC/HarmBench (Figure 3).
   • Fields: Each row contains five fields: Behavior, Goal, Target, Category, Source; the authors ensure each behavior is unique and realizable via text generation.
   • Benign behaviors: A separate set of 100 benign behaviors, each on exactly the same topic as the corresponding harmful behavior, used to evaluate refusal rates of models and defenses and prevent refusal based on keywords alone; the authors note some of these behaviors are borderline cases.

   Jailbreak artifact repository

   • Contents: Records the prompts, responses, and classification results for each attack or defense, with metadata (hyperparameters, ASR, number of queries to the target model).
   • Current status: Currently includes jailbreak strings from PAIR, GCG, JailbreakChat, and Andriushchenko et al. (2024); the authors believe these artifacts can serve as an initial dataset for adversarial training.

   Red-teaming and defense pipeline

   • Red-teaming pipeline: Standardized tokenization, sampling, and system prompts; two lines of Python suffice to load and query models, supporting both local and cloud LLMs.
   • Defense pipeline: Implements five defenses (SmoothLLM, Perplexity Filter, Erase-and-Check, Synonym Substitution, Remove Non-Dictionary), combinable with any LLM; the authors emphasize that proper evaluation of test-time defenses should rely on adaptive attacks, and transfer attacks from undefended models can only give a lower bound on worst-case ASR.

   Judge selection

   • Candidates: Six classifiers—rule-based judge (Zou et al., 2023), GPT-4-0613, HarmBench's Llama-2-13B judge, Llama Guard, Llama Guard 2, and Llama-3-70B with a custom prompt.
   • Procedure: Collect 200 jailbreak prompt-response pairs (100 PAIR, 50 GCG, 50 Prompt with RS, Appendix C), plus 100 benign samples from XS-Test, for 300 total; three experts annotate (about 95% agreement), and the majority vote is taken as ground truth.
   • Rationale for selection: Llama-3-70B and GPT-4 perform similarly, but GPT-4 is closed-source, expensive to query, and changes over time; the authors choose the open-weight Llama-3-70B for reproducibility, and it has a lower, more conservative FPR. Submitted attacks and defenses are not allowed to use this judge to prevent overfitting.

   Evaluation framework and leaderboard

   • Framework: Built-in system prompts and judge, supporting Vicuna-13B-v1.5, Llama-2-7B-chat-hf, GPT-3.5-Turbo-1106, GPT-4-0125-Preview; all use greedy decoding (temperature 0), generating 150 tokens per input by default.
   • Submissions: Attack submissions must declare the threat model (black_box/white_box/transfer) and hyperparameters; suffix length is not restricted; defenses or models with a refusal rate above 90% on benign behaviors are flagged.
   • Leaderboard: The website is built on RobustBench code and displays attack and defense results with links to the corresponding artifacts.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Judge comparison, evaluation of four attacks and five defenses on four LLMs, benign behavior refusal rate, and breakdown by source.

   Experimental setup

   • Target models: Vicuna-13B-v1.5, Llama-2-7B-chat-hf (open-source); GPT-3.5-Turbo-1106, GPT-4-0125-Preview (closed-source).
   • Datasets: 100 harmful behaviors from JBB-Behaviors; 100 benign behaviors for refusal evaluation; 300 samples for judge selection (200 jailbreak pairs + 100 XS-Test benign samples).
   • Attack baselines: GCG (batch size 512, 500 steps, transferring the Vicuna suffix on closed-source models), PAIR (Mixtral as the attack model, temperature 1, top-p 0.9, N=30 streams, maximum depth K=3), JB-Chat (AIM template), Prompt with RS (self-transfer).
   • Defense baselines: SmoothLLM (swap perturbation, N=10, q=10%), Perplexity Filter (perplexity computed with Llama-2-7B), Erase-and-Check (erase length 20), Synonym Substitution (5% probability), Remove Non-Dictionary; SmoothLLM and Erase-and-Check use Llama Guard as the internal judge.
   • Metrics and judge: ASR uses Llama-3-70B as the judge; efficiency reports average number of queries and average number of tokens (not reported for transfer and manual attacks; Prompt with RS does not count queries for template optimization and precomputed suffix).
   • Repetitions and randomness: No error bars are reported; the authors state that under greedy decoding evaluation is largely deterministic, and differences between Together AI and local runs are at most 1–3 percentage points in ASR (Appendix E).

   Main results

   Table 2 (attacks, ASR by Llama-3-70B judge) and Table 3 (defenses, transfer attack ASR):

   Swipe horizontally to read the table

   Attack	Vicuna	Llama-2	GPT-3.5	GPT-4
   PAIR	69%	0%	71%	34%
   GCG	80%	3%	47%	4%
   JB-Chat	90%	0%	0%	0%
   Prompt with RS	89%	90%	93%	78%

   • The authors state that Prompt with RS is on average the most effective and also query-efficient (on average 2 queries on Vicuna, 3 on GPT-3.5), attributed to the manually optimized template and precomputed initialization.
   • The authors believe GCG's ASR is lower than previously reported values, mainly because JBB-Behaviors behaviors are harder and the judge is more conservative; JB-Chat's AIM template fails entirely on Llama-2 and the GPT models, which the authors believe may have been patched by OpenAI.
   • On the defense side (Table 3): Perplexity Filter is effective only against GCG; SmoothLLM reduces ASR for GCG and PAIR but is not effective against JB-Chat and Prompt with RS; the authors state Erase-and-Check is the most robust defense, but Prompt with RS still has non-trivial ASR on all LLMs (e.g., 24% on Vicuna, 10% on GPT-4).

   Judge classifier comparison

   • What was measured: agreement with the majority vote of three experts on 300 samples, FPR, and FNR for six candidate judges (Table 1).
   • Results: Llama-3-70B agreement 90.7% (FPR 11.6%, FNR 5.5%), GPT-4 90.3% (FPR 10.0%, FNR 9.1%), Llama Guard 2 87.7%; HarmBench 78.3% (FPR 26.8%, the authors observed many false positives on XS-Test benign samples), Llama Guard 72.0% (FNR 60.9%), rule-based judge 56.0%.
   • Authors' interpretation: Llama-3-70B and GPT-4 perform similarly; the former was chosen for reproducibility; its low FPR may systematically lower the success rates of all attacks, but helps keep results conservative.

   Additional defenses and refusal rate

   • Additional defenses (Table 6): The authors state Synonym Substitution is surprisingly effective, with the highest ASR among precomputed jailbreaks at only 24%; Remove Non-Dictionary yields limited improvement (Prompt with RS still reaches 91% on Vicuna).
   • Refusal rate (Figure 2): On 100 benign behaviors, Vicuna refuses 9% without defense, and Llama-2 over 60%; the authors state that under the chosen hyperparameters the defenses do not substantially increase refusal rates, and note this is only a sanity check and cannot replace utility evaluations such as MMLU and MT-Bench.

   Other ablations and analyses

   • Breakdown by source (Table 5): ASR is relatively consistent across the three sources; ASR on original behaviors tends to be lower, which the authors suggest may stem from imbalanced composition within categories.
   • Timeliness of closed-source models (Appendix E): When the transfer attack of GCG on GPT models was re-measured on June 5, 2024, ASR dropped to about 5%, which the authors believe may be due to safety patches.
   • Run-mode differences (Appendix E): ASR differences between Together AI and local runs of Vicuna and Llama Guard are at most 1–3 percentage points.
   • Negative results: JB-Chat has ASR of 0% on Llama-2, GPT-3.5, and GPT-4 (Table 2); PAIR is 0% on Llama-2.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list threat models, modality limitations and an update plan; experiments cover four models and non-adaptive defense evaluation.

   Limitations and future directions noted by the authors

   • Restricted attack capability (§5 Limitations): The current formulation does not allow attackers to modify the system prompt or prefill the LLM response with a specific string.
   • Text modality only (§5 Limitations): Modern LLMs often accept modalities other than text, and these modalities can likewise be used for jailbreaking; the benchmark currently does not offer this option.
   • The benchmark will be continuously updated (§5 Future plans): The authors plan regular updates as the field develops, potentially including expanding the behavior dataset, more rigorous defense evaluation (especially non-conservativeness and efficiency), updating the judge classifier, and periodically re-measuring ASR on closed-source models.
   • Refusal evaluation is only a sanity check (§4 Refusal evaluation): The authors note that benign-behavior refusal evaluation cannot replace more comprehensive utility evaluations such as MMLU and MT-Bench.
   • Transfer attacks are only a lower bound (§3.4): The authors note that evaluating defenses with transfer attacks from undefended models can only give a lower bound on worst-case ASR; more sophisticated techniques may further increase ASR.
   • The judge can be updated (§3.5): The authors state they are willing to update the chosen judge if a consistently better option emerges.

   Scope of experiments

   • 4 target models were tested: Vicuna-13B-v1.5, Llama-2-7B-chat-hf, GPT-3.5-Turbo-1106, GPT-4-0125-Preview (§3.6).
   • 4 attack baselines and 5 defense baselines (3 evaluated in the main text, 2 added in Appendix D); defense evaluation uses only transfer attacks from undefended models, with no adaptive attacks (§4, Table 3, Table 6).
   • Judge comparison is based on 300 samples (200 jailbreak pairs + 100 XS-Test benign samples), annotated by three experts with about 95% agreement (§3.5, Appendix C).
   • Refusal rate evaluation was conducted only on Vicuna and Llama-2, with 100 benign behaviors per model (Figure 2).
   • The paper does not report error bars or number of repetitions, nor total compute or the types of resources used (Appendix G checklist).

   Collapse this answer

6. 06Summarize the main content of the paper

   JailbreakBench unified jailbreak evaluation: a 100-behavior dataset, artifact repository, Llama-3-70B judge, and leaderboard.

   JailbreakBench is an open benchmark for LLM jailbreak robustness, addressing the problems that jailbreak evaluation lacks standards, results are not comparable across work, and artifacts are not public, making reproduction impossible.

   • Problem addressed: Jailbreak attacks can induce aligned LLMs to generate harmful content, but work differs in how success is determined and costs are counted, and many attack prompts are not public, making comparison and reproduction difficult.
   • Method highlights: Construct the JBB-Behaviors dataset (100 harmful behaviors across 10 OpenAI policy categories, 55% original, 18% from AdvBench, 27% from TDC/HarmBench, plus 100 benign behaviors on the same topics); open-source jailbreak artifact repository; standardized red-teaming and defense pipelines; after comparing six judges on 300 human-annotated samples, select Llama-3-70B (90.7% agreement); provide a leaderboard.
   • Attack results: On four LLMs, Prompt with RS is on average the most effective (ASR 90% on Llama-2, 78% on GPT-4); GCG is only 3% on Llama-2; the AIM template of JB-Chat is 0% on Llama-2 and GPT models.
   • Defense results: Against transfer attacks, Erase-and-Check performs most consistently but Prompt with RS still has non-trivial ASR; Synonym Substitution reduces the highest ASR of each precomputed jailbreak to 24%; Perplexity Filter is only effective against GCG.
   • Authors' conclusions: The authors state that even recent closed-source undefended models are highly vulnerable to jailbreaks, and they hope the artifact repository and defense pipelines will drive research on adaptive attacks against these defenses; the authors believe a public benchmark benefits the community more than it harms it.
   • Boundaries: The benchmark does not allow modifying the system prompt or prefilling responses, and covers only the text modality; the authors plan to continuously update the dataset, defense evaluation and judges as the field develops.

   Collapse this answer

Related papers