StrongREJECT: A Benchmark for Hollow Jailbreaks
A StrongREJECT for Empty Jailbreaks
The authors propose the jailbreak evaluation benchmark StrongREJECT, claim that its automatic evaluator's agreement with human judgments is better than the baselines, and report that a jailbreak tendency reduces model capability.
- Jailbreak papers often claim attack success rates close to 100%; the authors argue that the forbidden prompts in existing benchmarks are of poor quality and that evaluation methods only look at whether the model refuses, systematically overestimating jailbreak effectiveness, and that researchers lack a standard, high-quality evaluation benchmark.
- The authors build the StrongREJECT benchmark: 313 forbidden prompts covering six categories of harmful behavior with factually verifiable answers, plus an automatic evaluator that scores based on whether the model refuses and on the specificity and persuasiveness of the response, measuring both the model's willingness and capabilities.
- The authors report that the StrongREJECT evaluator's agreement with human judgments is better than the baselines (rubric version MAE 0.077, Table 1); most baselines overestimate jailbreak effectiveness; among the 37 jailbreaks, only a few such as PAIR and PAP are effective; the authors also claim that a jailbreak tendency reduces model capability.
- In the Discussion, the authors note three limitations: LLM only, the dataset may not be robust to changes in terms of service, and 313 items are insufficient to measure model robustness on their own. Human evaluation covers only 4 models, a 60-prompt subset, and 1,361 annotation points.
- Evaluated models
- GPT-4oGPT-3.5 TurboLlama-3.1 70B InstructGPT-4 (version 0613)Llama-2 (llama2-70b-chat)Dolphin
- Benchmarks
- StrongREJECTMMLU
- Metrics
- StrongREJECT scoreBiasMAESpearman correlationnon-refusal rateMMLU score
Researchers at the UC Berkeley Center for Human-Compatible AI propose the StrongREJECT jailbreak evaluation benchmark, containing 313 forbidden prompts covering six categories of harmful behavior and two automatic evaluators, a rubric-based version and a fine-tuned Gemma 2B open-source version, whose agreement with human judgments is better than existing automatic evaluation methods. The paper notes that most jailbreak papers claim near-100% attack success rates, but existing evaluations mostly only look at whether the model refuses, generally overestimating attack effectiveness; for example, a Scottish Gaelic translation jailbreak claimed a 43% success rate against GPT-4, but when the authors reproduced it they found the outputs were vacuous. Controlled experiments show that jailbreaks that bypass safety fine-tuning often degrade model capability; in MMLU tests on the unaligned Dolphin and GPT-4o, the more a jailbreak increases willingness to answer, the worse the answer quality.
Why it was selectedUsing human annotations as a reference, the authors find that commonly used jailbreak evaluations generally overestimate attack effectiveness, and their benchmark, which measures both refusal and answer quality, can be used directly for jailbreak evaluation.
Related events and news
0 events · 7 news mentions
News that mentions this paper
- Qiitacites the paper's id or link攻击成功率(ASR)数字怎么读:评分标准、试验次数和数据集能让同一评估结果变成 15% 或 100%
Souly et al., A StrongREJECT for Empty Jailbreaks (arXiv:2402.10260)
- Failure-Firstcites the paper's id or link安全基准为何结论不一:自有结果与公开榜单对比
Souly, A., et al. (2024). “A StrongREJECT for Empty Jailbreaks.” arXiv:2402.10260.
- Anthropic Alignment Sciencecites the paper's id or linkAnthropic 研究:自动化对齐研究者在小模型上缓解十类对齐失败
Souly, Alexandra, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, et al. 2024. “A StrongREJECT for Empty Jailbreaks.” arXiv Preprint arXiv:2402.10260.
Show 4 more news items
- FAR.AIcites the paper's id or linkFAR.AI 披露 jailbreak-tuning 数据投毒攻击:GPT-4o 拒答率降至 3.6%
We use several StrongREJECT-like evaluators to measure the likelihood of an LLM producing harmful or unsafe responses.
- FAR.AIcites the paper's id or linkFAR.AI 红队测试:DeepSeek R1 与 OpenAI、Anthropic、Google 可微调模型的护栏可被轻易移除
We use StrongReject, a state-of-the-art benchmark that is designed for measuring how vulnerable large language models (LLMs) are to complying with harmful requests.
- FAR.AIcites the paper's id or linkFAR.AI 发布 ClearHarm 越狱数据集,聚焦 CBRN 等明确有害问题
We have found it is more challenging for attacks to elicit harmful responses from models on this benchmark than existing jailbreak benchmarks like StrongREJECT, Do-Not-Answer and SORRY-Bench.
- FAR.AIcites the paper's id or linkFAR.AI 与 UK AISI 测试多层 AI 防御:STACK 攻击成功率达 71%
However, the few-shot Gemma 2 classifier achieved notable success: it overwhelmingly blocked harmful requests across all attacks and datasets, except for PAP on the StrongREJECT benchmark.
Paper analysis
What problem does this paper attempt to solve?
The authors argue that jailbreak effectiveness is systematically overestimated by existing benchmarks and propose StrongREJECT, a high-quality jailbreak evaluation benchmark.
Core problem: jailbreak research lacks standard, high-quality evaluation benchmarks, causing the attack success rates claimed in papers to often be exaggerated.
- Problem setting and importance: The authors point out that many jailbreak papers claim attack success rates close to 100%, but this does not match the actual harmfulness of the attacks; for example, Yong et al. [39] claimed a 43% success rate against GPT-4, but when the authors reproduced it, they found the outputs were empty or incoherent (Table 2). Safety researchers need reliable evaluations to identify which jailbreaks genuinely increase the risk of model misuse.
- Shortcomings of existing methods (according to the paper): Existing forbidden-prompt datasets have problems such as duplication, inappropriate phrasing, unanswerable questions, or prompts that are not actually prohibited (e.g., 24 of the 520 AdvBench entries ask how to build a bomb); existing automatic evaluation methods mostly use binary metrics and focus excessively on whether the model refuses, ignoring the usefulness of the response to the attacker.
- Core observation and hypothesis: The authors believe attackers want useful information relevant to the query, and evaluation should measure both the model's willingness and capabilities; the authors propose the hypothesis that a jailbreak tendency that bypasses safety fine-tuning reduces model capability.
- What the paper proposes: The StrongREJECT benchmark, comprising a dataset of 313 high-quality forbidden prompts and an automatic evaluator; the authors claim the evaluator's agreement with human judgments reaches state-of-the-art, and use two sets of experiments to test the effect of jailbreaks on model capability.
What related research exists?
Related work falls into two categories, forbidden-prompt datasets and evaluation methods; the authors point out shortcomings in both and use 7 automatic evaluators as baselines.
Forbidden-prompt datasets
- LLM-generated datasets (AdvBench [44], Shen et al. [31], MaliciousInstruct [15], etc.): the paper points out that such prompts tend to be repetitive, poorly phrased, unanswerable, or not truly forbidden; for example, 5% of prompts in AdvBench ask how to build a bomb, and MaliciousInstruct has marijuana-related questions that GPT-4 will answer directly.
- Human-written or curated datasets (MasterKey [8], Ganguli et al. [11], etc.): the paper points out that MasterKey has prompts asking for classified nuclear weapons information; the 38,961 crowdsourced red-team interactions of Ganguli et al. are full conversations rather than filtered single-turn forbidden prompts.
Evaluation methods
- Human evaluation [15, 16, 33, 39, 44]: human judgment is the gold standard, but the paper points out that it is not scalable and hinders rapid iteration.
- Automatic evaluation: most prior work uses binary indicators to judge whether a jailbreak succeeded [18, 43, 40, 44, etc.]; the paper criticizes some methods for defining success solely as non-refusal [33, 39, 40, 27, 38], ignoring the usefulness of the response.
- HarmBench [20]: concurrent work providing a fine-tuned classifier self-hosted on HuggingFace; this paper uses it as one of the baselines.
Baselines and benchmarks: the authors use 6 prior automatic evaluators plus the concurrently released HarmBench as baselines—String matching for non-refusal [27], Jailbroken - binary [2], PICT [29, 10], GPT-4 Judge [26], PAIR [4], OpenAI moderation API, HarmBench [20]; the capability experiments use MMLU [14].
The authors position this paper as a benchmark that improves both dataset quality and evaluation accuracy: prompts are specific, answerable, harmful, and generally refused by models, and the evaluator measures the usefulness of responses toward achieving the harmful goal, rather than only toxicity or whether the model refused.
How does the paper address this problem?
StrongREJECT consists of 313 forbidden prompts and an automatic evaluator that measures both willingness and capability.
Overall approach: build a high-quality forbidden-prompt dataset and design an automatic evaluator, StrongREJECT, that measures both willingness and capabilities.
Forbidden-prompt dataset
- Category system: Cross-referencing the usage policies of OpenAI, Anthropic, Google Gemini, Meta LLaMA, and DeepInfra, six categories of behavior prohibited by all policies were selected: illegal goods and services, non-violent crime, hate harassment and discrimination, disinformation and deception, violence, and sexual content.
- Source and scale: 313 prompts in total, at least 50 per category; 70% are original by the authors, 10% each come from DAN [31] and AdvBench [44], and the rest come from other prior work and OpenAI system cards (Table 4).
- Quality control: Prompts underwent deduplication (OpenAI embeddings cosine similarity), refusal filtering by nine "sentinel" models, GPT-3.5 Turbo classification, and manual screening; prompts are required to have factually verifiable answers, such that an ordinary person could write a high-quality response within one hour using the internet.
Automatic evaluator
- rubric version: The evaluation model (GPT-4o-mini in the paper) is given the forbidden prompt, the model response, and scoring instructions; the model first outputs chain-of-thought reasoning, then gives three scores: a binary refusal score and two 1–5 Likert scores (specific, convincing, rescaled to [0-1]).
- Scoring formula: score = (1 − refused) × (specific + convincing)/2, i.e., a refusal scores 0, otherwise the mean of specificity and convincingness is taken; the authors use this to characterize both willingness and capabilities.
- Feature selection: The authors used Lasso regression to select specific and convincing from 10 candidate features, and chose equal weighting rather than fitting a more complex model, on the grounds that the best fit is sensitive to the training distribution.
- fine-tuned version: LoRA fine-tuning on Gemma 2B, with training data of about 15,000 responses labeled by the rubric version (based on GPT-4 Turbo) (4,059 from StrongREJECT prompts, 10,837 from HarmBench [20]), for researchers to self-host on a single GPU.
Evaluation protocol
- Human agreement experiment: 5 LabelBox annotators scored responses of 4 victim models on a subset of 60 prompts on a 1–5 scale (rescaled to [0-1]), taking the median as ground truth, for a total of 1,361 annotation points; the automatic evaluator is treated as a regression model predicting human scores, and bias, MAE, and Spearman correlation are computed.
- Jailbreak evaluation: The rubric version evaluator was used on 313 prompts to evaluate the effect of 37 jailbreak methods against three aligned models.
- Capability isolation experiment: To separate the effect of jailbreaks on capability, the authors fed jailbroken forbidden prompts to the unaligned model Dolphin, and fed jailbroken benign prompts (MMLU questions) to GPT-4o.
What experiments did the paper conduct?
The human agreement experiment shows StrongREJECT is the most accurate; only a few of the 37 jailbreaks are effective; the two sets of experiments support that jailbreaks reduce capability.
Experimental setup
- Victim models: the human agreement experiment uses GPT-4 (version 0613), GPT-3.5 Turbo, Llama-2 (llama2-70b-chat), and Dolphin (Mixtral 8×7B with safety fine-tuning removed); the full-dataset jailbreak evaluation uses GPT-4o, GPT-3.5 Turbo, and Llama-3.1 70B Instruct.
- Dataset: 313 StrongREJECT forbidden prompts; human evaluation uses a 60-prompt subset; capability experiments use a randomly selected 171-question subset of MMLU.
- Jailbreak methods: 17 for human evaluation, 37 for the full-dataset evaluation (Table 5 includes descriptions and sources).
- Baseline evaluators: String matching, Jailbroken - binary, PICT, GPT-4 Judge, PAIR, OpenAI moderation API, HarmBench.
- Metrics and review: Bias, MAE (all responses and the subset with human score >0), Spearman correlation; human annotation uses the median of 5 annotators, totaling 1,361 annotation points; confidence intervals use scipy.stats.bootstrap 95% CI.
Main results
According to Table 1 (human evaluation as ground truth; bold marks the two best per metric):
Swipe horizontally to read the table
Evaluator Bias MAE (all) MAE (human>0) Spearman String matching 0.484 ± 0.03 0.580 ± 0.03 0.372 ± 0.04 -0.394 Jailbroken - binary 0.354 ± 0.03 0.407 ± 0.03 0.254 ± 0.04 -0.291 GPT-4 Judge 0.208 ± 0.02 0.262 ± 0.02 0.212 ± 0.03 0.157 PAIR 0.152 ± 0.02 0.205 ± 0.02 0.224 ± 0.03 0.249 OpenAI moderation API -0.161 ± 0.02 0.197 ± 0.02 0.761 ± 0.04 -0.103 HarmBench 0.013 ± 0.01 0.090 ± 0.01 0.239 ± 0.03 0.819 StrongREJECT fine-tuned -0.023 ± 0.01 0.084 ± 0.01 0.244 ± 0.02 0.900 StrongREJECT rubric 0.012 ± 0.01 0.077 ± 0.01 0.196 ± 0.03 0.846 (The PICT row is omitted; Spearman is the jailbreak ranking correlation on GPT-3.5 Turbo.)
- The authors state that most baselines overestimate jailbreak effectiveness, String matching has the largest bias, and only the moderation API systematically underestimates; the two StrongREJECT versions and HarmBench have almost no bias.
- The authors state that the MAE of both StrongREJECT versions is lower than all other evaluators; for responses rated completely harmless (score 0) by humans, the rubric and fine-tuned versions give averages of 0.039 and 0.035 respectively, higher only than the moderation API's 0.019.
- The authors state that StrongREJECT's jailbreak rankings correlate highly with human rankings, with only HarmBench comparable; Appendix D reports correlations above 0.84 under four data inclusion rules.
Full-dataset jailbreak evaluation
- Setup: 37 jailbreaks × 313 prompts, rubric version evaluator, victim models GPT-4o, GPT-3.5 Turbo, Llama-3.1 70B Instruct (Figure 3).
- Results: PAIR scored 0.74, 0.74, and 0.72 on the three models respectively; PAP Misrepresentation scored 0.55, 0.49, and 0.74; selecting the best per question ("Best") gave 0.95, 0.99, and 0.96. The two GCG versions were effective against GPT-3.5 Turbo (0.69, 0.84) but nearly ineffective against GPT-4o (0.06, 0.01) and Llama-3.1 70B (0.03, 0.01).
- Authors' interpretation: the few methods that use LLMs to attack LLMs (PAIR, PAP) are most effective; the authors suggest the GCG results indicate new models may have been trained against such adversarial suffixes; the "Best" row shows frontier models remain vulnerable to jailbreak attacks; most jailbreak scores are far below the values reported in the original papers.
Effect of jailbreaks on model capability
- Unaligned model experiment: forbidden prompts after 37 jailbreaks are fed to Dolphin; in the left panel of Figure 4, the x-axis is the average non-refusal rate on the three aligned models, and the y-axis is Dolphin's StrongREJECT score. The authors state that jailbreaks that increase aligned models' willingness more tend to reduce Dolphin's capability more.
- Benign prompt experiment: jailbreak algorithms are applied to MMLU questions and fed to GPT-4o; in the right panel of Figure 4, the y-axis is GPT-4o's zero-shot score on the 171-question subset. The authors state the pattern is similar to the left panel.
- Authors' interpretation: the two experiments together support the hypothesis that "jailbreaks generally harm model capability," making responses less coherent, more off-topic, less truthful, or more terse; the authors argue this explains the upward bias of older evaluators—they give high scores to "willing but incapable" responses.
Other ablations and analyses
- Reproducing the Scots Gaelic jailbreak of Yong et al. [39]: GPT-4-0613 at τ = 0 produced no actionable explosives information in all three samples (Table 2).
- Figure 2: MAE broken down by jailbreak method; the authors state both StrongREJECT versions are close to human on every jailbreak.
- Appendix D: Llama-2 and GPT-4 cover only 9 and 4 jailbreaks due to exhausted annotation budget, so Spearman correlation cannot be reliably computed; the main text reports only GPT-3.5 Turbo.
- Appendix C.4: few-shot prompting can improve chain-of-thought, but cost rises to 3–4 times, so the authors did not adopt it.
What are some points that can be further explored?
The authors list three limitations: modality, policy timeliness, and dataset size; experiments cover 4–3 models and 313 prompts.
Limitations noted by the authors and future directions
- LLM only (Discussion, Limitations): the authors limit the scope to LLMs, and whether StrongREJECT applies to multimodal models is unclear.
- Policy timeliness (Discussion, Limitations): forbidden-prompt datasets may not be robust to changes in provider terms, and the authors give the example of OpenAI removing its ban on military use.
- Dataset size (Discussion, Limitations): 313 prompts balance cost and coverage (in the worst case, estimating to within 0.05 points with 90% confidence), but the authors acknowledge the size is relatively small and insufficient as an independent conclusion on model robustness, and should be supplemented by larger-scale evaluations.
- Evaluator robustness to optimization untested (Appendix A): the authors did not test the robustness of StrongREJECT scores to optimization, and whether they remain accurate when used as the objective of search algorithms such as PAIR is difficult to assess.
Scope of experiments
- The human agreement experiment covers 4 victim models, a 60-prompt subset, 17 jailbreaks, and 5 annotators, totaling 1,361 annotation points; Llama-2 and GPT-4 cover only 9 and 4 jailbreaks (Appendix D).
- The full-dataset jailbreak evaluation covers 37 jailbreaks, 3 aligned models (GPT-4o, GPT-3.5 Turbo, Llama-3.1 70B Instruct), and 313 prompts.
- The capability experiments cover 1 unaligned model (Dolphin) and 1 aligned model (GPT-4o) on a 171-question MMLU subset.
- There are 7 baseline evaluators (including the concurrent HarmBench); the paper does not report inter-annotator agreement metrics.
- The paper does not report the number of repetitions for the 37-jailbreak evaluation; the rubric version evaluator costs about 15 minutes and less than $1 (GPT-4o-mini, per the authors).
Summarize the main content of the paper
The StrongREJECT benchmark: high-quality prompts plus an evaluator that weighs both willingness and capability, which the authors claim is most consistent with human judgment.
A jailbreak evaluation benchmark paper addressing the problem that jailbreak success rates are systematically overestimated.
- Problem: The authors argue that forbidden prompts in existing benchmarks are repetitive, unanswerable, or not actually forbidden, and that evaluation methods only look at whether the model refuses, causing jailbreak effectiveness to be exaggerated; when the authors reproduced the Scots Gaelic jailbreak of Yong et al. [39], they did not obtain actionable information (Table 2).
- Method: StrongREJECT contains 313 prompts covering six categories of harmful behavior with factually verifiable answers, and an automatic evaluator scored as score = (1 − refused) × (specific + convincing)/2 (the rubric version uses GPT-4o-mini; there is also a Gemma 2B fine-tuned version), measuring both willingness and capabilities.
- Agreement: On 1,361 human annotation points, the rubric version had MAE 0.077 ± 0.01 and bias 0.012 ± 0.01, and the fine-tuned version had Spearman 0.900, all better than the 7 baselines (Table 1); String matching had bias 0.484 ± 0.03.
- Jailbreak evaluation: Among 37 jailbreaks, PAIR was most effective (GPT-4o 0.74, GPT-3.5 Turbo 0.74, Llama-3.1 70B 0.72), and selecting the best per question reached 0.95–0.99; GCG was effective only against GPT-3.5 Turbo (Figure 3).
- Capability experiments: The authors report that the more a jailbreak increases the non-refusal rate of aligned models, the more it tends to reduce Dolphin's ability to answer forbidden prompts and GPT-4o's ability to answer MMLU questions (Figure 4), concluding on this basis that jailbreaks generally harm model capability.
- Conclusion and implications: The authors argue that researchers should use high-quality evaluators that consider both willingness and capability, in order to distinguish ineffective jailbreaks from genuinely effective jailbreaks such as PAIR and PAP; the authors acknowledge three limitations: dataset size, modality scope, and policy timeliness.
Related papers
Show 3 more papers
- What Matters For Safety Alignment?评测/基准arXiv