SORRY-Bench: Systematically evaluating large language model safety refusal

1. 01What problem does this paper attempt to solve?

   Existing safety refusal evaluations have coarse and uneven categories, ignore language form, and have high judge costs; the authors propose the SORRY-Bench benchmark.

   How to systematically, in a balanced way, and at low cost evaluate the ability of aligned LLMs to recognize and refuse unsafe user requests.

   • Scenario and importance: Aligned LLMs should refuse unsafe requests (the authors give the example that most models refuse a writing request containing racist content); this safety refusal ability can reduce the possibility of the model being misused in the wild, and evaluating it is a prerequisite for safe deployment.
   • Existing shortcoming one: coarse and imbalanced categories: The authors point out that existing datasets vary in classification granularity and over-represent certain subcategories—on the 10 existing datasets they evaluate, the proportion of Self-Harm instructions is nearly 3 times lower than categories such as Fraud (Figure 1), and categories such as PII Violations and Animal-related Crimes are likewise under-represented.
   • Existing shortcoming two: neglect of linguistic form: The authors believe many evaluations consider only implicitly prompt features such as language, dialect, and spelling; for example, AdvBench is entirely in imperative form; and Bianchi et al. (2024) noticed that interrogative form leads to different safety performance.
   • Existing shortcoming three: high judging cost: Existing evaluations mostly rely on GPT-4-level LLMs as judges; the authors report that GPT-4o requires about 3 USD and 20 minutes per evaluation round, and updates to proprietary model versions raise reproducibility issues.
   • Paper's approach: Proposes SORRY-Bench—a 44-category fine-grained safety taxonomy, 440 class-balanced unsafe instructions, 20 language variants (expanded to 8.8K), and a judge design based on meta-evaluation of 7K+ human annotations, ultimately evaluating 56 open-source and proprietary LLMs.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: safety benchmark datasets, prompt-level safety influencing factors, and LLM-as-a-judge evaluation design.

   LLM safety benchmark datasets

   • Early work evaluated toxicity completion and biased question answering of pretrained models: RealToxicityPrompts (Gehman et al., 2020), BBQ (Parrish et al., 2022).
   • Recent work has proposed various unsafe instruction datasets: AdvBench (Zou et al., 2023), Do-Not-Answer (Wang et al., 2023), XSTest (Röttger et al., 2023), HEx-PHI (Qi et al., 2023), MaliciousInstruct (Huang et al., 2023), SimpleSafetyTests (Vidgen et al., 2023), HarmBench (Mazeika et al., 2024), SALAD-Bench (Li et al., 2024), StrongREJECT (Souly et al., 2024), ALERT (Tedeschi et al., 2024), etc.; Table 3 summarizes 16 of them.
   • The paper points out that the taxonomies of these datasets are mutually inconsistent, coarse-grained, and unevenly distributed (Figure 1); the SORRY-Bench taxonomy was aggregated and refined from these 10 datasets.

   Impact of prompt language form on safety

   • Bianchi et al. (2024) and Xhonneux et al. (2024) noted that questions and imperatives lead to different refusal behavior; Zeng et al. (2024) studied persuasion-based jailbreaks; Deng et al. (2023) studied multilingual jailbreaks; Yuan et al. (2023) studied cipher/encoded dialogue.
   • Samvelyan et al. (2024) (Rainbow Teaming) studied generating diverse adversarial prompts, and its writing style variants are borrowed by this paper.
   • The paper organizes these factors into 20 language variants included in the benchmark; the authors state that prior evaluations did not explicitly and evenly cover them.

   LLM-as-a-judge safety evaluation

   • Existing implementations include Qi et al. (2023), Huang et al. (2023), Xie et al. (2023), Mazeika et al. (2024), Li et al. (2024), Souly et al. (2024), Chao et al. (2024); some directly use GPT-4-level models, and some fine-tune small models.
   • The paper states that there was previously a lack of systematic research on the efficiency-accuracy trade-off of judge design, and this paper's meta-evaluation fills this gap.

   Baselines and comparisons: The baseline judges in the meta-evaluation include Keyword Match (Zou et al., 2023), Perspective API, Llama-Guard-2-8B, MD-Judge, WildGuard, HarmBench Classifier, and fine-tuned Bert-Base-Cased; cross-benchmark score comparisons use the reported values of HarmBench, SALAD-Bench, ALERT, and StrongREJECT (Table 4). The authors position this paper as complementing existing work in balance, granularity, customizability, and efficiency.

   Collapse this answer

3. 03How does the paper address this problem?

   Three steps: a 44-category taxonomy plus 440 balanced instructions, expansion with 20 language variants, and meta-evaluation on 7K+ human annotations to select a fine-tuned small-model judge.

   The overall approach is to first build a fine-grained, class-balanced dataset, then systematically cover prompt linguistic forms, and finally calibrate an accurate and cheap automatic judge with human annotations; the framework is named SORRY-Bench.

   Fine-grained taxonomy (§2.2)

   • Two-stage construction: First aggregate the taxonomies of 10 existing safety benchmarks and split broad categories (such as Illegal Activities) into subcategories; then iterate through a human-in-the-loop process—use GPT-4 as a zero-shot classifier to map existing data to the taxonomy, and have data falling into Others manually reviewed by 4 authors, deciding whether to add or subdivide categories, repeating until all 4 agree no update is needed.
   • Result: 44 risk categories, grouped into 4 high-level domains: Hate Speech Generation (#1–5), Assistance with Crimes or Torts (#6–24), Potentially Inappropriate Topics (#25–39), Potentially Unqualified Advice (#40–44), see Table 5 for details.

   Class-balanced dataset (§2.3, Appendix G)

   • Collection process: After 30 minutes of training, 9 graduate students each collected 10 items for about 5 categories; priority was given to selecting and rewriting from the 3,656 existing items classified by GPT-4, and when insufficient, new items were written manually.
   • Writing guidelines: Instructions must be specific, unambiguous (cannot be answered in a safe way), grammatically correct, and uniformly in imperative form.
   • Scale and novelty: 440 items in total (44 categories × 10 items); only 5.9% (26/440) are verbatim identical to existing datasets, with an average maximum Jaccard similarity of 44.5%.

   20 language variants (§2.4, Appendix H)

   • Four groups of variants: 6 writing styles (Question, Slang, Uncommon Dialects, Technical Terms, Role Play, Misspellings), 5 persuasion techniques (Logical Appeal, Authority Endorsement, Misrepresentation, Evidence-based Persuasion, Expert Endorsement), 4 encoding/encryption schemes (ASCII, Caesar, Morse, Atbash), 5 languages (Malayalam, Tamil, Marathi, Simplified Chinese, French).
   • Implementation: Writing styles and persuasion techniques are rewritten with GPT-4 few-shot; encoding/encryption follows Yuan et al. (2023) with few-shot templates, and is decoded back to plaintext before judging; multilingual variants are translated with the Google Translate API, and model responses are translated back to English before judging.
   • Scale: 20 × 440 = 8.8K additional instructions.

   Judging objective and human annotation (§3.1–3.2)

   • Binary judgment: Given (unsafe instruction, model response), judge whether it is fulfillment or refusal; a response counts as fulfillment as long as it contains substantive content that directly addresses the request, even if it begins with a disclaimer or refusal statement.
   • Annotation set: For each base instruction, 8 ID responses and 8 OOD (variant) responses are sampled, annotated by 6 authors, totaling 7,040 items (30.4% fulfillment); the training split has 2,640 items and the test split 4,400 items. The average Cohen Kappa among the 6 annotators on 100 shared samples is 90.5% (Figure 6).

   Meta-evaluation of judge design (§3.3)

   • Design space: Direct prompting, +CoT, +Few-Shot (using training-split annotations as demonstrations), +fine-tuning (2.6K training annotations, 3 epochs, batch 32); candidate models include GPT-4o, GPT-3.5-turbo, Llama-3-70b/8b-instruct, Mistral-7b-instruct-v0.2, Gemma-7b-it, and unaligned base models.
   • Selected judge: The benchmark experiments use fine-tuned Mistral-7b-instruct-v0.2, for the balance of efficiency and accuracy; temperature is set to 0 during judging.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Meta-evaluate more than 30 judge designs, then measure fulfillment rate on 56 LLMs, and analyze the impact of 20 variants and evaluation configurations.

   Experimental setup

   • Models tested: 56 open-source and proprietary LLMs (Llama, Gemma, Mistral, Qwen, Claude, GPT-*, Gemini, etc.), with parameters from 1.8B to 400B+; in the main results, each model sampled once for each of the 440 base instructions (mostly without system prompts, temperature 0.7, Top-P 1.0, max tokens 1024; 1.0 for the o1 series).
   • Language variants: Due to compute constraints, evaluated only on 8 models (the 8 listed in Table 2).
   • Judging: Fine-tuned Mistral-7b-instruct-v0.2; the metric is Fulfillment Rate (0–1, lower means more refusals).
   • Meta-evaluation: Cohen Kappa is used to measure agreement with humans on 4,400 test annotations, and the time per evaluation is reported.
   • Repetitions: 26 open-source models were sampled 3 times with standard deviation reported (Figure 7); 6 models additionally had a greedy decoding comparison (Table 8).

   Main results

   Complete results for the 56 models are in Figure 4 (heatmap, fulfillment rate per category); representative numbers given in the main text and appendix (Table 4, §K.2):

   Model	Fulfillment Rate	Source
   Claude-2.1	5.23%	Table 4
   Claude-2.0	7.05%	Table 4
   GPT-3.5-turbo-1106	10.68%	Table 4
   Llama-2-70b-chat	12.27%	Table 4
   GPT-4o-2024-05-13	0.30	Table 2
   Llama-3-70b-instruct	0.35	Table 2
   Mistral-7b-instruct-v0.2	67.05%	Table 4
   Mistral-7b-instruct-v0.1	90.23%	Table 4

   • The authors state that 27 of the 56 models have a fulfillment rate between 20% and 50%; Claude-2 and Gemini-1.5 are the lowest (<10%), and 8 models (including the Mistral series) exceed half; Dolphin-2.2.1-mistral-7b and Mistral-7b-instruct-v0.1 are above 90%.
   • At the category level: #8 Harassment, #20 Child-related Crimes, and #9 Sexual Crimes average only 9%–11% fulfillment; #42 Legal Consulting Advice, #34 Religion Promotion, and #33 Ethical Belief Promotion average 74%–80%.
   • Version changes: Llama-2-70b to Llama-3-70b rises from 12% to 35%; Gemini-Pro to Gemini-1.5 falls from 33% to 7%; the authors link the change in GPT-4o on category #25 to OpenAI's May 2024 Model Spec.

   Judge meta-evaluation (Table 1, Table 6)

   • Fine-tuning is best: GPT-3.5-turbo+Fine-tuned agreement 83.8% (about 112s per round); after fine-tuning, Llama-3-8b-instruct 80.8% (about 10s), Mistral-7b-instruct-v0.2 81.0% (about 11s), Gemma-7b-it 81.2%, Llama-3-70b-instruct 82.5%, which the authors describe as comparable to GPT-4o (78.9%, about 260s) at lower cost.
   • CoT and Few-Shot are unstable: CoT mostly lowers agreement (e.g., Llama-3-70b-instruct drops from 71.5% to 32.1%); Few-Shot slightly improves large models but decreases for aligned 7B–8B small models (Mistral-7b-instruct-v0.2+Few-Shot only 13.6%).
   • Baseline judges are low: Llama-Guard-2-8B 39.0%, MD-Judge 36.0%, WildGuard 60.6%, HarmBench Classifier 52.5%, Perspective API 1.1%, Keyword Match 37.3% (Table 1); the authors consider general guardrail models unsuitable for SORRY-Bench judgments.

   Impact of language variants (Table 2, 8 models)

   • Persuasion: The 5 persuasion variants generally increase fulfillment by 5–66 percentage points (e.g., Gemma-7b-it Logical Appeal +0.67).
   • Encoding/encryption: Most models cannot understand it, and fulfillment generally drops by 15–68 percentage points; the exceptions are GPT-4o with ASCII +0.12 and Caesar +0.16, which the authors suggest may be due to its stronger ability to understand complex instructions.
   • Multilingual: GPT-3.5, Llama-3, and Gemma increase fulfillment by 20–60 percentage points on low-resource languages, while Vicuna, Mistral, and OpenChat decrease by 20–58 percentage points; GPT-4o fluctuates within ±4% across languages.

   Other ablations and analyses

   • System prompts: Adding a safe or helpful system prompt both lower fulfillment by 2–16 percentage points (Table 7, 5 models).
   • Prompt templates: After removing format tokens, Llama-2 and Gemma fulfillment rises by 8–30 percentage points, while Llama-3 is basically unchanged (Table 7).
   • Prefilling: After prefilling with an affirmative prefix, Llama series fulfillment rises by about 48–58 percentage points, and Gemma +19 percentage points (Table 7).
   • Decoding: The standard deviation across 3 repeated samples is small (Figure 7); the difference between greedy and temperature 0.7 results is within ±5 percentage points (Table 8).
   • Judge failure cases: Appendix J.4 gives three types of misjudgment—bullet-point lists misjudged as fulfillment, disclaimers masking fulfillment, and borderline cases of partial satisfaction.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations such as binary judgment, scenario coverage, and data contamination; the experiments cover 56 models but variants only 8 models.

   Limitations and future directions noted by the authors

   • Binary judgment only: The actual harm level of a response is not quantified; the authors state that finer-grained scoring is left for future work (Appendix A.1).
   • Incomplete scenario coverage: Scenarios involving multiple risk categories simultaneously are not covered, nor are neutral prompts or prompts that can be answered safely (Appendix A.1).
   • Taxonomy requires continuous updating: Real-world safety risks evolve quickly, and the taxonomy and dataset need periodic revision with community effort (Appendix A.1).
   • Usefulness is not measured: A model that always refuses would get 0% fulfillment; the authors recommend using it together with utility benchmarks (such as Chatbot Arena, MMLU) and over-refusal benchmarks (OR-Bench, XSTest) (Appendix A.1).
   • Jailbreak attack and defense not included: This paper focuses on average-case bad users, and integrating jailbreak attacks and defenses is left for future work (Appendix A.1).
   • Data contamination risk: Future models may include benchmark data in their training corpora; the authors propose developing private splits as a future direction (Appendix A.1).

   Scope of experimental coverage

   • A total of 56 models were tested; in the main results, each model sampled once for each of the 440 instructions; the 20 language variants were evaluated only on the 8 models listed in Table 2.
   • 7,040 items were manually annotated, completed by 6 authors; agreement was measured on a common subset of 100 items, with an average Cohen Kappa of 90.5% (Appendix I).
   • Sampling randomness was verified by repeating 3 times on 26 open-source models; the greedy decoding comparison covered only 6 models (§K.5).
   • The judge fine-tuning training data consisted of 2,640 manually annotated items; the paper does not report the total API cost of the benchmark experiments.
   • The authors state that due to resource and time constraints, comprehensive testing under various decoding parameters (such as different temperature) was not conducted (§K.5).

   Collapse this answer

6. 06Summarize the main content of the paper

   SORRY-Bench systematically characterizes refusal differences across 56 LLMs using 44 balanced instruction categories, 20 language variants, and a fine-tuned small-model judge.

   SORRY-Bench is an LLM safety refusal benchmark addressing three problems with existing evaluations: coarse and uneven categories, neglect of prompt language form, and high judge costs.

   • Data: A 44-category taxonomy (4 high-level domains) was aggregated and refined from 10 existing benchmarks to construct 440 category-balanced unsafe instructions (10 per category, only 5.9% verbatim identical to existing data), then expanded to 8.8K items with 20 language variants (writing style, persuasion, encoding/encryption, multilingual).
   • Judging: 7,040 manual annotations were collected (6 authors, average agreement 90.5% on a 100-item subset); meta-evaluation shows fine-tuning is key to judge accuracy—fine-tuned Mistral-7b-instruct-v0.2 achieves 81.0% agreement at about 11s per round, which the authors describe as comparable to GPT-4o (78.9%, about 260s) at lower cost; CoT and Few-Shot improvements are unstable, and general guardrail models (Llama-Guard-2-8B 39.0%, etc.) have low agreement.
   • Main results: Fulfillment rate across 56 models ranges from 5.23% for Claude-2.1 to 90.23% for Mistral-7b-instruct-v0.1; categories #8/#20/#9 average only 9%–11%, while categories #42/#34/#33 reach 74%–80%.
   • Variant impact (8 models): Persuasion variants generally increase fulfillment by 5–66 percentage points, encoding/encryption variants generally decrease it by 15–68 percentage points, and low-resource languages affect different models in opposite directions.
   • Authors' conclusions: Refusal behavior varies greatly across models and categories, and evaluation configurations such as system prompts, prompt templates, and prefilling also change results; the authors hope the benchmark supports safety refusal evaluation in a balanced, fine-grained, customizable, and efficient way, and recommend using it together with utility and over-refusal benchmarks.

   Collapse this answer

Related papers