HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

1. 01What problem does this paper attempt to solve?

   Automated red teaming lacks standardized evaluation; the authors propose the HarmBench framework along with the adversarial training defense R2D2.

   Core problem: the automated red-teaming field lacks a standardized evaluation framework, making it difficult to fairly compare methods and hindering the coordinated development of attacks and defenses.

   • Scenario and motivation: the risk of malicious use of LLMs (malware, social engineering, bioweapon design, etc.) makes red teaming a key means of discovering defense vulnerabilities before deployment, but manual red teaming scales poorly and cannot cover adversarial and long-tail scenarios, so the field has turned to automated red teaming (Section 1).
   • Shortcomings of existing evaluations: the authors' survey finds that prior work has at least 9 distinct evaluation setups (Table 1) that do not overlap for comparison; most evaluations use fewer than 100 unique behaviors, and these are short, single-modality behaviors; a key parameter in evaluation, the number of generated tokens, is not standardized and can change ASR by as much as 30% (Figure 2); some work evaluates directly on the metric that the method optimizes, risking metric gaming (Section 3.2).
   • Core observation: the authors propose that red-teaming evaluation should have three properties—breadth, comparability, and robust metrics—and design the benchmark systematically on this basis.
   • Paper outputs: the HarmBench framework (510 behaviors + standardized evaluation pipeline + evaluation classifier), a large-scale comparison of 18 attacks × 33 models, and the adversarial training defense method R2D2; the authors state that the comparison yields conclusions such as "no attack or defense is consistently effective, and robustness is unrelated to model scale."
   • Threat model (Appendix B.3):
     • Attacker goal: design test cases to elicit specified harmful behaviors from a target LLM (targeted red teaming).
     • Attacker knowledge/access: three tiers—no access (transfer attacks), query access (black-box attacks), and parameter access (white-box attacks).
     • Victim system: an instruction-following LLM (including multimodal LLMs); defenses are divided into model-level and system-level, and the large-scale comparison in this paper focuses on model-level defenses.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into three lines: red-teaming methods, defenses, and adversarial training; the authors state that prior evaluations do not overlap and cannot be compared.

   Automated red-team methods

   • Text optimization type: GCG (Zou et al., 2023), PEZ (Wen et al., 2023), GBDA (Guo et al., 2021), AutoPrompt (Shin et al., 2020), UAT (Wallace et al., 2019), and other attacks based on token-level optimization.
   • LLM optimizer type: Zero-Shot / Stochastic Few-Shot (Perez et al., 2022), PAIR (Chao et al., 2023), TAP (Mehrotra et al., 2023), which use an attacker LLM to generate or iteratively test test cases.
   • Template and pipeline type: AutoDAN (Liu et al., 2023b), PAP (Zeng et al., 2024), Human Jailbreaks (Shen et al., 2023a); there are also image attacks against multimodal LLMs (Bagdasaryan et al., 2023; Qi et al., 2023a, etc.).

   Defenses

   • System-level: input/output filtering (Markov et al., 2023; Inan et al., 2023, etc.), input sanitization and rewriting, restricted inference; Glukhov et al. (2023) point out that output filtering can be bypassed by jailbroken models (e.g., generating encoded output).
   • Model-level: safety training and refusal mechanisms such as RLHF, DPO, RLAIF; the authors point out that model-level defenses have been little explored beyond safety fine-tuning.

   Adversarial training and existing evaluations

   • Jain et al. (2023) use a static harmful prompt dataset for adversarial training, with the attacker not optimizing against the model during fine-tuning, and report negative results; Ge et al. (2023) (contemporaneous with this paper) propose multi-round adversarial training, generating new test cases 4 times during training.
   • Existing behavior datasets: AdvBench (58 behaviors), TDC 2023 (99), MaliciousInstruct (100), Shen et al. (390), etc. (Table 5); the authors point out that these datasets have few behaviors and no multimodal or contextual categories.

   Baselines and benchmark: the baselines compared in this paper are 18 red-team methods (including the two baselines Human Jailbreaks and Direct Request), and the benchmark is HarmBench, built by the authors; on the defense side, the comparison is each existing LLM's own safety training (e.g., the Llama 2 Chat series).

   Authors' positioning: prior evaluation setups do not overlap and relative performance is unclear (Table 1); HarmBench fills this gap with a unified pipeline, a larger and more diverse behavior set, and a robust classifier.

   Collapse this answer

3. 03How does the paper address this problem?

   HarmBench = 510 behaviors + a three-step standardized evaluation pipeline + a robust classifier; they also propose R2D2 adversarial training.

   Overall approach: build the HarmBench evaluation framework according to three criteria — "breadth, comparability, robust metrics" — and use it for large-scale comparison; at the same time propose R2D2 (Robust Refusal Dynamic Defense) to demonstrate co-development of attacks and defenses.

   Problem Formalization and Metrics

   • Red-team task: given a behavior y, a red-team method g generates a set of test cases, the target model f generates a completion of T tokens with greedy decoding, and a classifier c judges whether it succeeds.
   • The main metric is attack success rate ASR(y, g, f) = 1/N ∑_i c(f_T(x_i), y), i.e., the proportion of test cases that elicit the behavior.
   • Standardized number of generated tokens N = 512, because the authors found that this parameter can change ASR by as much as 30% (Figure 2).

   Behavior Set Construction

   • Scale: 510 unique harmful behaviors = 400 text + 110 multimodal; the official split is 100 validation and 410 test, with the requirement not to tune parameters on the test set.
   • 7 semantic categories: Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Copyright Violations, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, General Harm.
   • 4 functional categories: standard (200), copyright (100), contextual (100, containing contextual strings), multimodal (110, containing images).
   • Curation principles: manually designed using the usage policies of OpenAI, Anthropic, Meta, and Inflection AI (summarized with GPT-4) as guidelines; filtering out "dual-intent" behaviors (e.g., writing an encryption function); emphasizing differential harm — Table 12 shows that the Google searchability rate of contextual behaviors is 0%, while MaliciousInstruct is 55% and AdvBench is 50% (20 sampled per dataset, 10 minutes each).

   Evaluation Pipeline and Classifier

   • Three-step pipeline (Figure 3): generate test cases → target model (including defenses) generates completions → classifier computes ASR.
   • Success criterion: a completion that exhibits or clearly attempts to exhibit the behavior counts as success (even if the code is wrong), to decouple model capability from safety mechanisms; the full criteria are in Appendix B.1.
   • Non-copyright behaviors: a fine-tuned Llama 2 13B Chat serves as the classifier, with training data distilled from GPT-4 over multiple rounds (15 rounds); copyright behaviors: a MinHash-based hash classifier, requiring the completion to actually contain the copyrighted text.
   • Average agreement between the classifier and human labels is 93.19% (Table 3); a validation classifier based on Mistral 7B (88.6% agreement) is also provided for use when optimizing methods, to prevent directly optimizing the test metric.

   R2D2 Adversarial Training

   • Motivation: GCG takes 20 minutes to generate a single test case on a 7B model (A100), making it difficult to incorporate into the training loop; drawing on fast adversarial training, a pool of test cases persisted across batches is used.
   • Pipeline (Algorithm 1): each round, sample n test cases from the pool, update them with GCG on the current model for m steps, then compute the model loss and update parameters.
   • Loss: L_total = L_away + L_toward + L_SFT — the away loss is adversarial to the GCG target string, the toward loss pushes toward a fixed refusal string, and the SFT loss maintains utility.
   • Every L model updates, randomly reset K% of the test cases to increase diversity.

   Collapse this answer

4. 04What experiments did the paper conduct?

   A large-scale comparison of 18 attacks × 33 models, plus the R2D2 defense, multimodal attacks, and classifier evaluation.

   Experimental Setup

   • Models tested: 33 — 24 open-source text models (Llama 2 7B/13B/70B Chat, Vicuna 7B/13B, Baichuan 2 7B/13B, Qwen 7B/14B/72B Chat, Koala 7B/13B, Orca 2 7B/13B, SOLAR 10.7B, Mistral 7B, Mixtral 8x7B, OpenChat 3.5, Starling 7B, Zephyr 7B, R2D2), 8 closed-source text models (GPT-3.5 Turbo 0613/1106, GPT-4 0613, GPT-4 Turbo 1106, Claude 1/2/2.1, Gemini Pro), 4 multimodal models (InstructBLIP, LLaVA 1.5, Qwen-VL-Chat, GPT-4V).
   • Attack methods: 16 text attacks (GCG, GCG-M, GCG-T, PEZ, GBDA, UAT, AutoPrompt, SFS, ZS, PAIR, TAP, TAP-T, AutoDAN, PAP-top5, Human Jailbreaks, Direct Request); 4 multimodal attacks (PGD, Adversarial Patch, Render Text, Direct Request).
   • Benchmark: HarmBench 510 behaviors (410 test set, 100 validation set).
   • Metric and judging: ASR, judged by a fine-tuned Llama 2 13B Chat classifier (non-copyright) and a MinHash hash classifier (copyright); classifier-human agreement 93.19% (Table 3).
   • R2D2 setup: fine-tuned on Mistral 7B base, M = 500 steps, N = 180 persistent test cases, m = 5 GCG steps per round, n = 8 cases, reset K = 20% every L = 50 steps; 16 hours on 8xA100; SFT data is UltraChat.
   • The paper does not report the number of repetitions for each attack × model combination.

   Main Results

   Table 6 (all behaviors, ASR%) selected key rows; "—" means not reported by the paper (white-box attacks do not apply to closed-source models):

   Swipe horizontally to read the table

   Model	GCG	PAIR	TAP	AutoDAN	Human	DR
   Llama 2 7B Chat	32.5	9.3	9.3	0.5	0.8	0.8
   Llama 2 13B Chat	30.0	15.0	14.2	0.8	1.7	2.8
   Vicuna 7B	65.5	53.5	51.0	66.0	39.0	24.3
   Mistral 7B	69.8	52.5	62.5	71.5	58.0	46.3
   Zephyr 7B	69.5	58.8	66.5	75.0	66.0	65.8
   R2D2 (Ours)	5.5	48.0	60.8	17.0	13.6	14.2
   GPT-4 0613	—	39.3	43.0	—	11.3	21.0
   Claude 2.1	—	2.8	2.5	—	0.3	2.0

   (According to Table 6; the remaining 25 models are omitted, among which the highest ASR is for models without safety training such as Zephyr 7B, and the lowest is for Claude 2/2.1.)

   • The authors state that no attack or defense is consistently effective: all attacks have low ASR on at least one model, and all models have high ASR under at least one attack (Figure 5).
   • The authors state that robustness is unrelated to model scale: within the same model family, ASR from 7B to 70B is close, while differences between families are large (Figure 6), based on which the authors argue that training data and algorithms matter more than scale; the exception is that ASR for copyright behaviors rises on the largest models, which the authors hypothesize is due to insufficient capability of small models.
   • ASR for contextual behaviors is higher than for standard behaviors, and ASR for copyright behaviors is low (Figure 11); the authors state that the latter is because the hash classifier's standard is stricter, and call the high ASR for contextual behaviors "concerning" because their differential harm is greater.

   R2D2 Defense Effectiveness

   • What was tested: robustness of Zephyr 7B + R2D2 against all attacks. Results: GCG ASR 5.5, lower than Llama 2 7B Chat's 31.8 and Llama 2 13B Chat's 30.2 (main text Section 6.2; in Table 6 the R2D2 row's GCG is 5.5); the authors state it is the strongest defense across the three GCG variants, and ranks third in average ASR (behind only Llama 2 7B/13B Chat).
   • Exceptions: smaller improvements against attacks that differ greatly from the GCG used in training — PAIR 48.0, TAP 60.8, SFS 43.5 (Table 6); the authors suggest that incorporating multiple attacks into adversarial training may be needed.
   • Utility: MT-Bench 6.0, versus Mistral 7B Instruct v0.2's 6.5 (Table 11); the authors state that adversarial training can improve robustness while maintaining general performance. Figure 8 shows that the loss cannot be driven down when GCG optimizes against the R2D2 model.

   Multimodal Attacks

   • What was tested: 4 attacks on 4 multimodal models (Table 9). Results: PGD-type attacks have high ASR (Qwen-VL-Chat 82.73, LLaVA 1.5 74.55, InstructBLIP 64.55), Render Text is low (1.82–20.00); GPT-4V PGD results are not reported.
   • The authors state that this corroborates findings in prior work that multimodal attacks can be stronger than text attacks.

   Classifier Evaluation

   • Agreement (Table 3): HarmBench classifier averages 93.19, GPT-4 is 88.37, AdvBench substring matching 69.93, GPTFuzz 75.42, ChatGLM 64.29, Llama-Guard 66.94.
   • Robustness pre-check (Table 4, three types of non-standard completions): HarmBench classifier averages 95.7, GPT-4PAIR 89.6, Llama Guard 74.2, GPTFuzz 65.8, AdvBench 32.0.

   Other Ablations and Analyses

   • Average ASR is close across semantic categories, except copyright (Figure 9); on individual models the category differences are large, e.g., Misinformation is highest on Llama 2 and GPT, Chemical & Biological is highest on Baichuan 2 and Starling (Figure 10).
   • Validation set and test set results are listed in Table 7 and Table 8 respectively, and are close to Table 6.
   • Multimodal attacks on text behaviors (Table 10): PGDBlankImage is 66.00 on LLaVA 1.5, RenderText is only 0.50–5.25.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that system-level defenses and automated behavior generation remain to be done; the experiments focus on model-level defenses and single-turn evaluation.

   Limitations and Future Directions Noted by the Authors

   • The large-scale comparison covers only model-level defenses; system-level defenses require adaptive attacks tailored to individual defenses to determine true robustness, which the authors explicitly leave for future work (Appendix B.3).
   • The evaluation focuses on manually curated behavior sets; automating the full red-teaming pipeline (including the selection of harmful behaviors) "still has work to be done," and the authors believe full automation may be challenging or inadvisable (Appendix E, Q15).
   • R2D2 yields smaller improvements against attacks that differ greatly from the GCG used in training (PAIR, TAP, Stochastic Few-Shot); the authors suggest that incorporating multiple different attacks into adversarial training may be needed to obtain generalizable robustness (Section 6.2).
   • The authors state that to obtain true robustness against all known attacks, training only against a limited attack set and expecting generalization may not be sufficient (Section 6.1).
   • Copyright behaviors show increased ASR on the largest models; the authors hypothesize that this is because smaller models cannot complete copyright behaviors, and this is only a hypothesis (Section 6.1).

   Scope of Experimental Coverage

   • 33 models tested in total: 24 open-source text, 8 closed-source text, and 4 multimodal (Appendix C.2).
   • 18 attack methods: 16 text and 4 multimodal; Perez et al.'s Supervised Learning and Reinforcement Learning, ARCA, and MasterKey were not included due to cost or code reasons (Appendix C.1).
   • In some methods, the closed-source LLM in the original implementation was replaced with Mixtral 8x7B; the authors note that this may reduce their effectiveness (Appendix C.1).
   • The human-annotated validation set for the classifier consists of 600 completions (Appendix B.5.1); the searchability experiment uses 20 behaviors per dataset (Table 12).
   • The paper does not report the number of repetitions for each attack × model combination; the Claude and Gemini APIs contain non-removable filters, and the authors note that the robustness of their model-level defenses cannot be directly measured (Appendix C.2).

   Collapse this answer

6. 06Summarize the main content of the paper

   HarmBench standardized red-team evaluation: 510 behaviors, 18 attacks × 33 models, plus the R2D2 adversarial training defense.

   A red-team evaluation benchmark paper: addressing the lack of unified evaluation for automated red teaming and the inability to compare across papers, the authors propose HarmBench and use it to conduct a large-scale attack-defense comparison, while also providing the adversarial training defense R2D2.

   • Problem: there were previously at least 9 non-overlapping evaluation setups (Table 1), behavior sets were generally fewer than 100, key parameters such as the number of generated tokens were not standardized (which can change ASR by as much as 30%, Figure 2), and metrics are easy to game.
   • Method: HarmBench contains 510 behaviors (400 text + 110 multimodal, divided into 7 semantic categories and 4 functional categories), a three-step standardized pipeline, a fine-tuned Llama 2 13B classifier (93.19% agreement with humans, Table 3) plus a MinHash copyright classifier; R2D2 uses a persistent pool of GCG test cases for adversarial training, 16 hours (8xA100).
   • Result one: the authors state that no attack or defense is consistently effective, and that robustness is unrelated to model scale and differs greatly between families (Figure 5, 6).
   • Result two: ASR for contextual behaviors is higher than for standard behaviors (Figure 11), and the authors state that their differential harm is greater and warrants vigilance.
   • Result three: R2D2 reduces GCG ASR to 5.5 (Llama 2 7B Chat is 31.8), MT-Bench 6.0 (Mistral 7B Instruct v0.2 is 6.5); but improvements are limited against attacks that differ greatly, such as PAIR (48.0) and TAP (60.8).
   • Authors' conclusion: standardized large-scale comparison is indispensable for the co-development of attacks and defenses; R2D2 can serve as a strong baseline defense, but generalized robustness may require training with multiple attacks.

   Collapse this answer

Related papers