On evaluating the durability of safety guardrails in open-weight large language models

1. 01What problem does this paper attempt to solve?

   The authors state that durability evaluation of open-weight defenses is itself highly error-prone and that minor changes can overturn the original conclusions.

   Core question: for durable safeguards for open-weight LLMs, is the safety evaluation itself trustworthy? The authors argue that in evaluations of such defenses, some seemingly trivial setting changes can cause the defense to fail, thereby misleading stakeholders into overestimating the durability of the defenses.

   • Background and importance: open-weight (or customizable via fine-tuning APIs) model weights can be directly modified, and defenses such as refusal training can be removed with a few fine-tuning steps; policymakers are treating durable safeguards as a path to managing dual-use risks (Introduction, Appendix B).
   • Shortcomings of existing methods: the authors state that recent methods such as RepNoise and TAR claim to resist fine-tuning attacks, but their evaluations cover only a small attack space, and their claims do not hold under a worst-case threat model (Section 3, 4).
   • Core observation: in the re-evaluation, merely enabling data shuffling to introduce randomness, replacing the fine-tuning trainer implementation, small changes to fine-tuning hyperparameters, or small changes to the evaluation prompt template all suffice to break the two defenses; the defenses' side effects (utility degradation, over-forgetting) are also easily missed in evaluation (Section 3.1–3.5).
   • Paper outputs: a set of case studies on the two defenses, a checklist for avoiding evaluation pitfalls (Appendix A), and recommendations to constrain claims within rigorously tested threat models (Section 4, 5).
   • Threat model (following the paper's restatement of the two evaluated defenses, Section 2.1–2.3):
     • Attacker goal: for RepNoise, maximize the model's satisfaction on HarmfulQA; for TAR, recover unlearned weaponized knowledge via fine-tuning.
     • Knowledge: model weights are open (open-weight).
     • Capability: modify weights only via fine-tuning; for TAR, multiple fine-tuning attacks under limited compute are considered (including LoRA), and other weight edits such as pruning are not considered.
     • Victim system: the open-weight model after release, with the defender unable to intervene after release.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: weight-level defenses and their removal, the two defenses evaluated, jailbreaks and unlearning, and the history of adversarial-example evaluation.

   Weight-level defenses and their fragility

   • Refusal training (SFT/RLHF; Wei et al., 2021; Ouyang et al., 2022; Bai et al., 2022)—training the model to refuse harmful requests, the current industry standard; the paper notes that a few fine-tuning steps (Zhan et al., 2024; Yang et al., 2023; Qi et al., 2024d) or pruning/low-rank modifications (Wei et al., 2024b) can remove it.
   • System-level defenses (moderation, monitoring, access control; OpenAI, 2024; Google, 2024; Inan et al., 2023)—the paper notes that once weights are open, such methods no longer apply.
   • Other explorations of durable defenses (Henderson et al., 2023; Deng et al., 2024)—the paper lists them as early efforts in the same direction and does not compare them in detail.

   The two defenses re-evaluated in this paper

   • RepNoise (Rosati et al., 2024)—pushes the layer-wise representations of HarmfulQA data points toward random noise to remove harmful information; the original paper reports that after fine-tuning on up to 10k HarmfulQA data points, the model can still refuse more than 90% of the questions in the test set.
   • TAR (Tamirisa et al., 2024)—first performs unlearning with Random Mapping, then reinforces with meta-learning; it claims to resist up to 5,000 fine-tuning steps under a red-team evaluation with "28 test-time adversaries". The paper selects these two methods because both explicitly specify a threat model and failure conditions.

   Jailbreaks and harmful-knowledge unlearning (Appendix C)

   • Jailbreak attacks are categorized by access level into black-box querying, white-box gradients, fine-tuning/weight editing, prefill generation, etc. (Shah et al., 2023; Zou et al., 2023; Qi et al., 2024d; Andriushchenko et al., 2024, etc.).
   • Harmful-knowledge unlearning (Li et al., 2024; Zhang et al., 2024)—reducing misuse risk by forgetting harmful knowledge; Shumailov et al. (2024) point out that attackers can reintroduce forgotten information, and Łucki et al. (2024), concurrently with this paper, question whether unlearning actually occurs.

   Lessons from adversarial-example evaluation (Section 5)

   • Carlini & Wagner (2017a;b), Athalye et al. (2018), Uesato et al. (2018), Tramer et al. (2020), etc.—top-conference defenses were broken after attack strategies were fine-tuned or implementation details changed; the authors draw this analogy in the hope that the durable-safeguards field avoids repeating the same mistakes.

   Baselines and benchmarks: this paper does not propose a new defense but re-evaluates existing defenses; the comparisons are undefended Llama-2-7B-Chat and Llama-3-8B-Instruct, and the benchmarks include BeaverTails, WMDP, HEx-PHI, SORRY-bench and 7 utility benchmarks. The authors position the paper as evaluation-methodology research: it does not target specific methods but shows that evaluation in this field is difficult, and advocates constraining claims within rigorously tested threat models (Section 1).

   Collapse this answer

3. 03How does the paper address this problem?

   Using two defenses as case studies for adversarial re-evaluation: change only one evaluation variable and observe whether the defense fails.

   Overall approach: rather than proposing a new defense, the authors perform an adversarial re-evaluation of RepNoise and TAR—on the original papers' own threat models and checkpoints, changing one evaluation variable at a time (randomness, trainer implementation, hyperparameters, prompt template, evaluation benchmark scope) to observe whether the defense conclusions flip, and then distill an evaluation checklist.

   Re-evaluation targets and setup

   • RepNoise: official checkpoint (a derivative of Llama-2-7B-Chat); the threat model is an attacker who improves HarmfulQA satisfaction only via fine-tuning; its training objective is L_RepNoise = L_retain − α L_forget + β L_noise, where L_noise is a KL term that pushes harmful-sample representations toward Gaussian noise (Appendix D.1).
   • TAR: official TAR-Bio-v1/v2 checkpoints (derivatives of Llama-3-8B-Instruct); the threat model is that the defender first unlearns weaponized knowledge and the attacker recovers it via fine-tuning; the first stage, Random Mapping, pulls forget-set representations toward random vectors, and the second stage uses meta-learning to maximize the attacker's loss (Appendix D.2).
   • Principle: the authors state that all attacks are strictly confined to the original papers' threat models, and that they mainly re-evaluate the configurations the original papers claim to defend (Section 2.2, 2.3).

   Re-evaluation variable design

   • Randomness (Section 3.1): the only difference from the original evaluation is enabling shuffling of the fine-tuning dataset (the original code disables shuffling and uses greedy decoding, with no randomness); each attack is run independently 5 times with different random seeds.
   • Trainer implementation (Section 3.2): same checkpoint, hyperparameters, data and evaluation pipeline, with only the original authors' self-implemented trainer replaced by the HuggingFace SFT Trainer; Appendix E.1.2 points out three problems in the original implementation: loss computation (loss is also computed on padding and prompt tokens), 75.3% overlap between training and test sets, and data filtering.
   • Fine-tuning configurations (Section 3.3): in addition to the 4 original configurations in Table 1 (Orig-Config 1–4), 2 new configurations are designed: New-Config 1 changes warmup from 30 steps to 100 steps and adds cosine decay, and New-Config 2 fine-tunes on the retain set; the rest (LR 2×10⁻⁵, AdamW, 1000 steps, batch 64) is unchanged.
   • Prompt template and answer extraction (Section 3.4): in addition to the WMDP official zero-shot QA format, the Llama-3 chat template is added (Figure 5), and GPT-4o and humans extract answers from free-text outputs, compared with the logits-ranking method; 3 seeds per configuration.

   Side-effect and mechanism checks

   • Benign-data fine-tuning (Section 4): RepNoise is fine-tuned with 100 AOA identity-shift examples and 100 purely benign Alpaca Salient examples to test the criterion that "unrelated data should not recover forgotten capabilities"; on the TAR side, New-Config 2 (retain set) is used for the same kind of check.
   • Side-effect evaluation (Section 3.5): a self-built BenignBio (100 benign biology questions, generated by GPT-4o and manually verified), with GPT-2 perplexity measuring output fluency; utility and safety are measured on MMLU, GSM8K, MATH, BBH, HumanEval, MT-Bench, TruthfulQA as well as Sorry-Bench and HEx-PHI, with 5 runs per metric and 95% confidence intervals reported.
   • Mechanism analysis (Section 4): loss and gradient-norm curves for the early fine-tuning of the TAR checkpoint are plotted (Figure 6) to locate the relationship between the failure of the original configuration and the initial gradient explosion.

   Outputs

   • Appendix A gives 5 checklist items: repeat with multiple seeds and report worst-case results; use attack implementations that have been extensively tested; match the threat model with evaluation coverage; cover common benchmarks in evaluation to detect side effects; be cautious in claiming "unlearning" (refuted by two tests: prompt-template variation and fine-tuning on unrelated data).

   Collapse this answer

4. 04What experiments did the paper conduct?

   The re-evaluation shows that changing seeds, changing the trainer, altering hyperparameters or changing the template can all break both defenses, and that TAR has large side effects.

   Experimental setup

   • Checkpoints tested: Llama-2-7B-Chat-RepNoise (official release), TAR-Bio-v1 and TAR-Bio-v2 (derived from Llama-3-8B-Instruct, v2 reported by default); controls are the undefended Llama-2-7B-Chat and Llama-3-8B-Instruct.
   • Fine-tuning attack configurations: on the RepNoise side, a subset of BeaverTails-30k-train, LR ∈ {3×10⁻⁵, 6×10⁻⁵, 8×10⁻⁵}, sample sizes {1000, 10000} (Table 4); on the TAR side, the 4 Orig-Configs and 2 New-Configs of Table 1, all 1000 steps, LR 2×10⁻⁵, AdamW.
   • Safety metrics and judging: BeaverTails scored 0–1 harmfulness by a fine-tuned DeBERTaV3-xsmall; HEx-PHI (330 items) scored 1–5 by GPT-4, with a score of 5 counted as harmful; SORRY-bench-base (450 items) judged 0/1 by a fine-tuned Mistral-7B-Instruct; WMDP reports multiple-choice accuracy in three domains (Bio 1273 questions, Chem 408 questions, Cyber 1987 questions).
   • Answer extraction: WMDP by default ranks candidate tokens by logits; under the chat template variant, answers are extracted from free text by GPT-4o (gpt-4o-2024-05-13) and manually, with failures to extract recorded as E (counted as incorrect); this setting is repeated 3 times.
   • Number of repetitions: 5 random seeds per fine-tuning attack configuration; the benchmark evaluations in Table 2, 3 are each run 5 times with 95% confidence intervals reported.
   • Compute: FLOPS for each configuration are estimated in Table 5, 6; the authors state the new configurations are of the same order of magnitude as the original ones.

   Main results

   The table below summarizes the key re-testing numbers (based on Figure 2, 4 and Table 2, 3; exact values for Figure 1, 3 are not labeled in the figures, see the notes after the table).

   Swipe horizontally to read the table

   Setting	Undefended/control	Defended checkpoint	Source
   Highest WMDP-Bio accuracy of TAR-Bio-v2 after fine-tuning under three Orig-Configs (5 seeds)	original model initially about 70% (green dashed line)	>60%, close to the original model	Figure 2
   WMDP-Bio accuracy of TAR-Bio-v1 after switching chat template	random baseline about 25%	>45% (consistent between GPT-4o and manual extraction)	Figure 4a
   GPT-2 perplexity on BenignBio	3.8±0.1	v1: 5.0±0.5; v2: 16.4±2.11	Table 2
   GSM8K	74.4±0.6	v1: 1.5±0.2; v2: 4.8±0.4	Table 3
   Sorry-Bench harmfulness rate	22.2±0.8	v1: 58.3±2.3; v2: 32.2±1.6	Table 3

   • The authors state that in Figure 1, at LR 6×10⁻⁵ and 8×10⁻⁵, the highest harmfulness score across 5 runs of the RepNoise checkpoint was almost the same as the undefended Llama-2-7B-Chat; after switching to the HuggingFace SFT Trainer, the attack broke the defense more consistently (Figure 1b).
   • The authors state that in Figure 3, New-Config 1 (changing only warmup and cosine decay) restored WMDP-Bio accuracy more consistently than Orig-Config 3, and fine-tuning on the retain set (New-Config 2) also restored it substantially; the LoRA configuration (Orig-Config 4) breaking TAR is a result already reported but not emphasized in the original paper, and the authors reproduced it.
   • Authors' interpretation: the failure of TAR's original configuration stems from gradient explosion early in fine-tuning (Figure 6 shows large gradient norms and loss spikes at the initial steps); New-Config 1 stabilizes training by extending warmup, after which the defense is bypassed, and the authors suggest TAR's effectiveness may depend on this initial instability.

   Fine-tuning on benign data to test unlearning

   • What was tested: fine-tuning the defended models on data unrelated to harmful content, to test whether the "forgotten" capabilities recover.
   • Results: after RepNoise was fine-tuned on only 100 benign AOA or 100 benign Alpaca Salient data points, harmfulness on BeaverTails, HEx-PHI and Sorry-Bench rose substantially (Figure 7, exact values not labeled in the figure); fine-tuning TAR on the retain set also substantially restored WMDP-Bio accuracy (Figure 3).
   • Authors' interpretation: the authors argue that the two defenses do not truly remove information, and that "unlearning" claims should be made cautiously; Appendix F.3 re-tests on completely non-overlapping BeaverTails attack/test sets (4986 attack, 699 test), and the conclusion is unchanged.

   Side-effect evaluation

   • Benign biology questions: TAR-Bio-v2's answer perplexity on BenignBio is higher than v1 and the undefended model (Table 2), with garbled examples given in Appendix G.3; the authors state that v2 fixes v1's template-sensitivity problem but becomes overly sensitive to benign biology knowledge.
   • Utility degradation: in Table 3, the post-TAR models decline on all of MMLU, GSM8K, MATH, BBH, HumanEval, MT-Bench and TruthfulQA, with GSM8K near zero; the authors attribute this to frequently meaningless outputs (examples in Appendix G) and cite Huang et al. (2024a) as also observing mode collapse.
   • Worsening of other safety metrics: harmfulness rates on Sorry-Bench and HEx-PHI rise (Table 3); the authors suggest unlearning defenses do not necessarily reduce, and may even increase, compliance with generally harmful instructions, possibly related to forgetting of refusal training.

   Other ablations and analyses

   • BOS token (Table 7): TAR-Bio-v1's WMDP-Bio accuracy rises from 24.0 (no template, no BOS) to 39.8 (no template, with BOS); the undefended model goes from 71.1 to 73.2.
   • BOS effect on utility (Table 8): TAR-Bio-v2 with BOS added has GSM8K 0.1±0.1, BBH 0.2±0.1, HumanEval 0.0±0.0, versus 4.8±0.4, 24.9±0.6, 21.1±2.6 without BOS; Llama-3-8B-Instruct is basically stable under both prompts.
   • BenignBio four-prompt combinations (Table 9): TAR-Bio-v2 perplexity ranges from 16.4±2.11 to 91.6±25.4, and the undefended model from 3.4±0.1 to 6.0±0.1.
   • Other WMDP domains (Figure 10): the authors state that in the Chem and Cyber domains, fine-tuning under specific configurations and seeds likewise restores accuracy, and cosine decay plus warmup is likewise effective.
   • How results are reported (Section 4): the authors point out that the original TAR paper averages results over 28 adversaries, while the LoRA configuration that breaks the defense accounts for only 2 of them, so the average downweights the most effective attack.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors provide directions and a checklist for improving evaluation; the re-testing covers only two defenses and attacks within their respective original threat models.

   Limitations and future directions noted by the authors

   • Scope of investigation: the authors state that the investigation is limited to these two specific methods, but these pitfalls may also arise when evaluating other defenses, and future work should examine them more carefully (start of Section 3).
   • Evaluation protocols: the authors state that future work needs at least to develop more reliable evaluation protocols that fully account for and mitigate these evaluation pitfalls (Section 4).
   • Constraining claims: the authors recommend that defenders explicitly state whether they provide protection against benign modifications, a limited modification space, or general adversarial robustness, and constrain claims within rigorously tested threat models (Section 4, 5).
   • Adaptive attacks: the authors state that robustness claims within a threat model require "adaptive" attacks constructed specifically against that defense, because small changes to hyperparameters change the attack success rate (Section 5).
   • Standardized evaluation: the authors recommend standardizing and making evaluation comprehensive, e.g., repeating with different random seeds, evaluating safety with multiple metrics and from multiple angles, and extensively testing the utility-safety trade-off (Section 5).
   • Room for method improvement: the authors state that durability may still be improved in the future by modifying these methods, and that these methods may still raise attacker costs in some cases (Section 1).

   Experimental coverage

   • The re-evaluation targets 3 official checkpoints of 2 defenses: Llama-2-7B-Chat-RepNoise, TAR-Bio-v1, TAR-Bio-v2; for TAR, only the weaponization knowledge restriction version was evaluated, not its harmful request refusal version (Section 2.3 footnote).
   • Attacks are strictly confined to the two original papers' threat models: only fine-tuning attacks, with other weight-editing methods such as pruning not evaluated (Section 2.1, 2.2).
   • The TAR re-evaluation covers 6 fine-tuning configurations (4 from the original paper, 2 newly designed); the 28 adversaries claimed in the original paper were not re-evaluated one by one (Table 1, Appendix E.2.1).
   • Number of repetitions: 5 seeds per configuration for fine-tuning attacks, 3 seeds for prompt-template experiments, 5 runs for benchmark evaluations; the paper does not report agreement numbers between judge models (GPT-4o, GPT-4, etc.) and human annotation.
   • The benign-data fine-tuning experiments use 100 AOA and 100 Alpaca Salient examples; BenignBio is 100 GPT-4o-generated, manually verified questions (Section 4, Appendix E.3.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   Case study: two open-weight defenses fail under several minor evaluation changes; the authors provide an evaluation checklist.

   A case study on the evaluation methodology of open-weight LLM defenses: the authors re-test two defenses claimed to be durable, RepNoise and TAR, and argue that durability evaluation of such defenses is highly error-prone and that minor changes can overturn the original conclusions.

   • Problem: the weights of open-weight models can be directly fine-tuned by attackers, and defenses such as refusal training are easily removed; newly emerging durable defenses claim resistance to fine-tuning attacks, but the authors argue their evaluations do not cover worst-case threat models and easily give a false sense of security.
   • Method: adversarial re-testing under the original papers' own checkpoints, threat models and configurations, changing only one variable at a time—enabling data shuffling to introduce randomness, switching to the HuggingFace SFT Trainer, minor changes to fine-tuning hyperparameters, WMDP prompt templates and answer extraction methods—and additionally testing fine-tuning on benign data, side effects and utility.
   • Result 1 (randomness): TAR-Bio-v2 run with 5 seeds each under three original configurations, the highest recovered accuracy on WMDP-Bio all exceeded 60%, close to the undefended Llama-3-8B-Instruct (Figure 2); for RepNoise at LR 6×10⁻⁵ and 8×10⁻⁵, the highest harmfulness score was almost equal to the undefended model (Figure 1).
   • Result 2 (prompt template): for TAR-Bio-v1, merely adding a chat template to the WMDP prompt and switching to GPT-4o or manual answer extraction raised accuracy from the random level of about 25% to above 45% (Figure 4a).
   • Result 3 (side effects): TAR-Bio-v2 on the benign biology questions of BenignBio had GPT-2 perplexity 16.4±2.11 (undefended 3.8±0.1), GSM8K dropped to 4.8±0.4 (undefended 74.4±0.6), and Sorry-Bench harmfulness rate rose to 32.2±1.6 (undefended 22.2±0.8) (Table 2, 3).
   • Result 4 (mechanism): the authors state that the failure of TAR's original configuration stems from gradient explosion early in fine-tuning, and that extending warmup with cosine decay (New-Config 1) stabilizes training and bypasses the defense; fine-tuning on only 100 benign data points also restored the capabilities the two defenses had "forgotten", on which basis the authors argue the information was not truly removed.
   • Conclusion and implications: the authors state that current methods cannot yet durably defend open-weight models under worst-case threat models, and provide a 5-item evaluation checklist (report worst-case results across multiple seeds, use validated attack implementations, match claims to the threat model, cover side-effect benchmarks, make unlearning claims cautiously), arguing that safety claims should be constrained to rigorously validated threat models (Appendix A, Section 5).

   Collapse this answer

Related papers