Tamper-Proof Guardrails for Open-Weight LLMs

1. 01What problem does this paper attempt to solve?

   Refusal and unlearning on open weights are easily removed by fine-tuning; TAR aims to make protection difficult to remove even after hundreds of fine-tuning steps.

   Refusal and forgetting in open-weight LLMs can be removed by weight-modifying fine-tuning, so defenses are needed that are hard to remove under tampering while preserving general capabilities.

   • Scenario: The authors say the most capable open-weight LLMs of the past year have become comparable to closed-source frontier models; downloadable use lowers end-user costs and also facilitates safety research. As capabilities rise, the authors worry about models being repurposed for malicious use.
   • Existing shortcomings: Existing open models often borrow API-oriented refusal and preference training. The authors say these defenses can resist input jailbreaks, but after weight modification they often fail within a few fine-tuning steps. The authors consider this problem difficult, with no prior method preserving defenses against tampering attacks, and it also exposes developers to liability risk under a duty of reasonable care.
   • Threat model:
     • Defender objective: Release weights θG with defense G such that both safety_metric and capabilities_metric are high, and safety_metric remains high after adversary actions.
     • Adversary: Compute-limited, with unrestricted access to θG, can directly modify weights (the authors call this tampering attacks), and must minimize the safety metric within reasonable compute; the paper's example is fine-tuning for 500 steps.
     • Scope: Only fine-tuning adversaries are considered, not input-space jailbreaking. It is assumed the adversary will not spend compute close to pretraining, otherwise they could train an undefended model themselves.
     • Victim system: An open-weight LLM with defenses. This paper addresses weaponization knowledge restriction and harmful request refusal.
   • Proposed in this paper: Method TAR (Tampering Attack Resistance). First install an initial defense that need not resist tampering, then perform adversarial training against a fine-tuning adversary at training time. The authors say this is the first class of LLM defenses that can resist multiple tampering attacks.

   Collapse this answer

2. 02What related research exists?

   Related work covers jailbreaks, unlearning, and a few fine-tuning-resistant methods; the authors state that no tamper-resistant method existed for autoregressive LLMs before.

   The paper organizes related work by adversarial attacks, protection and unlearning, and tamper-resistance, and states that no tamper-resistant method existed for autoregressive LLMs before.

   Input attacks and alignment

   • RLHF and DPO: Christiano et al. (2017), Ouyang et al. (2022) use human preferences for reinforcement learning; Rafailov et al. (2023) DPO directly optimizes preferences. The paper lists them as training methods for existing protection.
   • Jailbreaks and fine-tuning removal: Wei et al. (2023), Zou et al. (2023) et al. demonstrate jailbreaks; Qi et al. (2023), Yang et al. (2023), Zhan et al. (2023) remove protection by fine-tuning on a small amount of uncensored data. The paper states that existing alignment is insufficient under these attacks.

   Protection and machine unlearning

   • System-level and model-level: System-level methods rewrite or filter inputs and outputs; model-level includes adversarial training. Unlearning routes include influence functions, maximizing loss on the forget set, modifying representations, and circuit breaking.
   • Paper's critique: The paper states that jailbreak defenses are not sufficiently robust to adaptive adversaries; existing unlearning is not robust to adversaries with weight access (Lynch et al., 2024).

   Tamper-resistance and concurrent work

   • Non-autoregressive and vision: There is prior work on fine-tuning-resistant unlearning in image classification. Henderson et al. (2023) use meta-learning to prevent BERT from learning harmful tasks. Concurrent Deng et al. (2024) extend similar ideas to small-scale vision classifiers and diffusion models.
   • Discussion on LLMs: Liu et al. (2024) discuss the potential of robust unlearning for open-source safety; Lynch et al. (2024) propose evaluation metrics. Concurrent refusal work includes Huang et al. (2024) perturbation loss, and Rosati et al. (2024) immunization and RepNoise. The paper states these works evaluate few fine-tuning adversaries or have limited resistance; and states the latter two lack resistance under this paper's tampering attacks.

   Baselines and benchmarks

   • Baseline methods: On the weaponized knowledge side, RMU, LLMU, Max Entropy, Min Posterior, and MLAC-AR, SOPHON-AR rewritten for autoregressive LLMs. On the refusal side, Refusal Trained, Representation Rerouting (RR), R2D2, RepNoise.
   • Benchmarks: WMDP, MMLU, HarmBench, MT-Bench.

   The authors state that to their knowledge, no method exists for autoregressive LLMs that resists tampering attacks; this paper uses adversarial training and meta-learning to make protection difficult to remove after weight modification.

   Collapse this answer

3. 03How does the paper address this problem?

   TAR first installs an initial defense, then performs MAML-like adversarial training against a fine-tuning adversary, with the outer-loop loss different from the adversary's loss.

   TAR has two phases: first install an initial guard, then do adversarial training against a set of fine-tuning adversaries, so that the safety metric of the attacked agent remains high while preserving capability.

   Problem setup and metrics

   • Definition of tamper resistance: attack maps θG to θG′. The authors say that if the safety_metric remains high after attack on a set of strong test attacks A_test, then the guard is tamper-resistant. Changing the weights to noise can also be tamper-resistant, but the model is no longer usable, so the capabilities_metric must also be considered.
   • Weaponized knowledge restriction: prevent writing weaponized knowledge while preserving benign domains. safety_metric is the error rate on the forget set, capabilities_metric is the accuracy on the retain set. The forget set is WMDP's biosecurity, chemical security, or cybersecurity; the retain set is the complement of that topic in MMLU. WMDP has 3,668 multiple-choice questions and measures expert-level proxy knowledge of dangerous domains, not dangerous knowledge itself.
   • Refusal of harmful requests: safety_metric is the complement of the average ASR over several jailbreaks; capability is measured with MT-Bench. After attack, ASR is measured with HarmBench's fixed test set.

   Two-phase training

   • Phase one: install an existing guard into the base model θ. Knowledge restriction can use circuit breaking or constrained gradient ascent; refusal can use RLHF or DPO. The authors say the initial guard need not be tamper-resistant, but this step is important for the pre-attack safety metric.
   • Phase two: starting from the guarded parameters, do adversarial training on a training attack set A_train so that the safety metric of the attacked agent increases. The attack may not be differentiable, so it is written as adding a perturbation to the original parameters, backpropagation uses a straight-through estimator, and a first-order MAML approximation is used.
   • Difference from conventional meta-learning: conventional meta-learning wants an initialization that is near-optimal on multiple test distributions; here we want to be far from optimal on multiple adversarial test distributions. The outer-loop tamper-resistance loss differs from the adversary loss.

   Losses

   • Outer-loop objective: the defender solves min_θ λ_TR·E[L_TR(attack(θ); D_TR)] + λ_retain·L_retain(θ; D_retain). Minimizing L_TR is treated as a proxy for maximizing safety_metric.
   • Choice of L_TR: for weaponized knowledge, use negative entropy rather than negating the adversary's cross-entropy. The authors say the latter diverges and only raises the adversary loss in the first few inner-loop steps. Figure 3 compares the two over 750 steps of TAR: with negative cross-entropy the adversary recovers quickly; with negative entropy the entire inner-loop trajectory is eventually blocked. For refusal, L_TR uses DPO: the adversary does SFT on rejected completions, and the outer loop still prefers rejected completions.
   • Retain term: pull the retain residual stream toward the initial guarded model, i.e., L_retain = E_x[LLM(θ, x) + ||h_θ(x) − h_θG0(x)||₂²]. The authors say this helps maintain capabilities_metric.

   Initial guard, sampling, and settings

   • Random Mapping: the initial guard for weaponization experiments. Push the residual stream at each layer on the forget distribution toward a fixed Gaussian vector hashed from the input tokens, and do cross-entropy on D_retain. Appendix C.2 gives the objective.
   • Sampling: reuse intermediate parameters along a fine-tuning trajectory of length K to get shorter, non-independent trajectories, then sample every 4 adversary steps to compute L_TR. All inner-loop steps share the same held-out x_TR batch.
   • Settings in this paper: weaponization uses Llama-3-8B-Instruct, outer loop N=750, ScheduleFree AdamW, learning rate 2×10−5; λ_TR=4.0 for biological and cybersecurity, 3.0 for chemical security, λ_retain all 1.0. Sample 6 full-parameter SFT adversaries of 64 steps per domain, with learning rate 2×10−5 or 4×10−5. Refusal outer loop N=100, learning rate 6×10−5, λ_TR=0.1; adversaries do 64 steps of SFT on the rejected completions of Anthropic-HH-RLHF. The forget data is split 80–20, with the held-out part used for L_TR.

   Collapse this answer

4. 04What experiments did the paper conduct?

   In Table 1, TAR's post-attack WMDP forget accuracy is lower than the baselines in the same table, and its refusal ASR is also lower than in Table 2.

   On Llama-3-8B-Instruct, TAR's post-attack WMDP forget accuracy is lower than the Table 1 baseline; under refusal settings, post-attack ASR is also lower than Table 2, while capability scores decrease.

   Experimental Setup

   • Models: TAR and Table 1 baselines use Llama-3-8B-Instruct (table header says Llama-3-8B). Refusal additionally reports R2D2 on Zephyr-7B and RepNoise on Llama-2-7B. Pile synthetic labeling uses openchat-3.5.
   • Data and Metrics: WMDP has 3,668 questions across three domains; capability is the complement of corresponding topics in MMLU (57 tasks). Refusal uses HarmBench fixed set for ASR and MT-Bench for dialogue. The paper does not give the number of HarmBench questions or the MT-Bench judge model name.
   • Baselines: RMU, LLMU, Max Entropy, Min Posterior, No Defense, and Random (25.0%). Refusal baselines are Refusal Trained, RR, R2D2, RepNoise. MLAC-AR and SOPHON-AR are in Appendix G.3.
   • Attacks: Weaponization defaults to 500-step SFT. Biosecurity has 26 adversaries, chemistry and cyber each 24 (Section 5.3, Appendix F.1); Table 1 note says averaged over 26 attacks. Each adversary has 3 seeds. Refusal does 5 attacks on Toxic-DPO v0.2 (541 examples), each 10 epochs. The paper does not report repetition counts for refusal.
   • Training adversaries: Separate from test adversaries, all 64-step SFT. Appendix C.1 states that the initial retain data contained forget content; after cleaning and retraining with the same hyperparameters, the main tables show updated results.
   • Implementation: 8 80GB A100s, FSDP with ZeRO Stage 3. After TAR for refusal, an additional 100-step SFT on Magpie-Align to improve benign capabilities.

   Main Results

   Table 1 post-attack average Forget accuracy (%). Biosecurity corresponds to 26 attacks; chemistry and cyber adversary counts differ from the table note, see above.

   Swipe horizontally to read the table

   Method	Bio post-attack	Chem post-attack	Cyber post-attack	Bio pre-attack Retain
   TAR	35.2	27.1	28.6	54.7
   RMU	64.9	46.0	42.2	65.8
   LLMU	61.3	44.3	41.3	65.5
   Max Entropy	65.8	45.7	41.7	65.0
   Min Posterior	66.0	47.5	41.9	65.6
   No Defense	70.5	47.8	46.4	67.3
   Random	25.0	25.0	25.0	25.0

   • Author interpretation: The table note states that only TAR maintains low Forget after attack, while Retain is high and pre-attack Forget is low. Pre-attack TAR Forget is 28.1% for bio, 28.4% for chem, and 23.6% for cyber. In the table, TAR's Retain is lower than other non-random methods.
   • Trade-off: The authors state there is a trade-off between capability and resistance, and state that Retain decreases by an average of 10.6%. TAR Retain for chem and cyber is 56.5% and 60.7%, while No Defense is 68.2% for both.
   • Figure 2: The authors state that TAR has higher tamper-resistance at similar MMLU. Y-axis is Post-Attack Weaponization Knowledge Error, X-axis is Capabilities (MMLU). Pure text cannot read point coordinates.

   Harmful Request Refusal

   Table 2: Average HarmBench ASR after 5 fine-tunings, and pre-attack MT-Bench. Backbones differ, so cannot be combined into one row.

   Method (backbone)	MT-Bench↑	Post-attack ASR↓
   Refusal Trained (Llama-3-8B-Instruct)	8.1	72.5
   RR (Llama-3-8B-Instruct)	8.0	84.8
   R2D2 (Zephyr-7B)	6.0	78.3
   RepNoise (Llama-2-7B)	6.2	74.5
   TAR (Llama-3-8B-Instruct)	6.3	63.9

   • Inconsistency with main text: The main text says TAR rises to 61.7 and other methods rise above 77. Table 2's TAR is 63.9%; Refusal Trained and RepNoise are 72.5% and 74.5%, not all above 77.
   • Pre-attack ASR: Table 11 shows TAR 31.4%, RR 1.4%, Refusal Trained 14.7%, R2D2 25.0%, RepNoise 18.8%. The authors state that pre-attack ASR is not a priority target.
   • Capability: The main text says MT-Bench decreases by 1.7; in the table, Refusal Trained is 8.1 and TAR is 6.3. The authors state it is still higher than some capable open-source models; the paper does not give comparison scores.

   Red Teaming, Long Attacks, and Benign Fine-tuning

   • Figure 4: Per-adversary post-attack accuracy, random level marked as 25% (RC), No Defense marked as ND (bio 71, chem 48, cyber 46). The authors state that baselines have defenses removed under most attacks, and TAR blocks more adversaries; also state instability with PEFT and some out-of-distribution learning rates. In the TAR row, readings above 50 are visible in the text, but columns and Adv numbers do not match, so not bound per cell.
   • 1,000 steps: The authors state that in Figure 6, biosecurity SFT on TAR (learning rate 2×10−5) has cross-entropy not below 7 within 1,000 steps, plateauing after about 200 steps; the same attack on LLMU enters recovery region in under 20 steps. Figure 5 uses five dataloader seeds: plateaus are consistent within 500 steps, then variance increases.
   • Benign economics fine-tuning (Table 3): TAR-Bio benign domain from 59.7% to 64.7%, WMDP Forget from 28.1% to 30.1%; TAR-Chem from 58.6% to 62.8%, Forget from 28.4% to 27.5%. The authors state that economics questions can improve by 5.0 percentage points; in the table, the bio domain improves by exactly 5.0 percentage points.

   Other Ablations and Analyses

   • Removing adversarial training (Table 5, biosecurity): post-attack Forget 61.6%, pre-attack 27.3%; full TAR is 35.2% and 28.1%.
   • Removing initial protection (Table 5): pre-attack Forget 47.3%, post-attack 35.5%; the authors state that initial protection reduces pre-attack Forget accuracy by 19.2 percentage points.
   • λ_TR (Table 6, λ_retain=1.0): at 1.0, post-attack Forget 39.9%, Retain 62.5%; at 4.0, 35.2% and 54.7%. The table note says resistance improves by nearly 10.0%, inconsistent with the difference between the two cells.
   • Adding R→F training adversaries (Table 4): all adversaries from 44.9% to 35.2%, R→F from 58.7% to 29.7%.
   • Inner loop length K (Figure 7, K = 8, 16, 32, 64): the authors state that larger K leads to higher and longer-lasting test loss plateaus.
   • Appendix comparisons (Table 12): SOPHON-AR Retain is 27.2%, 23.3%, 24.4%; the authors state that even after tuning, they cannot get a usable pre-attack model; MLAC-AR post-attack Forget is 50.6%, 33.6%, 35.1%.

   Collapse this answer

5. 05What are some points that can be further explored?

   Appendix A states that only supervised fine-tuning is covered, larger models have computational cost, and retain is also low after attack.

   Appendix A lists attack types, model scale, and post-attack retain as limitations; the experiments themselves stop at the listed models, SFT red teaming, and two tasks.

   Limitations and future directions noted by the authors

   • Attack types: Only supervised fine-tuning is considered; broader open-weight tampering techniques require more diverse red teaming in the future (Appendix A).
   • Scale: Scaling to larger models poses computational challenges and requires optimization to reduce overhead (Appendix A).
   • Post-attack retain: When TAR reduces post-attack forget, post-attack retain is also low. The authors observed in preliminary experiments that many baselines still have high post-attack retain, and state that post-attack retain is the attacker's responsibility and is acceptable for the defender (Appendix A).
   • Benign use: If benign fine-tuning data is mixed into the forget set, performance after fine-tuning will be poor, potentially making the method harder to use in practice. The authors list maintaining high post-attack retain as a future direction (Appendix A).
   • Risk boundaries: The authors state that tamper-resistance cannot fully block malicious use: it raises the initial cost, but can ultimately still be bypassed; once a model is released it cannot be recalled. It should be a complement to improve the attack-defense balance (Appendix A).

   Experimental coverage

   • Models: TAR and the Table 1 baselines are trained and evaluated on Llama-3-8B-Instruct; Table 2 additionally reports R2D2 on Zephyr-7B and RepNoise on Llama-2-7B (Section 5).
   • Weaponized red teaming: WMDP three domains; 26 SFT adversaries for Biosecurity, 24 each for Chemistry and Cybersecurity; default 500 steps; 3 seeds per adversary (Section 5.3, Appendix F.1). The training inner loop is K=64 steps. Table 1 gives pre-attack Retain and post-attack Forget; the paper does not report MMLU Retain values after tampering attacks.
   • Refusal: HarmBench and MT-Bench; 5 attacks on Toxic-DPO v0.2, 10 epochs each (Table 2, Appendix F.2). The paper does not report the MT-Bench evaluation model, nor the number of repetitions for that setting.
   • Threat model description: Only SFT weight modification is considered; input-space jailbreaks are not included as adversaries, and TAR does not explicitly optimize for input-space robustness (Section 3.1, Section 5.3).
   • Comparisons: Compared against RMU, LLMU, Max Entropy, Min Posterior, RR, R2D2, RepNoise, and rewritten MLAC-AR and SOPHON-AR (Section 5, Appendix G).

   Collapse this answer

6. 06Summarize the main content of the paper

   TAR first installs a defense then adversarially fine-tunes; after WMDP attack, forget accuracy is lower than baselines, with some drop in capability scores.

   TAR is a tamper-resistant training method for open-weight LLMs, used for weaponized knowledge restriction and harmful request refusal.

   • Problem: Refusal and unlearning are often removed by a small amount of fine-tuning. After the defender releases θG, a computationally limited adversary can arbitrarily modify weights, with the goal of lowering the safety metric within, e.g., 500 fine-tuning steps. This paper does not consider input jailbreaks, nor does it assume the adversary spends compute close to pretraining.
   • Approach: First install initial protection that need not be tamper-resistant, then do outer-loop adversarial training against 64-step SFT adversaries. The outer-loop loss for weaponized knowledge is negative entropy; for refusal it uses DPO; the retain term pulls the residual stream back to the initial protected model. The initial protection for weaponized experiments is Random Mapping.
   • Weaponized results: On Llama-3-8B-Instruct, the average post-attack Forget in Table 1 is 35.2% for TAR biosecurity, 27.1% for chemistry, and 28.6% for cybersecurity; RMU in the same table is 64.9%, 46.0%, 42.2%, and no protection is 70.5%, 47.8%, 46.4%. Random level is 25.0%. TAR's pre-attack Retain is 54.7%, 56.5%, 60.7%, and the authors state it is on average 10.6% lower than no protection.
   • Refusal results: In Table 2, TAR's pre-attack MT-Bench is 6.3, and the average HarmBench ASR after 5 fine-tunings is 63.9%; Refusal Trained is 8.1 and 72.5%, RR is 8.0 and 84.8%. In Table 11, TAR's pre-attack ASR is 31.4%, higher than RR's 1.4%.
   • Authors' conclusion: The authors state that under their red-team evaluation, TAR is the first method that still preserves protection, and state that tamper-resistance for open weights can advance. They also state that this only raises the adversary's initial cost, cannot alone eliminate malicious use, and models cannot be recalled after release.

   Collapse this answer

Related papers