When Safety Fails Collectively: Robust Alignment Against Harmful Fine-Tuning via Joint Weight Perturbation

1. 01What problem does this paper attempt to solve?

   The authors state that jointly applying the parts of a harmful perturbation amplifies safety degradation, and JWP trains accordingly during the alignment stage.

   Downstream harmful fine-tuning weakens the safety of aligned large models; the authors state that existing alignment-stage defenses do not address the joint effect of the parts of the harmful weight perturbation.

   • Setting: When a model is released as open-source weights, or customized with user data via fine-tuning-as-a-service, the data may contain harmful samples, sometimes unintentionally. Work cited by the authors states that fine-tuning on a small amount of such data can weaken safety while task performance may remain good, so safety at the end of alignment cannot guarantee safety after subsequent training.
   • Existing gap: Vaccine, RepNoise, and Booster use harmful samples and simulate harmful perturbations during alignment, but usually treat the perturbation as a whole. The authors state that on Llama-2-7B, the HS of each defense rises quickly during sustained harmful fine-tuning; Booster delays the onset, but safety still collapses after fewer updates (Figure 1). The authors consider this to reflect insufficient handling of interactions within the perturbation.
   • Observation: For disjoint parts of the same harmful update, applying each alone has limited effect, and applying them jointly is more harmful than the sum of their individual effects; applying only a small part largely preserves safety, and as the proportion rises, HS increases faster (Section 4). The authors call this collective failure.
   • Method: Propose an alignment-stage defense, JWP. It estimates the harmful weight perturbation from the current model, masks different parts of it, and trains safe responses on the resulting states, to reduce sensitivity to perturbation accumulation during subsequent fine-tuning.
   • Threat model:
     • Goal: Remain safe after harmful fine-tuning while retaining the ability to learn downstream tasks.
     • Victim system: A model that is first aligned and then fine-tuned downstream. By default, aligned Llama-2-7B Base, then fine-tuned on a mixture of benign task data and BeaverTails harmful samples, with a nominal harmful proportion of 10%.
     • Capability: The downstream set may contain harmful prompt–response pairs. It is not assumed that they can be reliably identified and removed before fine-tuning. The channel is fine-tuning-as-a-service or local fine-tuning of open-source weights.
     • Defender: At alignment time, Dalign (prompts and safe targets) and Dharm (harmful prompts and unsafe responses) are available. Intervention is only at the alignment stage, and the downstream pipeline keeps its standard form, unlike the downstream reweighting of Antibody.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into harmful fine-tuning, alignment-stage defenses, defenses at other stages, and safety structure in the weights.

   The paper organizes related work by harmful fine-tuning, alignment-stage defenses, other-stage defenses, and safety structure in weights; the experimental baselines are SFT, Vaccine, RepNoise, and Booster.

   Harmful fine-tuning

   • Qi et al. (2024), Bowen et al. (2025), Guan et al. (2025): Fine-tuning on a small number of harmful samples can remove safety alignment, and benign fine-tuning data can also reduce safety. Unlike the prompt jailbreaks of Zou et al. (2023), such attacks directly change the weights.
   • Qi et al. (2025b): A defense validated under one configuration may fail after the number of fine-tuning steps or the learning rate changes. The authors therefore emphasize evaluating along the full harmful fine-tuning trajectory rather than only at a single endpoint.

   Alignment-stage defenses

   • Vaccine (Huang et al., 2024b) and T-Vaccine (Liu et al., 2025): The former perturbs hidden embeddings to mitigate harmful embedding drift; the latter restricts such perturbations to safety-critical layers.
   • RepNoise (Rosati et al., 2024) and Booster (Huang et al., 2025c): The former pushes harmful data representations toward random noise; the latter penalizes the drop in harmful loss caused by one simulated harmful gradient step.
   • TAR (Tamirisa et al., 2025): Uses meta-learning to train protection against simulated multi-step tampering trajectories. Appendix C compares state construction: TAR samples along simulated trajectories, while JWP combines partial components of the same fitted update.

   The paper states that these defenses act on hidden representations or on aggregated simulated perturbations and trajectories, and do not construct combinations of different parts of the same learned harmful weight update.

   Other-stage defenses

   • Fine-tuning stage: Mix safety data into user data (Huang et al., 2024a; Wang et al., 2024; Eiras et al., 2025), constrain updates to the initial response tokens (Qi et al., 2025a; Peng et al., 2025), or make updates deviate from safety-related subspaces (Li et al., 2025).
   • After fine-tuning: Subspace projection (Hsu et al., 2024), pruning harmful weights (Huang et al., 2025b), correcting perturbations (Wang et al., 2026).
   • Antibody (Nguyen et al., 2026): The alignment objective seeks flatness and reweights user samples during fine-tuning. The paper states that these methods need to control the downstream pipeline and incur overhead on every request; JWP runs only at the alignment stage and is complementary to them.

   Safety structure in weights

   • Wei et al. (2024): Safety-critical regions are sparse; removing a small fraction of parameters breaks safety while largely preserving utility.
   • AHD (Huang et al., 2025a): Safety mainly depends on a few attention heads; during training, attention heads are randomly masked with a fixed probability, spreading safe behavior across more heads, targeting jailbreak prompts.
   • Safety basin (Peng et al., 2024): Safety is preserved while random weight perturbations stay within the basin, and declines after leaving it. The paper also mentions work characterizing safety along random directions (Zou et al., 2024).

   The paper states that the above work characterizes safety through the model's own components or random directions; this paper analyzes the components of one fitted harmful update and states that the safety degradation from joint application exceeds their individual effects. AHD and JWP both use random masking, but the masked objects and defense goals differ.

   Baselines and benchmarks: Experiments compare SFT, Vaccine, RepNoise, and Booster. Alignment data is 5,000 BeaverTails entries; downstream tasks are SST-2, AGNEWS, GSM8K, and ARC-Challenge, by default mixed with 10% harmful samples and benign task samples. The authors position JWP as: decomposing the fitted harmful update and optimizing safety on states where its parts are applied, to target the joint effect among perturbations.

   Collapse this answer

3. 03How does the paper address this problem?

   JWP estimates the harmful weight perturbation and randomly masks it, training safe responses on partially applied states.

   JWP estimates the harmful weight perturbation during safety alignment, randomly masks part of it, and makes the model still give safe responses on these temporary states.

   Setup and collective failure

   • Pipeline and data: First safety alignment, then downstream fine-tuning on user data. Dalign is prompts with safe target responses; Dharm is harmful prompts with unsafe responses. θ is the weights, and the change ∆h caused by harmful fine-tuning is called the harmful weight perturbation.
   • Analysis object: Align Llama-2-7B by different methods, then obtain ∆h through the same sustained downstream harmful fine-tuning. On a coordinate subset S, ∆hS keeps only the components in S. In the analysis, θ and ∆h are fixed and only S is changed (Section 4).
   • Disjoint subsets: Sample two disjoint sets Aq and Bq, each comprising a proportion q∈[0, 0.5] of the entries of ∆h. Joint harm amplification is I(q)=HS(A_q∪B_q)−HS(A_q)−HS(B_q)+HS(∅), i.e., the HS increase from joint application minus the individual increases of the two subsets. HS(∅) is the aligned model without perturbation. The authors treat it as an empirical indicator on the HS scale.
   • Proportional retention: Each scalar entry of ∆h is independently retained with probability p, Mi following Bernoulli(p). p=0 is the aligned weights, p=1 is the full perturbation. The authors state that for Booster, Vaccine, and RepNoise, I(q) is positive on larger subsets, and HS rises faster with p within the shaded region of Figure 2; the main text does not give point-by-point readings.

   Masked alignment objective

   • Expected loss: Given a fixed ∆h, p is sampled from U(0,1), then Bernoulli masking is applied at that probability, and Dalign's cross-entropy is computed on the partially applied weights. L_JWP(θ∣Δh)=E_p∼U(0,1)E_{M∼Bernoulli(p)}[L_align(θ+M⊙Δh)].
   • Purpose stated by the authors: Sampling p changes the expected applied proportion, and sampling M changes the composition, thereby covering different combinations. These states need not be checkpoints on a simulated fine-tuning trajectory.
   • Coverage property (Appendix D): If the perturbation is divided into d components, the number retained is uniform over 0 to d, and subsets of the same size are equally likely. The probability that any specified r components are retained simultaneously is 1/(r+1). With a fixed retention probability, the subset size follows a binomial distribution. Each alignment step uses one sampled mask to estimate this expectation.

   Blockwise estimation and updates

   Training proceeds in blocks: K steps to estimate the perturbation, then K steps of safety alignment (Algorithm 1). Default K=100.

   1. At the start of a block, freeze θt; the temporary parameters start from an effective perturbation δ=0 and optimize Lharm(θt+δ) for K steps. Lharm is the cross-entropy on Dharm. The fitting learning rate is 10^−4, using a separate LoRA with the same rank and scaling as the alignment adapter; alignment parameters are frozen during fitting.
   2. Then freeze ∆h and perform K alignment steps. Each step newly samples pi∼U(0,1) and a mask, and minimizes the partially applied Lalign on a mini-batch. Gradients on the perturbed states update only the alignment parameters.
   3. After the same ∆h is reused for K steps, the next round's perturbation is re-estimated from the updated weights.

   In BeaverTails, the unsafe responses of harmful samples are used to estimate the perturbation, and the paired safe responses are used for alignment; for benign samples, the alignment target is the original response (Appendix A.1).

   Training choices shared with the experiments

   Both safety alignment and downstream fine-tuning use LoRA with rank 32, α=4, dropout 0, applied to query, key, and value. Alignment runs for 4 epochs with a learning rate of 5×10^−4; downstream runs for 30 epochs with a learning rate of 10^−5, with LoRA newly initialized each time. AdamW, weight decay 0.1, effective batch size 10. These budgets are the same for all methods. The paper states that extending alignment would incur a large alignment tax, so it uses a utility-constrained alignment budget.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On Llama-2-7B, JWP's average HS across the four tasks is 11.50%, lower than each baseline (Table 2).

   Experimental Setup

   • Models: Evaluated from base checkpoints: Llama-2-7B, Llama-3.1-8B, Qwen2.5-7B. Unless otherwise noted, the default model is Llama-2-7B and the default downstream task is SST-2.
   • Data: 5,000 BeaverTails examples for alignment. Downstream tasks are SST-2, AGNEWS, GSM8K, ARC-Challenge. The default fine-tuning set is 1,000 examples with a harmful ratio of 10%, i.e., 900 benign task samples plus 100 harmful samples. Within the same task, each method sees the same, ordered downstream samples.
   • Baselines: SFT, Vaccine, RepNoise, Booster, all used at the alignment stage, followed by the same downstream fine-tuning. The comparison with TAR in Appendix C is a state construction; the experimental tables do not report TAR.
   • Metrics: HS is the percentage of responses judged harmful. FA is downstream accuracy after fine-tuning: classification accuracy for SST-2 and AGNEWS, whether the final numeric value matches the reference answer for GSM8K, and acc_norm for ARC-Challenge (the option with the highest length-normalized log-likelihood). UP is task accuracy after alignment and before fine-tuning, default 5-shot, reporting accuracy itself.
   • Evaluation: Beaver-Dam-7B, threshold 0.5. Main results use prompts from the first 1,000 unsafe samples of the BeaverTails 30k test set, in fixed order. Greedy decoding, at most 512 new tokens. The paper does not report the number of repeats, nor does it report the agreement of this classifier with human judgment.
   • Budget: 4 epochs of alignment, 30 epochs of downstream harmful fine-tuning. LoRA rank 32, α=4, projections query, key, value. AdamW, weight decay 0.1, effective batch size 10. Alignment learning rate 5×10^−4, downstream 10^−5. JWP default K=100.

   Main Results

   HS (%) after fine-tuning on the four downstream tasks, for the default model Llama-2-7B at the default harmful ratio of 10%. Per Table 2. The table caption does not restate the model and ratio; the values correspond to the default setting in Section 6.1.

   Swipe horizontally to read the table

   Method	SST-2	AGNEWS	GSM8K	ARC-Challenge	Average
   SFT	68.4	67.4	48.4	65.0	62.30
   Vaccine	60.1	65.4	14.2	59.6	49.83
   RepNoise	62.6	63.7	54.8	60.3	60.35
   Booster	62.4	63.6	22.2	62.9	52.78
   JWP	14.7	16.5	3.6	11.2	11.50

   • Authors' interpretation: The authors state that JWP's HS is lower than each baseline on all four tasks. The abstract states a 4.3–5.4× reduction in four-task average HS relative to existing defenses including Booster. JWP's average UP is 59.39, the highest in Table 2; average FA is 64.50, lower than Booster's 64.93.
   • Utility is not highest on every item: On AGNEWS, JWP's FA is 88.90, lower than SFT's 90.00 and Booster's 89.90; on GSM8K, Booster's FA is 20.70, higher than JWP's 20.20. UP is highest in the table only on SST-2 (92.20) and ARC-Challenge (54.10); AGNEWS's UP is 78.84, lower than Booster's 79.87. The authors state that downstream accuracy remains competitive and is highest on SST-2.
   • Differences among baselines: Vaccine's HS on GSM8K is 14.2, lower than its 60.1, 65.4, 59.6 on the other three tasks, and still higher than JWP's 3.6. The paper does not explain this task difference.

   Harmful Ratio and Model Families

   • Ratio (Table 1): The table caption does not state the model and task; it follows the default setting. clean means no harmful samples, r=0. JWP's HS at r=0, 0.05, 0.10, 0.15, 0.20 is 1.40, 4.50, 14.70, 28.00, 45.50, averaging 18.82; the lowest baseline average is Booster's 47.24. At r=0.20, the paper states 200 harmful samples are trained for 30 epochs, corresponding to 6,000 harmful-sample exposures; in that column the lowest baseline HS is RepNoise's 71.50, and JWP's is 45.50. The authors state that all defenses are weakened, and JWP remains lower than the strongest baseline at that ratio. JWP's average FA is 94.61, the highest in the table.
   • Model families (Table 3): On the ARC-Challenge mixture, JWP's HS for Llama-2-7B, Llama-3.1-8B, Qwen2.5-7B is 11.20, 11.00, 13.50, averaging 11.90. Baseline average HS ranges from Booster's 49.30 to RepNoise's 64.93. On Qwen2.5-7B, Booster is 30.00, a smaller gap from JWP's 13.50 than the other baselines on that model. JWP's average FA is 56.68, the highest in the table; average UP is 57.57, lower than Booster's 57.65. On Qwen2.5-7B, Booster's UP is 61.77, higher than JWP's 59.56.
   • Figure 1: The authors state that on Llama-2-7B, HS rises with harmful fine-tuning for each evaluated defense; Booster delays the onset relative to Vaccine and RepNoise, and safety still collapses after fewer updates, and they state that JWP resists more strongly. The horizontal axis is labeled up to 3k steps. The main text does not give endpoint readings of the curves.

   Masking, Block Length, and Joint Amplification

   • Retention probability (Table 4, K=100): With fixed p, HS at p=0, 0.2, 0.4, 0.6, 0.8, 1 is 69.20, 66.00, 61.00, 56.80, 51.00, 31.40; with p∼U(0,1) it is 14.70. The authors state that fixed p concentrates the combination on similar retention ratios, and only resampling p covers different scales. The table caption does not state the model and task; 14.70 is the same as JWP's HS at r=0.10 in Table 1.
   • Block length (Table 5): HS at K=10, 20, 50, 100, 250, 500 is 68.60, 65.70, 54.90, 14.70, 10.30, 28.50; UP ranges from 89.91 to 92.20, with UP 92.20 at K=100. The lowest HS is at K=250. The authors state that short blocks may be insufficient to simulate harmful fine-tuning, and that protection weakens at K=500, consistent with over-long reuse making the perturbation less representative of the current model. The table caption does not state the model and task.
   • Figure 2: The authors state that JWP's I(q) on larger disjoint subsets is smaller than existing defenses', and HS is lower and rises more slowly under the full perturbation, and they suggest this may explain its resistance to sustained harmful fine-tuning. The left panel's vertical axis is Joint Harm Amplification and horizontal axis is Harmful Subset Size (%); the right panel's vertical axis is Harmful Score and horizontal axis is Harm Probability (%). The main text gives no point-by-point readings.

   Other Ablations and Analyses

   • Efficiency (Table 6, Llama-3.1-8B, single A800-80GB): Alignment time is SFT 0.25 h, Vaccine 0.49 h, JWP 0.67 h, Booster 0.84 h, RepNoise 1.65 h; alignment peak memory is 44.10, 44.97, 46.67, 47.09, 52.28 GiB respectively. Downstream fine-tuning is 0.27 or 0.28 h for each method, with peak memory 41.94 GiB for all.
   • r=0: With only benign downstream data, HS is still nonzero: JWP 1.40, Vaccine 3.00, Booster 6.00, SFT 7.30, RepNoise 7.80 (Table 1). The paper does not separately explain this column.
   • HS rebound: At r=0.20, JWP is 45.50 (Table 1); at K=500 it is 28.50, higher than 10.30 at K=250 (Table 5); with fixed p=1 it is 31.40, higher than 14.70 with random p (Table 4).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that the choice of K, jailbreak, and cross-stage combinations remain to be studied, and joint amplification is currently measured only by HS.

   Limitations and future directions noted by the authors

   • Choice of K: Needs further exploration on more models and training settings (Section 7, Limitations).
   • Threat scope: Evaluation focuses on harmful fine-tuning; related threats such as jailbreak are left for exploration (Section 7, Limitations).
   • Standalone use: JWP is evaluated independently; combining it with other-stage defenses may provide complementary protection (Section 7, Limitations).
   • Measurement: Joint amplification is currently measured by HS, which captures behavioral consequences rather than intrinsic model properties. Developing model-level interaction measures and explaining their causes is stated as a future direction (Section 7, Quantifying joint amplification).

   Experimental coverage

   • Models and comparisons: The base models used in experiments are Llama-2-7B, Llama-3.1-8B, and Qwen2.5-7B; alignment-stage comparisons are SFT, Vaccine, RepNoise, and Booster (Section 6.1). Appendix C discusses how TAR constructs states; the experimental tables do not report TAR scores.
   • Tasks and mixtures: Downstream tasks are SST-2, AGNEWS, GSM8K, and ARC-Challenge (Table 2). Comparisons at harmful proportions of 0%, 5%, 10%, 15%, and 20% are in Table 1; the three-model comparison uses the ARC-Challenge mixture (Table 3).
   • Judgment: HS is judged by Beaver-Dam-7B with a threshold of 0.5, using the first 1,000 unsafe samples of the BeaverTails 30k test set as prompts; greedy decoding, at most 512 new tokens (Appendix A.3).
   • Training: 4 epochs of alignment, 30 epochs downstream; default downstream set of 1,000 entries with a harmful proportion of 10%; JWP defaults to K=100, and several values of p and K are reported (Section 6.1, Table 4, Table 5). Efficiency measurements are on Llama-3.1-8B with a single A800-80GB (Table 6).
   • Not reported: The paper does not report the number of repetitions, nor the agreement between Beaver-Dam-7B and human judgment.

   Collapse this answer

6. 06Summarize the main content of the paper

   On Llama-2-7B, JWP aligns on the masked harmful-perturbation state, with a four-task average HS of 11.50%.

   JWP is an alignment-stage defense intended to reduce the erosion of safety alignment by subsequent harmful fine-tuning.

   • Problem: In open-weight or fine-tuning-as-a-service settings, harmful samples in downstream data rewrite the weights. The authors observe that disjoint parts of a single harmful weight perturbation have limited effect on their own, but when applied jointly the HS increase can exceed the sum of the two, and they call this collective failure.
   • Method: Every K steps, starting from the current frozen weights, fit a perturbation ∆h on Dharm; then over K steps, each step samples a retention probability p∼U(0,1) and a Bernoulli mask, and performs a cross-entropy update with Dalign's safety objective on the partially applied weights. Default K=100.
   • Main results: On Llama-2-7B with a harmful ratio of 10%, the four-task average HS is 11.50, lower than SFT's 62.30, Vaccine's 49.83, RepNoise's 60.35, and Booster's 52.78 (Table 2). The authors state a 4.3–5.4× average HS reduction relative to these defenses. Average UP is 59.39, and average FA is 64.50.
   • Other settings: On the default SST-2, JWP's average HS across harmful ratios 0%–20% is 18.82, and Booster's is 47.24 (Table 1). Across three models on the ARC-Challenge mixture, the average HS is 11.90 (Table 3). With p∼U(0,1), HS is 14.70, lower than 31.40 with fixed p=1 (Table 4). HS is 10.30 at K=250 and 28.50 at K=500 (Table 5).
   • Authors' conclusions: The authors argue that joint amplification should be an objective of safety alignment; on the tested attack trajectories, data mixtures, downstream tasks, and model families, JWP resists harmful fine-tuning more strongly and largely preserves utility. The authors also note that K still needs to be explored in more settings, that jailbreak was not evaluated, and that JWP was evaluated on its own.

   Collapse this answer

Related papers