Skip to content

Addition by Subtraction: Toward Mitigating Fragile Safety Alignment in Large Language Models

Addition by Subtraction: Toward Mitigating Fragile Safety Alignment in Large Language Models

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.06553· Hamid Dashtbani, Alexandra DmitrienkoDepartment of Computer Science, University of Duisburg-EssenIndexed by this site · 15 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose the ABS-Align defense: progressively prune safety-critical neurons during fine-tuning and reconstruct safety behavior with DPO; the authors state that on four models the ASR of some attacks drops by up to about 90%.

Threat modelThe defender is the model vendor; the malicious user has white-box access, can observe the model's internal states (weights and activations), and uses pruning-type attacks or activation steering methods to jailbreak; adversarial fine-tuning attacks are outside the threat model.

Questions
Existing research shows that LLM safety alignment is highly sparse: pruning a small number of safety-critical neurons or ablating a single refusal direction can break alignment. Existing guardrails or general fine-tuning defenses do not directly address this sparsity, and the authors state they still have the same sparsity problem.
Methods
Proposes ABS-Align: first use TwinBreak to identify safety-critical neurons, progressively prune these neurons from highest to lowest importance during DPO fine-tuning, forcing the model to reconstruct safety behavior through other neurons, and restore all pruned neurons after fine-tuning ends.
Experiments and results
Evaluated on Gemma-2-2B-IT, Llama-3-8B/3.2-3B-Instruct, Qwen-3-4B: TwinBreak cumulative ASR drops from 0.71–0.81 to 0.11–0.33 (Table 1), SORRY-Bench ten-category black-box attack average ASR drops to 13.24%–33.02% (Table 3), utility benchmarks mostly change slightly, TruthfulQA rises.
Limitations and future work
The authors note: safety neuron detection methods are imperfect and may miss or falsely detect neurons; the absolute density of safety alignment is not directly quantified; DPO requires maintaining a frozen reference model, with computational and memory overhead (Section 5.1). Adversarial finetuning attacks are out of scope for the threat model.
Experimental setupGemma-2-2B-IT、Llama-3-8B-Instruct et al. · StrongREJECT、SORRY-Bench et al. · ASR、StrongREJECT score et al.
Evaluated models
Gemma-2-2B-ITLlama-3-8B-InstructLlama-3.2-3B-InstructQwen-3-4B
Benchmarks
StrongREJECTSORRY-BenchXSTestMMLUTruthfulQAARC-ChallengeWinoGrandeHellaSwagPKU-SafeRLHFAlpaca
Metrics
ASRStrongREJECT scoreOver-Refusal ratefew-shot accuracy
AI summaryThe paper proposes the Addition by Subtraction method, which progressively removes identified safety-critical neurons during fine-tuning and uses preference optimization to rebuild safety-aligned behavior in the rest of the model. The method aims to make safety alignment less dependent on sparse model components, targeting the weakness that pruning a small number of safety-critical neurons or ablating a single refusal direction can substantially weaken aligned behavior. Experiments on four models from three vendors show that the fine-tuned models have significantly improved robustness to pruning attacks, and are also more robust to sparsity-based interventions such as refusal-direction ablation and to various black-box jailbreak attacks, with the success rate of some attacks reduced by up to 90%. The fine-tuned models show improved truthfulness, and general capabilities are largely preserved.

The paper proposes the Addition by Subtraction method, which progressively removes identified safety-critical neurons during fine-tuning and uses preference optimization to rebuild safety-aligned behavior in the rest of the model. The method aims to make safety alignment less dependent on sparse model components, targeting the weakness that pruning a small number of safety-critical neurons or ablating a single refusal direction can substantially weaken aligned behavior. Experiments on four models from three vendors show that the fine-tuned models have significantly improved robustness to pruning attacks, and are also more robust to sparsity-based interventions such as refusal-direction ablation and to various black-box jailbreak attacks, with the success rate of some attacks reduced by up to 90%. The fine-tuned models show improved truthfulness, and general capabilities are largely preserved.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Addressing the sparsity problem of safety alignment: pruning a small number of safety neurons can break alignment, and the authors propose the ABS-Align defense.

    Core problem: LLM safety alignment is highly sparse; pruning a small number of safety-critical neurons or ablating a single refusal direction can break aligned behavior. The authors attempt to make safety alignment no longer depend on sparse components.

    • Problem setting: Existing research (papers cited as [22][23][24][25]) shows that safety alignment is concentrated in a few neurons, and pruning these neurons or ablating a single refusal direction can induce harmful outputs, with little effect on general capabilities. The authors consider this especially concerning for open-weight models, because attackers can directly access model parameters.
    • Shortcomings of existing methods: The authors state that existing defenses (external input/output guardrails, fine-tuning methods that improve general robustness) do not directly address the sparsity of safety alignment, and their experiments show these methods still have the same sparsity problem as the original model.
    • Core idea: During fine-tuning, progressively and temporarily prune the identified safety-critical neurons, using preference optimization to force the model to rebuild safety behavior through other neurons, making safety alignment "denser" (the paper's term, meaning less fragile and less dependent on sparse safety neurons).
    • Paper output: Proposes the defense method ABS-Align (Addition by Subtraction), and evaluates its robustness to white-box pruning attacks, refusal-direction ablation, and black-box jailbreaks on four models from three vendors.
    • Threat model (Section 3.2):
      • Attacker goal: Jailbreak the target model so that it generates harmful content it would otherwise refuse.
      • Attacker knowledge: white-box access, able to observe the model's internal states, including weights and activations.
      • Attacker capability: Can use pruning-type attacks or activation-steering methods.
      • Victim system and scope: decoder-only open-weight LLMs that have completed safety alignment; adversarial finetuning attacks are out of scope for the threat model.
  2. What related research exists?

    Related work falls into three categories: black-box/white-box jailbreak attacks, three types of defenses (input/output/fine-tuning), and research on the sparsity of safety alignment.

    Jailbreak attacks

    • Black-box attacks: early work relied on manually constructed adversarial prompts (role-playing, persuasion, obfuscating malicious intent); later work systematized this, e.g., PAIR [19] uses an attacker LLM to iteratively optimize jailbreak prompts, requiring only black-box access.
    • White-box attacks: attackers can access the architecture, weights, and internal states, e.g., GCG [14] uses gradients to optimize adversarial suffixes, and AutoDAN [15] uses a hierarchical genetic algorithm to iteratively improve jailbreak prompts.

    Defense methods

    • Input/output-time defenses: Alon et al. [29] use perplexity to detect adversarial prompts; Kumar et al. [28] propose erase-and-check; Llama Guard [26] scans inputs and outputs; SmoothLLM [27] randomly perturbs prompts and takes a majority vote; Aligner [54] corrects output text. The paper notes that such defenses do not improve the model's own robustness, and the model remains vulnerable once they are bypassed.
    • Fine-tuning-based defenses: W-DOOR [30] decomposes the DPO loss into the dual objectives of refusal training and harmful knowledge forgetting; Circuit Breaker (Zou et al. [41]) reroutes harmful internal representations; Alignment-Weighted DPO (Hu et al. [31]) uses CoT safety data and assigns different weights to reasoning traces and final answers. The paper notes that none of these works address the sparsity of safety alignment and they remain vulnerable to intervention-based and neuron-pruning attacks.

    Sparsity of safety alignment

    • Wei et al. [24] use Wanda and SNIP scores to locate safety-critical neurons; TwinBreak [23] compares intermediate activations of paired benign/malicious prompts and iteratively prunes; NeuroStrike [22] trains linear probes to identify safety neurons; Arditi et al. [25] find that refusal behavior is mediated by a single direction in activation space. These works show that pruning 1–5% of neurons can weaken safety alignment while barely affecting general capabilities.

    Baselines and benchmarks: the comparison baselines are Circuit Breaker [41], W-DOOR [30], and Deep Refusal [42]; attack evaluation uses TwinBreak, NeuroStrike, refusal-direction ablation, and SORRY-Bench black-box attacks, with metrics based on the StrongREJECT and SORRY-Bench judges.

    Positioning of this paper: the authors state that ABS-Align directly targets the sparsity of safety alignment, distributing safety behavior across more neurons through progressive pruning plus fine-tuning, rather than relying on external mechanisms or general robustness training.

  3. How does the paper address this problem?

    ABS-Align: TwinBreak identifies safety neurons, progressive pruning by importance during DPO fine-tuning, and all neurons restored after it ends.

    Overall idea: first identify safety-critical neurons, then during the DPO fine-tuning of ABS-Align (Addition by Subtraction) progressively and temporarily prune them from highest to lowest importance, forcing the model to reconstruct safety behavior through other neurons, and restore all pruned neurons after fine-tuning ends.

    Safety neuron identification and pruning (Section 3.4.1)

    • Detection method: TwinBreak [23] is used for its simplicity and efficiency; paired malicious/benign prompts (similar in content and structure) are input, and the average activation differences of the MLP block's gate and up projection layers are compared.
    • Scoring formula: T(ℓ) = 1/n∑i=1n abi(ℓ−1) Wgate(ℓ) − ami(ℓ−1) Wgate(ℓ), i.e., the average difference in gate projection activations at layer ℓ between benign and malicious prompts, used to rank neurons.
    • Iterative pruning: each round takes the top 1% neurons with the largest difference (denoted I^(ℓ)) and sets their corresponding weight columns to zero; repeated for at most 5 rounds, with neurons identified in previous rounds remaining pruned.

    Preference fine-tuning with progressive pruning (Section 3.4.2)

    • Procedure: freeze all identified safety neurons → first prune the most important r → after every tp training steps prune the next batch of r → by the late stage of fine-tuning all original safety neurons are pruned → restore all neurons after training ends (Algorithm 1).
    • Training objective: DPO loss LDPO = −𝔼log σ(β log (πθ(yc|x))/(πref(yc|x)) − β log (πθ(yr|x))/(πref(yr|x))), where yc is the preferred response such as a refusal and yr is the harmful response.
    • Why progressive: the authors give two reasons—pruning all safety neurons at once would greatly weaken safety alignment, making it harder to reconstruct safety behavior and requiring more data and training; it would also harm non-safety task capabilities (reported in prior pruning attack papers).
    • Why prune the most important first: the authors state that pruning unimportant neurons first would leave the model with strong existing safety alignment during fine-tuning, resulting in an insufficient learning signal.

    Key design parameters

    • Freezing scope: the entire model is fine-tuned, but the first and last transformer blocks remain frozen; the authors' rationale is that lower layers encode surface lexical information and the last layer directly generates output tokens, so freezing preserves existing capabilities.
    • Hyperparameters (Appendix 1): train for 2 epochs, 1,000 preference samples randomly drawn from PKU-SafeRLHF, batch size 2; for Llama models, an additional 850 Alpaca and 150 XSTest benign samples are added to preserve general capabilities; pruning step tp = 200, pruning rate 20% (every 200 steps, prune the safety neurons detected in one TwinBreak iteration); optimizer AdamW, linear cosine scheduler; learning rate 1e-5 to 2e-4 per model, DPO beta 0.1–0.3.
    • One round suffices: the authors observe that applying this procedure once to the initially identified safety neurons is enough to make sparsity attacks ineffective, without needing to re-identify and prune new safety neurons after fine-tuning.
  4. What experiments did the paper conduct?

    White-box pruning, refusal-direction ablation, black-box jailbreak, over-refusal, and utility are tested on four models, and compared with three defense baselines.

    Experimental setup

    • Models tested: Gemma-2-2B-IT (Google), Llama-3-8B-Instruct and Llama-3.2-3B-Instruct (Meta), Qwen-3-4B (Alibaba), four in total.
    • Attacks: white-box attacks TwinBreak [23], NeuroStrike [22], refusal-direction ablation [25]; black-box attacks are 10 categories selected from the 20 transformation categories of SORRY-Bench [21] (440 prompts per category, excluding multilingual and encoding/decoding categories).
    • Metrics and judges: white-box attacks use StrongREJECT [48] (313 malicious prompts, a fine-tuned LLM-as-a-judge scores 0–1, lower is safer); black-box attacks use SORRY-Bench's own fine-tuned judge, reporting the fulfillment rate over 440 prompts (i.e., ASR).
    • Over-refusal: XSTest [40], 150 of 250 benign prompts used for training and 100 for evaluation, reporting the false refusal rate using XSTest's keyword-matching judge.
    • Utility: few-shot accuracy on MMLU, TruthfulQA, ARC-Challenge, WinoGrande, HellaSwag.
    • Baselines: W-DOOR (only Gemma-2-2B), Deep Refusal and Circuit Breaker (only Llama-3-8B), all using the original papers' hyperparameters and not extended to other models.

    Main results

    According to Table 1 (TwinBreak, StrongREJECT score, lower is better):

    Swipe horizontally to read the table

    ModelMethodPer-iteration ASR (iterations 1–5)Cumulative ASR
    Gemma-2-2B-ITNo Defense0.55/0.58/0.59/0.55/0.530.71
    Gemma-2-2B-ITABS-Align0.07/0.15/0.15/0.15/0.130.23
    Llama-3-8B-InstructNo Defense0.64/0.66/0.68/0.69/0.680.81
    Llama-3-8B-InstructABS-Align0.04/0.04/0.07/0.06/0.030.11
    Llama-3.2-3B-InstructNo Defense0.59/0.54/0.36/0.40/0.350.74
    Llama-3.2-3B-InstructABS-Align0.19/0.22/0.15/0.15/0.110.33
    Qwen-3-4BNo Defense0.68/0.66/0.65/0.62/0.580.79
    Qwen-3-4BABS-Align0.04/0.04/0.04/0.07/0.050.13
    • The authors state that ABS-Align consistently reduces TwinBreak's per-iteration and cumulative scores on all four models, and that TwinBreak's ability to remove safety behavior by identifying and pruning safety neurons is impaired.
    • The authors state that the reductions in cumulative scores are about 70% (Gemma), 55% (Llama-3.2-3B), and 83% (Qwen-3-4B), with Llama-3-8B dropping by 0.7 points.

    Generalization to other sparsity attacks (Table 2)

    • What was tested: NeuroStrike (NS) and refusal-direction ablation (Rf-Dir) applied to ABS-Align fine-tuned models, evaluated with StrongREJECT.
    • Results: on Gemma-2-2B-IT, NS drops from 0.50 to 0.05 and Rf-Dir from 0.80 to 0.08; on Llama-3-8B, NS drops from 0.84 to 0.03 and Rf-Dir from 0.75 to 0.05; on Llama-3.2-3B, NS drops from 0.79 to 0.37 and Rf-Dir from 0.57 to 0.30; on Qwen-3-4B, NS drops from 0.89 to 0.44 and Rf-Dir from 0.70 to 0.09.
    • Authors' interpretation: robustness is not limited to TwinBreak's neuron identification method and can generalize to other attacks that exploit the sparsity of safety alignment; the NS improvements on Llama-3.2-3B and Qwen-3-4B are relatively small, and the authors state that they still exceed 50% in each setting.

    Generalization to black-box attacks (Table 3, Appendix 3)

    • What was tested: the ten categories of SORRY-Bench black-box transformation attacks, 440 prompts per category, reporting the fulfillment rate.
    • Results: average ASR over the ten categories, Gemma-2-2B-IT from 41.58% to 13.24%, Llama-3-8B from 48.34% to 17.79%, Llama-3.2-3B from 53.07% to 33.02%, Qwen-3-4B from 68.75% to 27.20% (Table 3).
    • Authors' interpretation: the fine-tuning targets only the objective of replicating safety neurons, without using black-box attack data augmentation or explicit guidance, yet still brings black-box robustness improvements.

    Comparison with baselines (Table 4)

    • Comparison with W-DOOR on Gemma-2-2B-IT: ABS-Align achieves TB 0.23, NS 0.05, SB 13.24%, Ref-Dir 0.08, over-refusal 4.00, all better than W-DOOR's 0.62, 0.19, 35.36%, 0.55, 9.00; on TruthfulQA, ABS-Align's 64.29 is higher than no defense's 55.15 and W-DOOR's 52.71.
    • Comparison with Deep Refusal and Circuit Breaker on Llama-3-8B: ABS-Align has the lowest ASR on the four attacks (TB 0.11, NS 0.03, SB 17.79%, Ref-Dir 0.05), but Circuit Breaker's SB is 16.06%, slightly lower than ABS-Align; Deep Refusal's over-refusal rate is 48.00, while ABS-Align's is 1.00.
    • Authors' interpretation: the authors state that ABS-Align has better robustness on all tested attacks and lower over-refusal; Circuit Breaker is slightly better on SORRY-Bench and slightly higher on MMLU and ARC, and is the exception in the table.

    Over-refusal and utility (Tables 5, 6)

    • Over-refusal (Table 5): Gemma-2-2B-IT rises from 2.00 to 4.00; Llama-3-8B drops from 5.00 to 1.00; Llama-3.2-3B stays at 4.00; Qwen-3-4B drops from 3.00 to 1.00 (denominator is 100 evaluation prompts).
    • Utility changes (Table 6): changes on WinoGrande, MMLU, and HellaSwag are mostly within ±3 percentage points for the four models; TruthfulQA rises for all, Gemma +9.14, Llama-3-8B +14.26, Llama-3.2-3B +13.78, Qwen-3-4B +11.75.

    Adaptive attacks (Table 8)

    • What was tested: increasing TwinBreak's pruning iteration budget from 5 to 10 and 15, simulating an adaptive attacker who knows the defense mechanism.
    • Results: ASR@15 is Gemma 0.27, Llama-3-8B 0.25, Llama-3.2-3B 0.39, Qwen-3-4B 0.29, corresponding to no-defense cumulative ASR of 0.71, 0.81, 0.74, 0.79.
    • Authors' interpretation: increasing the pruning budget brings only limited gains to the adaptive attacker, far from restoring the attack success rate of the undefended model.

    Other ablations and analyses (Table 7, Gemma-2-2B-IT, ASR is TwinBreak)

    • No neuron pruning (DPO only): ASR 0.52; random neuron pruning: 0.50; full method: 0.23.
    • Non-progressive (prune all at the start): 0.38; decreasing (prune all at the start then gradually restore): 0.31; randomly prune a portion of safety neurons at each step (r=0.2/0.4/0.6): 0.41/0.32/0.31.
    • Unfreezing the first and last transformer blocks: ASR unchanged at 0.23, but over-refusal rises to 7.0% and utility drops more.
    • Training for 4 epochs: ASR 0.32, over-refusal 5.0%; training for 1 epoch: ASR 0.45.
  5. What are some points that can be further explored?

    The authors note that neuron detection is imperfect, density is not quantified, and DPO has overhead; future directions include extending to other capabilities and improving the training stage.

    Limitations and future directions noted by the authors

    • Safety neuron detection is imperfect (Section 5.1): the detection method relies on heuristics (e.g., taking the top k with the largest activation differences), may miss some safety neurons and may wrongly affect unrelated neurons; some neurons have multiple functions, making it hard to attribute them exclusively to safety alignment. The consequence is that it is difficult to obtain a fully unaligned model for fine-tuning, and mistakenly pruning multifunctional neurons may harm general capabilities.
    • Safety alignment density was not directly quantified (Section 5.1): the authors state that, due to limitations of the detection method, they did not directly quantify the absolute density of safety alignment in the fine-tuned model; "denser safety alignment" only means reduced reliance on the originally identified sparse neuron set.
    • Computational and memory overhead of DPO (Section 5.1): DPO requires maintaining a frozen reference model in addition to the fine-tuned model, which the authors state still brings considerable computational and memory overhead.
    • Extension to other capabilities (Section 5.2): the authors propose future work on using the method to make the representation of specific capabilities (e.g., a task or language) more distributed.
    • Establishing non-sparse refusal representations in the main training stage (Section 5.2): the authors propose future work on whether non-sparse refusal representations can be established directly in the main stage of preference optimization, rather than being handled post-hoc.

    Scope of experiments

    • The models tested are Gemma-2-2B-IT, Llama-3-8B-Instruct, Llama-3.2-3B-Instruct, and Qwen-3-4B, four in total (Section 4.1).
    • White-box attack evaluation covers three: TwinBreak, NeuroStrike, and refusal-direction ablation; black-box attacks cover the ten transformation categories of SORRY-Bench, with multilingual and encoding/decoding categories excluded (Appendix 2).
    • Baseline comparisons are conducted only on the models each baseline was evaluated on in its original paper: W-DOOR only on Gemma-2-2B, Deep Refusal and Circuit Breaker only on Llama-3-8B (Section 4.5).
    • Adaptive attack evaluation covers only one strategy: increasing TwinBreak's pruning iteration budget (5→10→15) (Section 4.7).
    • Adversarial fine-tuning attacks are outside the threat model and were not evaluated in the paper (Section 3.2); the paper does not report the number of repetitions for each experiment.
  6. Summarize the main content of the paper

    ABS-Align makes safety alignment more distributed through progressive pruning plus DPO fine-tuning, and ASR of multiple attack types drops on four models.

    A defense paper: targeting the structural weakness that "safety alignment is highly sparse and pruning a small number of neurons can break it," it proposes the fine-tuning defense ABS-Align.

    • Problem: Existing research shows that safety alignment is concentrated in a few safety-critical neurons and a single refusal direction, and attackers of open-weight models can directly prune or ablate these components; the authors state that existing guardrails and general fine-tuning defenses do not address this sparsity.
    • Method: Use TwinBreak to identify safety-critical neurons; during DPO fine-tuning, prune a batch every 200 steps in order of importance from high to low (pruning rate 20%); in the later stage of fine-tuning all original safety neurons are temporarily pruned, forcing the model to rebuild refusal behavior through other neurons; after training all neurons are restored; the first and last transformer blocks are frozen.
    • White-box results: TwinBreak cumulative ASR on four models drops from 0.71–0.81 to 0.11–0.33 (Table 1); ASR for NeuroStrike and refusal-direction ablation also drops, e.g., on Llama-3-8B from 0.84 and 0.75 to 0.03 and 0.05 respectively (Table 2).
    • Black-box results: SORRY-Bench ten-category attack average ASR drops from 41.58%–68.75% to 13.24%–33.02%, and fine-tuning did not use black-box attack data (Table 3).
    • Cost and comparison: Over-refusal is essentially unchanged (Table 5), utility benchmarks mostly fluctuate slightly while TruthfulQA rises by 9.14–14.26 percentage points (Table 6); compared with W-DOOR, Deep Refusal, and Circuit Breaker, the authors state ABS-Align has lower ASR on almost all tested attacks, with the exception that Circuit Breaker is slightly lower on SORRY-Bench (16.06% vs. 17.79%).
    • Authors' conclusion: The authors state that ABS-Align provides a simple method to mitigate the fragility of safety alignment, extending robustness from neuron-level interventions to representation-level attacks and black-box jailbreaks; limitations include imperfect neuron detection, unquantified alignment density, and DPO's computational overhead.

Related papers

Show 3 more papers
Read originalarxiv.org