Protecting safety in reasoning training with low-rank Fisher regularization
Protecting Safety during Reasoning Training with Low-Rank Fisher Regularisation
The authors propose SP-EWC, which projects the Fisher onto the safety subspace identified by cPCA, maintaining jailbreak resistance at 92.0% (92.7% before training) during inference fine-tuning of Llama-3.1-8B-it without excessive refusal.
Defender scenario: a model already safety-aligned by the developer (θref) loses jailbreak resistance after benign SFT on about 1B tokens of reasoning traces. The defender has access to a safety dataset DSafety (harmful prompts paired with safe responses, sampled from θref) and a utility dataset DUtil, and adds a regularization term to the fine-tuning objective; no attacker manipulating the training data is involved.
- Safety guardrails in instruction models degrade after benign SFT on reasoning traces: the authors reproduce that at a high learning rate the mean across four jailbreak resistance benchmarks drops from 92.7% to 17.3%. Existing mitigation methods either protect poorly, introduce over-refusal, or incur high computational cost.
- Add a subspace penalty term on top of EWC: project each layer's gradients onto an r-dimensional safety subspace, compute the r×r projected Fisher to preserve interactions among safety directions, and compare three ways of localizing the subspace: LoRA, ActSVD, and cPCA.
- Math reasoning is trained on Llama-3.1-8B-it and OLMo-2-7B-it. The cPCA version performs best: on Llama, JR 92.0% (92.7% before training) and OA 87.0%; SafeInstr has higher JR (96.9%) but OA drops to 41.0%. Training time overhead is about +14%.
- The authors note: the evaluation covers only jailbreak and refusal benchmarks, and does not include bias, multilingual, or long-horizon tasks; only SFT is studied, with RL left for future work; the overhead is higher than cheaper methods such as SafeInstr. The experiments cover only two 7B–8B models.
- Evaluated models
- Llama-3.1-8B-itOLMo-2-7B-itGemma-2-9B-itMinistral-8B-itQwen-2.5-7B-it
- Benchmarks
- open-r1/Mixture-of-Thoughts (OpenR1-Math-220k)MATH-500OlympiadBenchsafety-eval: DoAnythingNow, WildJailbreak, WildGuardTest, TrustLLM (JailbreakTrigger), XSTestWildGuardMix
- Metrics
- Jailbreak resistance (JR)Over-refusal avoidance (OA)Pass@k (k=1, n=4)
The study proposes SP-EWC, which projects gradients onto a low-rank subspace where safe behaviors reside, protecting LLM safety guardrails during fine-tuning such as reasoning training. The method addresses the shortcoming of EWC's diagonal Fisher approximation, which ignores parameter interactions; the safety subspace can be identified in three ways: LoRA, ActSVD, and cPCA. The choice of subspace significantly affects performance; SP-EWC combined with cPCA maintains safety guardrails better than existing methods without introducing excessive refusal.
Paper analysis
What problem does this paper attempt to solve?
Addressing the problem that reasoning fine-tuning breaks safety guardrails: the authors propose SP-EWC, which performs non-diagonal EWC regularization within a safety subspace.
Core problem: safety guardrails in instruction models degrade after benign reasoning fine-tuning; how to preserve jailbreak resistance while learning new capabilities without over-refusal.
- Problem setting: the authors reproduce an existing finding—after SFT on about 1B tokens of reasoning traces, the model's mean across four jailbreak resistance benchmarks drops from 92.7% to 17.3% (at a high learning rate, a drop of 75.4 percentage points, introduction).
- Shortcomings of existing methods: the authors state that baselines such as mixing in safety data, gradient modification, safety LoRA, new regularization terms, and continual learning methods either fail to protect jailbreak resistance, introduce new problems such as over-refusal, or incur substantial computational cost; Alssum et al. (2025) found that standard EWC also does not preserve guardrails well.
- Core observation: prior work has shown that safety-related behavior lies in a low-rank subspace within the model (Arditi et al., 2024; Wei et al., 2024, etc.). Diagonal EWC characterizes only per-parameter sensitivity and ignores interactions between parameters, while the full Fisher is infeasible at LLM scale; the low-rank subspace makes a non-diagonal approximation feasible.
- Proposed in the paper: SP-EWC (Subspace-Projected EWC), which projects each layer's gradients onto a safety subspace and computes the r×r projected Fisher to penalize parameter drift; it also compares three ways of localizing the safety subspace: LoRA, ActSVD, and cPCA.
- Threat model:
- Scenario: defender's perspective, no malicious attacker; the threat comes from guardrail degradation caused by benign fine-tuning itself.
- Defender's knowledge: access to the reference model θref, the safety dataset DSafety (harmful prompts paired with safe responses), and the utility dataset DUtil.
- Defender's capabilities: modify the fine-tuning objective (add a regularization term), precompute statistics on θref.
- Victim system: an off-the-shelf instruction model safety-aligned by the developer (e.g., Llama-3.1-8B-it), to be SFT-trained for reasoning capability.
What related research exists?
Related work falls into four groups: fine-tuning breaking guardrails and its mitigation, EWC and its application to LLMs, and Fisher/Hessian approximation; this paper uses the low-rank nature of safety to improve EWC.
Fine-tuning breaks guardrails and mitigation methods
- Qi et al. (2024), Zhan et al. (2024), Yang et al. (2024), Casper et al. (2026)—find that benign fine-tuning alone can remove safety guardrails. Mitigation directions include lowering the learning rate (Kim et al., 2025; Lin et al., 2026), mixing in safety samples (Bianchi et al., 2024; Huang et al., 2024a; Fang et al., 2026), and extrapolating safety-critical directions (Perin et al., 2025).
- Complementary findings: refusal is mediated by a single direction (Arditi et al., 2024), safety training mainly affects the first few refusal tokens (Qi et al., 2025), a "safety basin" exists in weight space (Peng et al., 2024), and there exist sparse safety-critical weights that cannot be protected even by freezing (Wei et al., 2024). The authors state that this paper is precisely intended to exploit this low-dimensionality to protect safety during fine-tuning.
Elastic Weight Consolidation
- After Kirkpatrick et al. (2017) proposed EWC, it became a common baseline in continual learning (Huszár, 2018; Li et al., 2025b), and has been used on LLMs for regularized fine-tuning of world models (Xiang et al., 2023), continual pretraining of Gemma 2 on Lithuanian (Šliogeris et al., 2025), and regularization of low-rank updates (Wei et al., 2025; Zheng et al., 2026).
- Alssum et al. (2025) is closest to this paper; the paper notes that they found standard EWC does not preserve safety guardrails well, and the authors state that this paper aims to bridge this gap.
Fisher information matrix and Hessian approximation
- Curvature approximation in second-order optimization: K-FAC (Martens and Grosse, 2015), Shampoo (Gupta et al., 2018) and variants; low-rank Hessian approximation in influence functions (Schioppa et al., 2022).
- Ritter et al. (2018) combine such methods with EWC to characterize parameter interactions, but the paper notes that this introduces additional computational cost; the authors state that this paper instead uses the low-dimensionality of the safety subspace to efficiently characterize interactions among safety directions.
Baselines and benchmarks: the compared methods are Low-LR SFT (Kim et al., 2025; Lin et al., 2026), SafeInstr (Bianchi et al., 2024), SafeGrad (Yi et al., 2026), the KL variant of DER (Buzzega et al., 2020), and EWC (Kirkpatrick et al., 2017). Training uses a subset of open-r1/Mixture-of-Thoughts; evaluation uses MATH-500, OlympiadBench, and the AllenAI safety-eval suite (DoAnythingNow, WildJailbreak, WildGuardTest, TrustLLM, XSTest).
The authors position this paper as: using safety low-rank evidence to make non-diagonal EWC feasible, and state that (to our knowledge) it is the first to use contrastive PCA as a training intervention.
How does the paper address this problem?
SP-EWC: adds a "projected Fisher within the safety subspace" penalty on top of diagonal EWC, with the subspace identified by cPCA from contrasts between safety and utility activations.
Overall approach: on top of standard diagonal EWC, additionally penalize parameter drift within a safety low-rank subspace, using the projected r×r Fisher to characterize interactions among safety directions; the method is named SP-EWC.
Problem setting
- The reference model θref (e.g., Llama-3.1-8B-it) has completed safety training; SFT is performed on the reasoning dataset DSFT, with the goal of maintaining safety while minimizing over-refusal when learning a new task.
- Assume access to a safety dataset DSafety (M harmful prompts paired with safe responses) and a utility dataset DUtil; in the experiments, DSafety is obtained by sampling completions from θref for prompts from the safe split of the WildGuardMix training set and keeping those judged safe by a safety reviewer (Han et al., 2024), and DUtil is constructed from benign prompts via the same procedure (Appendix B).
Diagonal EWC and its shortcomings
- EWC objective: LEWC = LSFT + (λD)/2 ∑i Diemp Δi2, i.e., penalizing parameter deviation from θref weighted by the diagonal empirical Fisher.
- The authors construct DSafety by sampling from the model to estimate the true Fisher (following the recommendation of van de Ven, 2025).
- The authors state that even with λ tuned very high, the diagonal term alone is insufficient to maintain safety (λ sweep in Appendix E), hence the added subspace penalty.
Subspace-projected Fisher
- Project each layer's gradient gj onto the r-dimensional safety subspace to obtain g̃j = Pr gj, and average to obtain the projected Fisher: F̃ = 1/M ∑j g̃j g̃j⊤, which is only r×r per layer and characterizes interactions among the r output-side safety directions.
- Parameter changes are likewise projected to obtain Δ̃ = Pr Δ; the penalty term tr(Δ̃ᵀ F̃ Δ̃) is non-negative and grows quadratically with the change; the SP-EWC objective is LEWC plus the sum of this term over each linear layer (scaled by λLR).
- Motivation given by the authors: safety behavior is concentrated in a low-dimensional activation subspace (Wei et al., 2024), and modeling only output-side directions avoids the cost of input-output two-dimensional interactions; changes orthogonal to the safety directions are constrained only by the standard EWC term.
Three subspace localization methods
- LoRA: train a rank-r adapter on θref with safety data, perform SVD on ΔW = BA, and take the top-r left singular vectors to form Pr.
- ActSVD (Wei et al., 2024): take the top-r directions of safety activations to form Ps, then project out the utility directions Pu (Ŵ = (I − Pu)Ps W) and take the singular vectors.
- cPCA (Abid et al., 2018): find directions with large variance on safety activations and small variance on utility activations, i.e., maximize vᵀ(Cs − αCu)v, taking the top-r eigenvectors of Cs − αCu; α controls the strength of suppression of utility variance. The authors state that cPCA, unlike ActSVD, does not remove the utility subspace wholesale, avoiding discarding safety information shared with utility.
Implementation and evaluation
- Key parameters (Appendix B): r = 8, α = 100; cPCA caches linear-layer inputs using 10,000 samples each from DSafety and DUtil, taking only the first 20 tokens before the completion; the Fisher is computed sample by sample with batch size 1; embeddings, normalization layers, and the LM head are excluded.
- Success criteria: jailbreak resistance (JR) is the mean proportion of safe responses across four benchmarks, and over-refusal avoidance (OA) is the mean proportion of correct compliance across two benchmarks, both using AllenAI safety-eval default settings; reasoning uses Pass@k (k=1, n=4) with temperature 0.6, top_p 0.95, max_tokens 4096.
What experiments did the paper conduct?
Training math reasoning on Llama-3.1-8B-it and OLMo-2-7B-it: SP-EWC-cPCA preserves JR without over-refusing, with about +14% overhead.
Experimental Setup
- Models tested: Llama-3.1-8B-it and OLMo-2-7B-it; chosen because of stronger pre-training guardrails (Table 4: their JR is 92.7% and 87.8% respectively, versus Gemma-2-9B-it 57.8%, Ministral-8B-it 29.7%, Qwen-2.5-7B-it 47.9%).
- Training data: the math subset OpenR1-Math-220k of open-r1/Mixture-of-Thoughts, leaving 126k samples after filtering out those over 4,096 tokens (Appendix B).
- Baselines: Low-LR SFT, SafeInstr, SafeGrad, DER (KL variant), EWC.
- Metrics: JR = mean proportion of safe responses on DoAnythingNow, WildJailbreak (harmful), WildGuardTest (adversarial), TrustLLM (JailbreakTrigger); OA = mean proportion of correct compliance on XSTest and WildJailbreak (benign); reasoning = Pass@k on MATH-500 and OlympiadBench (k=1, n=4).
- Evaluation method: AllenAI safety-eval default settings; when constructing safety data, a safety judge (Han et al., 2024) determines whether completions are safe; the paper does not state the judge model for each benchmark inside safety-eval.
- Learning rate: after a sweep (Table 5), a low learning rate was chosen as the stronger baseline; the paper does not state the specific value for the main experiments in the main text.
Main Results
According to Table 2 (%, higher is better; M-500 = MATH-500, Olymp = OlympiadBench):
Swipe horizontally to read the table
Method M-500 Olymp JR mean OA mean Llama before training 44.9 16.1 92.7 88.2 Low-LR SFT 55.6 23.2 47.2 100.0 SafeInstr 52.6 23.0 96.9 41.0 SafeGrad 54.8 21.8 69.9 94.8 DER 53.8 21.1 76.1 94.0 EWC 51.8 20.8 77.3 96.4 SP-EWC-cPCA 51.0 19.8 92.0 87.0 OLMo before training 35.5 12.8 87.8 95.2 Low-LR SFT 47.5 20.5 76.6 92.0 SafeInstr 44.5 21.3 96.7 82.8 SafeGrad 33.1 11.7 89.4 95.2 DER 36.4 12.2 90.8 93.6 EWC 39.5 14.1 88.0 93.6 SP-EWC-cPCA 43.1 16.7 87.0 92.4 - The authors state that SafeInstr has the highest JR, even exceeding pre-training, but introduces substantial over-refusal (OA 41.0% on Llama), reproducing the finding of Bianchi et al. (2024).
- The authors state that SafeGrad and DER retain most of the math gains on Llama but fail to maintain JR; on OLMo JR is maintained better, but SafeGrad is worse on both math items and DER appears to stagnate.
- The authors state that SP-EWC-cPCA achieves a better balance among the three objectives; the cost is that on Llama the gains on MATH-500 and OlympiadBench are smaller than those of the tuned baselines.
Comparison of Subspace Localization Methods
- Figure 5 gives the percentage-point change of each method relative to before training: on Llama, LoRA's JR drops by 19–71 points across benchmarks and ActSVD by 22–72 points, while cPCA changes between −3 and +6; on OLMo, LoRA is −1 to −5 on each item, and ActSVD and cPCA are close (cPCA's MATH-500 change +8, ActSVD +5).
- The authors suggest that LoRA may capture directions adapted to the new safety data rather than existing guardrail directions; they suggest that ActSVD's poor performance on Llama may be related to OLMo's more robust guardrails and the complex interaction of its two hyperparameters, safety/utility rank (the authors use "speculate").
- Figure 4 applies cPCA projection to activations of Llama's layer 16 MLP down projection (2,000 safety + 2,000 utility samples): at α = 0 the two classes of projections overlap, and as α increases the variance in the safety direction increases relative to utility (the labeled var ratio safe/util rises from 10.7, 6.1 to 247.0, 188.8 at α = 100); the authors state this is consistent with cPCA finding a more safety-specific direction, and caution that the 2D projection should be interpreted carefully.
Ablation of Diagonal and Subspace Terms
- Table 1 (SP-EWC-cPCA decomposition): on Llama, "diagonal only" JR 77.3%, "subspace only" 89.9%, "diagonal + subspace" 92.0%; on OLMo the three JR values are 88.0%, 84.7%, 87.0%, but "subspace only" has MATH-500 of 46.4% (higher than the full version's 43.1%).
- The authors state that the trend differs by model: combining the two terms is key to maintaining JR on Llama, while on OLMo it helps recover more in-domain gains.
Other Ablations and Analyses
- EWC λ sweep (Figure 6, Llama): increasing λ better maintains JR, but at very large λ the in-domain gains keep shrinking and JR declines again (JR 69.2% at λ = 5000).
- DER safety data size sweep (Figure 7, Llama): a sufficiently large safety batch distribution is key, and protection weakens markedly with little data (JR 52.6% at n=10, 70.9% at n=31k).
- Learning rate sweep (Table 5): on Llama, JR 17.3% at 1×10⁻⁵ and 47.2% at 1×10⁻⁶; on OLMo, JR 18.9% at 5×10⁻⁵.
- SafeInstr safety data ratio sweep (Table 6): mixing in 1%–5% all maintains JR at 94.6%–98.7%, but OA drops to 34.2%–46.2% (Llama) and 80.4%–85.8% (OLMo).
- Computational cost (Table 3, OLMo, 4xH200): SFT 12.5h; EWC +15%, SP-EWC (cPCA) +14%, SafeGrad +69%, DER +93%.
What are some points that can be further explored?
The authors list limitations including the scope of evaluation, SFT-only, and higher cost than cheap methods; the experiments cover two 7B–8B models and mathematical reasoning training.
Limitations and future directions noted by the authors
- The evaluation operationalizes "safety" only with jailbreak and refusal benchmarks; its scope does not cover other safety aspects such as bias, multilingual behavior, or long-horizon tasks (Limitations section).
- Only SFT-based reasoning training is studied; effects during RL are left for future work (Limitations section).
- The overhead relative to standard EWC is negligible, but SP-EWC still costs more than cheaper methods such as SafeInstr (Limitations section).
- The method focuses on refusal rather than "safety" more broadly: the safe/unsafe classification depends on the policy written by the model developer, and a low ASR does not mean that harmful responses will not be produced under repeated sampling (Discussion section).
- Future direction: extend SP-EWC to capability protection beyond safety that likewise lies in a low-dimensional subspace (Discussion section).
Scope of the experiments
- The models tested are Llama-3.1-8B-it and OLMo-2-7B-it, two in total; the authors state that due to compute constraints they consider only models with at most 10B parameters (Appendix F).
- The training task is only math reasoning SFT (126k samples after filtering OpenR1-Math-220k, Appendix B).
- The safety evaluation uses four JR benchmarks and two OA benchmarks from safety-eval; the reasoning evaluation uses MATH-500 and OlympiadBench.
- The subspace dimension is fixed at r = 8 and cPCA's α = 100 (Appendix B); the paper does not report a sweep over r.
- The paper does not report the number of random seeds or repetitions for the main experiments (only the reasoning evaluation uses n=4 sampling to reduce variance), nor does it report the agreement between evaluation judgments and human judgments.
Summarize the main content of the paper
SP-EWC performs non-diagonal EWC within a safety subspace, uses cPCA to locate the subspace, and preserves jailbreak resistance while avoiding over-refusal on two models.
A defense-oriented paper: it addresses the degradation of safety guardrails in instruction models after benign reasoning SFT; the authors reproduce that at a high learning rate the mean across four jailbreak resistance benchmarks drops from 92.7% to 17.3%.
- Method: in addition to EWC's diagonal Fisher penalty, project each layer's gradients onto an r-dimensional safety subspace, compute the r×r projected Fisher, and penalize parameter drift in the safety directions (SP-EWC); compare three ways of localizing the subspace: LoRA, ActSVD, and cPCA, with key parameters r = 8, α = 100.
- Main results (Table 2): on Llama-3.1-8B-it, SP-EWC-cPCA has JR 92.0% (92.7% before training) and OA 87.0%; Low-LR SFT has JR of only 47.2%, and EWC 77.3%.
- Comparison: SafeInstr has higher JR (Llama 96.9%, OLMo 96.7%) but OA drops to 41.0% (Llama); the authors state it over-optimizes jailbreak resistance and introduces over-refusal.
- Subspace choice (Figure 5): cPCA is the most stable on both models; LoRA reduces JR by 19–71 percentage points across benchmarks on Llama, and ActSVD also causes JR declines on Llama.
- Cost (Table 3, OLMo): SP-EWC (cPCA) increases training time by 14% relative to SFT, versus SafeGrad +69% and DER +93%.
- Authors' conclusion: exploiting the low-rank nature of safety makes non-diagonal EWC feasible, achieving a better balance among jailbreak resistance, over-refusal, and reasoning gains; the authors also emphasize that the evaluation covers only the refusal dimension, that safety is not limited to preserving existing guardrails, and that RL settings and protection of broader capabilities are left for future work.