Decoupling Forgetting and Utility: Lightweight LLM Unlearning via Orthogonal Null-Space Projection
Disentangling Forgetting from Utility: Lightweight LLM Unlearning via Orthogonal Null-Space Projection
The authors propose UUE/UUE-L, recasting LLM unlearning as target-free editing under a left-null-space constraint of retained knowledge, and report a better unlearning–utility balance than baselines on TOFU and WMDP.
- LLMs memorize and reproduce private or harmful content, necessitating unlearning mechanisms; the authors state that mainstream fine-tuning-based unlearning struggles to balance computational efficiency and general utility, and that targeted unlearning may learn only surface-level refusal.
- Proposes UUE: insert an adapter after the FFN of a selected Transformer block, constrain the update to the approximate left null space of the retain-set representations, and obtain a closed-form solution with a convex objective that pushes forget-set representations toward the antipodal point; UUE-L instead uses LoRA low-rank iterative optimization.
- On TOFU (Phi-1.5B, LLaMA-2-7B) and WMDP (Phi-1.5B, Zephyr-7B, etc.), the authors report that UUE/UUE-L's FQ and ROUGE-L Forget outperform the baselines, that on WMDP the forgetting–utility trade-off outperforms GA, NPO, and RMU, and that the highest MIA AUC is 0.6372.
- The authors note that MIA is sensitive to the construction of member/non-member samples and serves only as an auxiliary diagnostic, and that the goal is knowledge removal rather than a formal privacy guarantee; the experiments cover 5 models and two benchmarks, TOFU and WMDP, and the paper does not report statistical significance tests or multiple repetitions.
- Threat model
- Defender scenario: the model owner holds a trained LLM and needs to perform untargeted unlearning on a specified forget set DF (without presupposing a replacement answer) while maintaining utility on an approximate retain set DR; backbone parameters are frozen, and only the inserted adapters are updated.
- Evaluated models
- Phi-1.5BLLaMA-2-7BZephyr-7BLlama-3-8B-InstructYi-34B-Chat
- Benchmarks
- TOFUWMDPMMLUWikiText
- Metrics
- Forget Quality (FQ)Model Utility (MU)ROUGE-L Forget/RetainWMDP accuracyMMLU accuracyMIA AUCruntime
Researchers propose the UUE framework, which reframes LLM unlearning as controllable model editing that removes private or dangerous knowledge with minimal interference to general capabilities. The method projects parameter updates onto the orthogonal left null space of retained knowledge, uses a closed-form solution to a convex objective to push the forgetting-set embeddings toward their antipodes, and is deployed via pluggable adapters (including a low-rank variant, UUE-L) without modifying the frozen backbone. On the TOFU and WMDP benchmarks, the unlearning effectiveness of UUE and UUE-L is significantly better than existing methods.
Paper analysis
What problem does this paper attempt to solve?
The authors aim to address the difficulty in LLM unlearning of balancing "forgetting specified knowledge" and "preserving general utility," and propose the editing-based framework UUE.
Core problem: how to make an LLM delete specified knowledge (private or harmful content) without harming the model's general utility.
- Scenario and motivation: the authors state that LLMs memorize and reproduce private, sensitive, or illegal content during training (e.g., a financial QA model memorizing customer account information), so unlearning must be performed on specified forget data while preserving the ability to answer unrelated content.
- Shortcomings of existing methods: the authors argue that mainstream fine-tuning-based unlearning methods have two problems—computational inefficiency, and unexpected degradation of general utility due to knowledge entanglement in the parameters (CH-1); parameter-preserving methods (guardrails, prompt engineering) are easy to deploy but unsafe, as forgotten knowledge can still be extracted under adversarial queries.
- Core observation: the authors reframe unlearning as controlled model editing—guiding parameter updates into the left null space of retain-knowledge representations can remove undesired knowledge with controlled, minimal interference; targeted unlearning pulls forget-set representations toward a single refusal target, which may cause distribution distortion and learn only surface-level suppression (Figure 1(a)).
- Method proposed in the paper: the UUE framework—insert a lightweight pluggable adapter after the FFN of a selected Transformer block (backbone frozen), use an untargeted editing objective to push forget-set representations toward their antipodal point, and constrain the update with a null-space projection derived from retain-set representations; there is also a low-rank variant UUE-L.
- Threat model:
- Goal: obtain an unlearned model θu that forgets the knowledge corresponding to the forget set DF, maintains performance on the retain set DR, and ideally approximates a model retrained from scratch on DR only.
- Setting: untargeted unlearning—no replacement answer is presupposed (as distinct from targeted unlearning, which forces output of a fixed refusal).
- Defender capabilities: access to DF and an approximate retain set (composed of neighboring data semantically related to DF and general-domain data such as WikiText), ability to insert adapters in selected layers and collect FFN output representations.
- Victim system: a trained LLM (in the experiments the backbone parameters are frozen and only the adapters are updatable).
What related research exists?
The paper groups related work into three categories—non-LLM unlearning, parameter-preserving LLM unlearning, and parameter-updating LLM unlearning—and uses 8 methods including GA and RMU as baselines.
Non-LLM machine unlearning
- Early exact unlearning work (Bourtoule et al. [9–11]) achieves unlearning by partitioning data subsets, training submodels separately, and then ensembling them; recent work performs approximate unlearning using Hessian/Fisher information matrices [12–15] or fine-tuning [16–18]. The paper notes that these methods are not applicable to LLMs due to the parameter scale and training corpus scale of LLMs.
Parameter-preserving LLM unlearning
- Guardrail-type methods [19, 20] suppress inputs and outputs related to the forget data using keyword matching, OOD detection, etc.; prompt-engineering-type methods [6, 21–23] guide the model away from undesired outputs by prepending constructed prompts. The paper notes that because the model itself is unchanged, unlearned knowledge may still be accessible under adversarial queries.
Parameter-updating LLM unlearning
- Loss-design type: log-likelihood-type methods penalize responses to forget data [24–26], reward performance on retain data [27], or enforce refusal [28, 29]; entropy-enhancement-type methods maximize output entropy on forget data [30] or minimize divergence from a uniform distribution [31]; knowledge-distillation-type methods use a reference model to guide fine-tuning [32–34].
- Three cross-cutting directions: controlling unlearning–retention interference (e.g., adjusting gradient direction [4], selecting update locations by forget/retain impact [35]); drawing on knowledge editing (AlphaEdit projects edits into the null space of retained knowledge [7], Editing as Unlearning treats refusal or empty answers as the unlearning target [36]); parameter-efficient methods (LoKU initializes LoRA with Fisher information [37], ReGLU initializes LoRA with retained-representation geometry and regularizes outputs to the orthogonal complement of the retained subspace [38]).
Baselines and benchmarks
- Comparison baselines: GA, GradDiff, IDK, DPO, NPO, ME+GD, FLAT, RMU, and the paper's own UU-Naive; on WMDP, DPO, IDK, FLAT, and U-Trivial are excluded due to data-format constraints. The benchmarks are TOFU and WMDP, with utility evaluated using MMLU.
Positioning of this work: The authors state that these works collectively highlight a challenge—how to construct effective unlearning updates without presupposing alternative answers, while limiting interference with retained knowledge and without modifying the pretrained backbone; UUE responds to this challenge through a forget-conditioned editing objective and a null-space-constrained update applied to a detachable adapter.
How does the paper address this problem?
UUE formulates unlearning as a convex editing problem under a left-null-space constraint of the retained set: the adapter update has a closed-form solution, and UUE-L uses LoRA low-rank alternating optimization.
Overall idea: rewrite LLM unlearning as null-space-guided model editing—insert an adapter after the FFN of a selected Transformer block, first "reinforce" forget-set representations toward the original answer direction and then use their antipodal point as the editing target, while constraining the update to the approximate left null space of retain-set representations so that interference with retained knowledge has an upper bound.
Problem Setup
- Model p(·|x; θ), forget set DF ⊆ D, retain set DR = D \ DF; the goal is for θu to approximate a model θr retrained on DR only.
- In practice the full DR is often unavailable; the paper follows the TOFU protocol to construct an approximate retain set: neighboring data semantically related to DF (TOFU uses fictional author samples, WMDP uses benign academic text) plus general-domain data (WikiText).
Lightweight Unlearning Adapter
- Based on existing findings that FFNs store factual and semantic knowledge and that adapters can serve as modular carriers of knowledge [40–44], an adapter is inserted after the FFN of a selected block: apply a = U h to the FFN output h, with U initialized to the identity matrix I, so insertion initially does not change the computation.
- The edit is stored in U rather than the FFN weights; removing the adapter restores the original block computation; during unlearning the backbone is frozen and only the selected adapters are updated.
Null-Space Projection
- Collect the FFN outputs of the forget set and retain set at the selected layer to form representation matrices HF and HR; the optimization objective is Δ U = argmin Lf + |Δ Ũ HR|2, i.e., the forgetting loss plus a retain-representation shift penalty.
- By Proposition IV.1, the left null space of HR equals the null space of its Gram matrix HR HR⊤, so EVD is performed on the d×d Gram matrix, and eigenvectors with eigenvalues below threshold γ span the approximate null space, with projection matrix N = W2 W2⊤.
- When the update is constrained to ∆U N, it is bounded: |(Δ U N)HR|F ≤ √γ|Δ U N|F: the retain-side protection is approximate and controlled by γ; a smaller γ gives tighter protection but fewer editable directions.
Untargeted Editing Objective and Closed-Form Solution
- The naive approach directly maximizes the distance between the edited forget representations and the target representations, but the target has no lower bound; UUE instead uses a convex quadratic objective Lf−UUE = |(U+Δ U)HF + AF|2, "pulling" the forget representations toward the antipodal point −AF.
- Target representation construction: add a residual to each forget token representation and use gradient descent to optimize that residual to increase the likelihood of the original answer (backbone and adapter fixed), yielding ai = U hi + δh*i, then use −ai as the target in representation space; this target does not presuppose a replacement answer.
- Under the constraint E = EN and λ1 regularization, the objective is strictly convex, and the closed-form solution is ∆U_UUE = −(U HF + AF)HF⊤ N (HF HF⊤ N + λ1 I)^(−1), which can be solved via a symmetric positive-definite linear system without explicit inversion; once the target representations are constructed, the adapter update requires no iterative optimization.
UUE-L Low-Rank Variant
- The update is parameterized as ∆U = QP N (Q ∈ R^{d×r}, P ∈ R^{r×d}, r ≪ d), reducing trainable parameters per layer from d² to 2dr; the objective is the low-rank version with regularization coefficient λ2.
- Q and P are optimized by alternating gradient descent: P is initialized to N (0,1) and Q to 0, iterating until convergence or the maximum number of iterations T.
- Computational cost: constructing the Gram matrix per layer is O(d²nr), EVD is O(d³); UUE's closed-form update is O(d²nf) plus an O(d³) linear solve, while UUE-L replaces the solve with iterative optimization; both variants bear the projection and target-construction overhead.
What experiments did the paper conduct?
Compares 8 baselines on TOFU and WMDP; the authors report that UUE/UUE-L forget more strongly and preserve utility better, and conduct sensitivity, ablation, MIA and runtime analyses.
Experimental Setup
- Models evaluated: TOFU uses the Phi-1.5B and LLaMA-2-7B released by TOFU (both fine-tuned on the full TOFU); WMDP uses pretrained Phi-1.5B and Zephyr-7B, with supplementary experiments using Llama-3-8B-Instruct and Yi-34B-Chat (compared with RMU only).
- Benchmarks: TOFU (200 synthetic author biographies, each with 20 QA pairs; forget01/05/10 correspond to forgetting 1%/5%/10% of authors); WMDP (three subsets WMDP-Bio, WMDP-Cyber, WMDP-Chem; the forget set is the domain documents provided by WMDP, and the retain set is WikiText).
- Baselines: GA, GradDiff, IDK, DPO, NPO, ME+GD, FLAT, RMU, and the paper's own UU-Naive (a naive objective with null-space projection).
- Metrics: TOFU reports FQ (KS-test p-value against the output distribution of the retrained model), MU (harmonic mean of 9 sub-metrics), ROUGE-L Forget/Retain; WMDP reports WMDP accuracy (ideal value about 25%, i.e., random guessing) and MMLU accuracy.
- Implementation details: adapters are inserted in middle layers (layers 10–15 for Phi-1.5B, layers 12–18 for LLaMA-2-7B and Zephyr-7B); UUE-L's LoRA settings are r=16, α=32, dropout 0.05 on Phi-1.5B, and r=32, α=64, dropout 0.05 on LLaMA-2-7B and Zephyr-7B; hyperparameter grid search; hardware is two NVIDIA A800 GPUs.
- Evaluation method: all metrics are computed automatically (ROUGE-L, KS test, QA accuracy); the paper does not use LLM judging; the paper does not report the number of repetitions.
Main Results
According to Table I (TOFU forget01/05) and Table II (WMDP):
Swipe horizontally to read the table
Model Task Method FQ ↑ MU ↑ R-F ↓ R-R ↑ Phi-1.5B forget01 UUE 0.0541 0.5204 0.3178 0.9228 Phi-1.5B forget01 UUE-L 0.0415 0.5287 0.3985 0.9235 Phi-1.5B forget01 RMU 0.0030 0.4999 0.9031 0.9235 LLaMA-2-7B forget01 UUE 0.0268 0.6135 0.4562 0.9109 LLaMA-2-7B forget01 UUE-L 0.0198 0.6146 0.4486 0.9220 Phi-1.5B forget05 UUE 2.61E-07 0.5002 0.3967 0.9116 LLaMA-2-7B forget05 UUE-L 6.24E-08 0.5865 0.5030 0.8842 WMDP (Table II, WMDP-Avg. ↓ / MMLU ↑): on Phi-1.5B, UUE is 0.3440 / 0.4160, UUE-L is 0.3553 / 0.4125, RMU is 0.3869 / 0.4100, NPO is 0.4073 / 0.4090; on Zephyr-7B, UUE is 0.3539 / 0.4870, UUE-L is 0.3446 / 0.4925, RMU is 0.3564 / 0.4240. The original models are 0.4332 / 0.4390 and 0.5449 / 0.6078, respectively.
- The authors state that UUE/UUE-L have higher FQ and lower ROUGE-L Forget on both tasks, with the forgetting improvement especially pronounced on Phi-1.5B; forget05 is overall harder, with most methods having FQ below 0.01.
- The authors state that GA and ME+GD push WMDP accuracy close to random guessing but at the same time greatly lower MMLU, indicating that low WMDP accuracy alone is insufficient to demonstrate selective forgetting; UUE/UUE-L retain more MMLU performance while lowering WMDP accuracy.
- The authors state that the relative merits of UUE and UUE-L depend on the model: on Phi-1.5B, UUE forgets slightly more strongly, while on Zephyr-7B, UUE-L is slightly better on both metrics; no consistent advantage with increasing scale is established.
Supplementary WMDP Experiments on Larger Models
- What was measured: WMDP unlearning on Llama-3-8B-Instruct and Yi-34B-Chat, compared only with RMU (Table III).
- Results: on Llama-3-8B, UUE's WMDP-Bio/Cyber are 0.316/0.229 (RMU is 0.469/0.283), MMLU 0.590 (RMU 0.572, original model 0.631); on Yi-34B-Chat, UUE is 0.452/0.274 (RMU is 0.512/0.290), MMLU 0.710 (RMU 0.706, original model 0.726).
- The authors state that UUE has lower WMDP accuracy and slightly higher MMLU on both models, but MMLU is still lower than the original models.
Membership Inference Attack (MIA)
- What was measured: MIA AUC using Min-k% Prob on TOFU forget01, Phi-1.5B; a higher AUC indicates that forgotten samples look more like non-members (Table V).
- Results: UUE's AUC is 0.6372, UUE-L is 0.6154, RMU is 0.6127, and the original model is 0.4391; the authors state that all unlearning methods increase AUC relative to the original model.
- The authors emphasize that MIA serves only as an auxiliary diagnostic: prior research shows it is sensitive to the construction and distribution of member/non-member samples, and the goal of this paper is knowledge removal and utility preservation, not a formal privacy guarantee.
Runtime
- What was measured: end-to-end runtime on the same hardware on TOFU forget01, Phi-1.5B (Figure 4).
- Results: UUE is 179.17 s (null-space EVD 94.69 s + target representation computation 74.57 s + adapter update 9.91 s), UUE-L is 174.82 s; among baselines, FLAT is 208.03 s, ME+GD 215.42 s, GradDiff 217.96 s, DPO 258.77 s, NPO 349.56 s, RMU 393.17 s.
- The authors state that the closed-form update accounts for only 5.5% of total runtime, and the bottleneck is subspace construction and target representation computation, which explains why UUE-L's runtime differs little from UUE's.
Other Ablations and Analyses
- Regularization strengths λ1, λ2 (Figure 2, TOFU forget05, Phi-1.5B): as regularization increases, MU and R-R rise and R-F increases (forgetting becomes weaker); if too weak, utility decreases; the chosen values are marked with dashed lines in the figure.
- Null-space threshold γ (Table IV, TOFU forget01, Phi-1.5B): at γ=0.001, FQ 0.0423, R-F 0.3321; at γ=0.1, FQ 0.0540 but MU drops to 0.5039 and R-R drops to 0.9081; default γ=0.01.
- Edited layer range (Table IV): early layers L1–6 have FQ 0.0289, MU 0.4581, R-R 0.6717; late layers L18–23 have FQ 0.0063, R-F 0.6552; middle layers L10–15 are the most balanced.
- Null-space projection ablation (Figure 3(a)): removing the projection (setting N to the identity matrix), UUE-w-N and UUE-L-w-N show decreased MU and R-R, and FQ also decreases; the authors state that the projection is necessary to constrain destructive updates.
- Retain-set coverage (Figure 3(b)): when the number of WikiText samples is reduced, MU and R-R begin to deteriorate when the sample size is too small, while R-F is almost unchanged; the authors state that a medium-sized retain set suffices to approximate the retain subspace.
- Case study (Table VI): NPO repeatedly reproduces answers close to the original source, and U-Trivial outputs repeated phrases or generalized refusals; the authors state that UUE does not reproduce answers beyond the information in the question itself, but some responses are incomplete or incoherent, which only illustrates forgetting behavior and cannot prove answer quality.
What are some points that can be further explored?
The limitations the authors explicitly state center on the reliability of the MIA diagnostic and the absence of a formal privacy guarantee; the experiments cover 5 models and 2 benchmarks.
Limitations and future directions noted by the authors
- MIA serves only as an auxiliary diagnostic: the authors note in Section V-G that prior studies show LLM MIA performance is sensitive to the construction and distribution of member and non-member samples, limiting its reliability as a standalone unlearning metric.
- No formal privacy guarantee: the authors note in Section V-G that the main goal of this paper is knowledge removal and utility preservation, not establishing a formal privacy guarantee.
- Impact of retained-set coverage: the authors note in Section V-F that a finite retained set can only approximate the knowledge encoded by the model, that overly sparse coverage weakens this approximation and reduces utility preservation, and that maintaining sufficient retained-set representativeness is important.
- Evidential boundaries of the case study: the authors note in Section V-H that some UUE answers are incomplete or incoherent, and that these examples demonstrate unlearning behavior but cannot prove answer quality.
- The paper has no dedicated Limitations or Future Work section; the above are all boundary statements marked by the authors themselves in the experimental analysis sections.
Scope of experimental coverage
- 5 models were tested: Phi-1.5B and LLaMA-2-7B on TOFU; Phi-1.5B and Zephyr-7B on WMDP, with supplementary experiments on Llama-3-8B-Instruct and Yi-34B-Chat (compared only against RMU).
- The benchmarks are TOFU (two task levels, forget01 and forget05) and WMDP (Bio and Cyber subsets), with utility evaluated using MMLU; the forget10 level of TOFU is defined in the paper but no results are reported.
- The sensitivity, ablation, MIA, and runtime experiments were all conducted only under the single setting of TOFU forget01 or forget05 with Phi-1.5B (Figure 2, Figure 3, Table IV, Table V, Figure 4).
- The paper reports no number of repetitions, random seeds, or statistical significance tests for any experiment; the MIA experiments report no member/non-member sample sizes.
- On WMDP, DPO, IDK, FLAT, and U-Trivial are not applicable because they require conversational forget data, and the paper accordingly did not include these baselines.
Summarize the main content of the paper
UUE performs LLM unlearning using retained-set null-space projection plus a target-free editing objective, and reports a better unlearning–utility balance than baselines on TOFU/WMDP.
An LLM machine unlearning methods paper: it reframes unlearning from "fine-tuning penalty" to controlled model editing, with the goal of deleting specified knowledge while constraining interference with general capabilities within a provable bound.
- Problem: the authors state that fine-tuning-based unlearning is computationally inefficient and harms utility due to knowledge entanglement; targeted unlearning pulls representations toward a fixed refusal target and may learn only surface-level suppression.
- Method: UUE inserts an identity-initialized adapter after the FFN of selected middle layers (backbone frozen, pluggable removal); it first uses gradient descent to construct forget-set representations that reinforce the original answers, then obtains a closed-form solution for the adapter update via a convex quadratic objective targeting their antipodal point; the update is constrained to the approximate left null space spanned by directions of the retain-set Gram matrix with eigenvalues below threshold γ, and the retain-representation shift has a √γ upper bound; UUE-L changes the update to a LoRA low-rank decomposition with alternating optimization.
- TOFU results (Table I): on Phi-1.5B forget01, UUE's FQ is 0.0541 and ROUGE-L Forget is 0.3178, versus baseline RMU's 0.0030 and 0.9031; on LLaMA-2-7B forget01, UUE-L's MU of 0.6146 is among the highest in the whole table.
- WMDP results (Table II): on Zephyr-7B, UUE-L's WMDP-Avg. is 0.3446 and MMLU 0.4925, versus RMU's 0.3564/0.4240; although GA pushes WMDP down to 0.2453, MMLU is only 0.2431, on which basis the authors point out that low WMDP accuracy does not equal selective forgetting.
- Supplementary evidence: on MIA AUC, UUE is 0.6372 (Table V; the authors emphasize it is only an auxiliary diagnostic); on TOFU forget01, runtime is 179.17 s for UUE and 393.17 s for RMU (Figure 4), with the bottleneck in null-space construction and target representation computation.
- Authors' conclusion: on the evaluated models, UUE/UUE-L improve forgetting on TOFU and provide a competitive forgetting–utility trade-off on WMDP; the authors also acknowledge the limitations of MIA, the absence of a formal privacy guarantee, and that some case responses are incomplete and cannot prove generation quality.