Distillation makes LLM unlearning more robust

1. 01What problem does this paper attempt to solve?

   The authors state that current large language model machine unlearning only produces superficial behavioral suppression and is vulnerable to reversal by relearning attacks such as fine-tuning, necessitating robust capability removal.

   The problem that large language model machine unlearning methods are vulnerable to fine-tuning attack reversal and cannot robustly remove latent dangerous capabilities.

   • Problem Scenario and Importance: Large language models acquire harmful capabilities (e.g., knowledge related to cyber weapon development) during pretraining on massive, insufficiently filtered data. Post-training alignment (e.g., RLHF) only suppresses harmful outputs under common inputs; the underlying capabilities remain intact and can be induced by malicious attacks.
   • Shortcomings of Existing Methods: The authors state that existing machine unlearning methods are essentially behavioral suppression rather than true capability deletion; attackers only need a small number of fine-tuning steps on the unlearned model to recover the unlearned capabilities.
   • Core Observation and Hypothesis: The authors observe that even when the model's output behavior is completely consistent with an ideal Oracle model that has never seen harmful data, the model still retains latent capabilities that can be quickly recovered under fine-tuning; distilling the unlearned model's outputs into a new network can transfer only the desired behavior while leaving latent capabilities behind.
   • Proposed Solution: The authors propose the Unlearn-and-Distill and UNDO (Unlearn-Noise-Distill-on-Outputs) frameworks, establishing an adjustable trade-off between computational cost and unlearning robustness through a three-step process of unlearning, parameter noise addition, and output distillation.
   • Threat Model:
     • Attack Goal: Recover suppressed undesirable or dangerous capabilities in the victim model through adversarial relearning attacks.
     • Attacker Knowledge: The paper assumes black-box or gray-box relearning fine-tuning; the attacker does not need access to internal inversion gradients from the unlearning process, but has access to the forget set or related distribution data.
     • Attacker Capability: The attacker can perform gradient fine-tuning on the forget set with multiple learning rates, different data formats (e.g., QA pairs), different sequence lengths, and strategies such as noise-then-fine-tune for worst-case attacks.
     • Victim System: Large language models processed by machine unlearning algorithms (e.g., GradDiff, MaxEnt, RMU) or hardened by UNDO distillation before deployment.

   Collapse this answer

2. 02What related research exists?

   Related work covers classical and large-model unlearning, unlearning vulnerability evaluation, and robust unlearning defenses. The authors state that this paper uses distillation and noise addition to break the retention of latent capabilities in the original parameter space.

   Large model unlearning research focuses on behavior approximation and vulnerability assessment, and existing robust solutions are mostly limited to local optimization within the original model's parameter space.

   Classical and Large Model Machine Unlearning

   • Exact and approximate unlearning: Cao and Yang [10] (2015) proposed the concept of machine unlearning to simulate a model trained without the forgotten data; Bourtoule et al. [8] (2021) studied exact unlearning, but the authors point out that it is difficult to apply to large models due to scale and non-convexity; Maini et al. [50] (2024), Jang et al. [35] (2023), and Yao et al. [89] (2023) formulated large model unlearning as optimization problems that maximize prediction loss or align distributions on the forget set.

   Unlearning Vulnerability and Relearning Evaluation

   • Fine-tuning recovery and representation residue: Lo et al. [47] (2024), Hu et al. [33] (2025), Lynch et al. [49] (2024), and Deeb and Roger [18] (2024) pointed out that unlearned models can recover suppressed capabilities within a few fine-tuning steps; Zhang et al. [92] (2024) pointed out that weight quantization attacks can also exploit numerical instability to induce unlearning failure.

   Robust Defense Schemes Against Relearning

   • Flatness and noisy representation optimization: Fan et al. [23] (2025) introduced SAM to find flat loss regions to reduce model sensitivity to fine-tuning; Rosati et al. [61] (2024) proposed RepNoise to inject noise at the representation layer; Dong et al. [19] (2024) proposed UNDIAL, a self-distillation method, which the authors note differs in that it does not use global parameter corruption; Chundawat et al. [15] (2023) explored a student-teacher framework where unlearning is guided by an incapable teacher.

   Baseline Methods and Evaluation Benchmarks

   • Experimental comparison baselines and benchmarks: The baseline methods selected in the paper include GradDiff [50], MaxEnt [44], RMU [42], SAM [23], RepNoise [61], and UNDIAL [19]; evaluation benchmarks include synthetic language datasets built from FineWeb and FineWeb2, synthetic arithmetic datasets, the weapons of mass destruction proxy benchmark WMDP (WMDP-Bio, WMDP-Cyber) [42], and the TOFU benchmark [50].

   Difference from this work: The authors state that previous methods mainly perform local modifications or behavior suppression within the original parameter space where harmful knowledge has been learned, thus leaving traces of latent capabilities; this work instead uses the unlearned model's outputs as a teacher to distill into a newly initialized or globally perturbed parameter space, turning the distillation mechanism into a filter that blocks latent knowledge transfer.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors propose the UNDO framework, which first performs behavior unlearning on the model, then corrupts the model through global parameter perturbation, and finally distills on unlabeled pretraining data to recover retain capability.

   The authors propose Unlearn-and-Distill and the generalized framework UNDO (Unlearn-Noise-Distill-on-Outputs), whose core idea is to suppress model behavior outputs through unlearning methods, then introduce global parameter corruption, and finally distill the unlearned model's outputs into the corrupted network, achieving a balance between computational cost and robustness.

   Behavior Unlearning Phase (Unlearn)

   • Teacher model construction: Based on a pretrained model with dual capabilities (retain and forget), a standard unlearning algorithm is used to generate a teacher model with suppressed behavior.
   • Unlearning algorithms and objectives:
     • Gradient Difference (GradDiff) uses the objective function ℒ(θ) = −ℒ_forget(θ) + ℒ_retain(θ), where both terms are cross-entropy losses, aiming to push away the prediction probabilities on forget samples while maintaining retain capability.
     • Maximum Entropy (MaxEnt) minimizes the sum of KL divergence to a uniform distribution and the cross-entropy loss on the retain set, making the model's outputs on the forget task maximally uncertain.
     • Representation Misdirection (RMU) uses mean squared error (MSE) at intermediate layers to push the activation vectors of forget samples toward random directions while maintaining activation similarity for retain samples.

   Parameter Corruption Phase (Noise)

   • Controlled parameter perturbation: To balance the high computational cost of full reinitialization and the low robustness of the original parameter space, the authors design a perturbation update formula: θ_perturbed = (1 − α)θ_suppressed + αβ N This formula uses a mixing coefficient alpha and a noise scaling factor beta to perform a convex combination of the suppressed model parameters and sampled noise, globally corrupting the network.
   • Perturbation mechanism details: The noise matrix N is sampled using Xavier initialization for two-dimensional weight tensors, and one-dimensional bias tensors are set to zero; when alpha=1 and beta=1, it is equivalent to full network random initialization, and when beta=0, it is equivalent to parameter shrinkage.

   Output Distillation Phase (Distill)

   • Forward KL divergence distillation: Using the behavior-suppressed teacher model as the target, the corrupted student model is distilled on unlabeled pretraining data using forward KL divergence, prompting the student to recover retain capability.
   • Stopping condition and data ratio: Distillation continues until the student model reaches at least 95% of the teacher's performance on the retain set, or completes one full epoch of the pretraining data, whichever comes first. The retain-to-forget data mixing ratio is set to 1:1 for language tasks and 1:3 for arithmetic tasks.

   Post-Training Unlearning Extension

   • Base model replacement: For harmful behaviors acquired during post-training, the authors point out that if the pretraining phase did not encounter relevant data, the original base model without post-training can be directly used as the distillation target network, without random initialization from scratch.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The paper evaluates fine-tuning attacks and quantization attacks on synthetic language, synthetic arithmetic, WMDP, and TOFU benchmarks, comparing 6 unlearning baselines and retraining from scratch with data filtering.

   Experimental Setup

   • Models under test: Gemma-2-0.1B (100M parameters, for language tasks), Gemma-2-0.3B (300M parameters, for arithmetic tasks), Gemma-2-2B (for the WMDP benchmark and TOFU experiments), Phi-1.5 (for TOFU experiments).
   • Evaluation datasets and scale: Language tasks use FineWeb-Edu (English retained) and FineWeb2 (Korean forgotten), each sampled at 10 million rows, with validation sets of 500 thousand tokens each; arithmetic tasks contain 1 million four-operation problems (addition and subtraction retained, multiplication and division forgotten), with a training set of 500 thousand arithmetic problems and validation sets of 500 thousand each; WMDP covers the Bio and Cyber domains (full evaluation set used, with a 40% subset of MMLU used for the retained evaluation); TOFU contains question-answer pairs for 200 fictitious authors.
   • Comparison baselines: GradDiff, MaxEnt, RMU, SAM, RepNoise, UNDIAL, and the gold standard of full filtering and retraining from scratch (Data Filtering).
   • Metric definitions: Language tasks evaluate cross-entropy loss (CE Loss, on the retained and forgotten sets); arithmetic and WMDP tasks evaluate accuracy (Accuracy); TOFU evaluates Forget QA Prob, Forget Truth Ratio, and Model Utility.
   • Evaluation and attack methods: The fine-tuning attack trains for at most 500 steps on the forgotten set, performing a grid search over 10 logarithmically spaced learning rates (1e-3 to 1e-6) to select the best attack performance under the worst case; on WMDP, 0-shot prompting is used to evaluate forgetting and 5-shot prompting is used to evaluate MMLU retained accuracy; the paper does not use an external LLM as a judge for scoring.
   • Number of repetitions and hardware setup: WMDP experiments use the mean over 4 random seeds; parameter analysis experiments report the median and error bars over 5 random seeds; experiments are run on multiple H200 and A100 GPUs.

   Main Results

   The evaluation results of the INT4 weight quantization attack against Gemma-2-2B on the TOFU benchmark are as follows:

   Swipe horizontally to read the table

   Method and variant	Precision	Forget Q&A Prob	Forget Truth Ratio	Model Utility
   GradDiff (not distilled)	FP16	0.000	0.000	0.495
   GradDiff (not distilled)	INT4	0.008	0.185	0.242
   GradDiff + Distilled	FP16	0.000	0.001	0.442
   GradDiff + Distilled	INT4	0.003	0.017	0.391

   (Note: Data from Table 14; the model under test is Gemma-2-2B; lower metric values indicate better forgetting.)

   • Forgetting metrics under the quantization attack: The authors report that under INT4 quantization, the Forget Truth Ratio of non-distilled GradDiff rises to 0.185, whereas that of distilled GradDiff+Distilled is only 0.017 (according to Table 14).
   • Model utility retention: The authors state that the distilled model maintains better utility under the quantization attack (0.391 vs. 0.242, according to Table 14).
   • Representation-level mechanistic explanation: The authors believe this result supports the hypothesis of distillation as a capability filter: the student network never possesses the forgotten knowledge in parameter space, and thus lacks numerically unstable latent structure that quantization could exploit.

   Relearning Robustness on Pretraining Tasks and the Pareto Frontier

   • What was tested: On synthetic language and synthetic arithmetic tasks, the performance of non-distilled unlearning (GradDiff, MaxEnt, RMU), the noise-distillation method UNDO, other robust baselines (SAM, RepNoise, UNDIAL), and the gold standard of filtering and retraining from scratch under 500-step fine-tuning relearning was compared.
   • Results: In language tasks, UNDO's resistance to relearning increases as the perturbation parameter alpha increases from 0.1 to 0.8, and under stronger settings the relearning trajectory approaches the gold standard of retraining from scratch on filtered data (according to Figure 4 and Figure 5); under a 500-step high-compute attack, UNDO extends the Pareto frontier of retained performance and forgetting resistance (according to Figure 7).
   • The authors' interpretation: The authors state that conventional unlearning methods degrade rapidly under fine-tuning pressure, whereas the Pareto frontier established by UNDO is closer to the gold standard of full retraining, without requiring large-scale data annotation.

   Evaluation of Fine-Tuning-Resistant Defenses on the Real Benchmark WMDP

   • What was tested: On Gemma-2-2B, the performance of MaxEnt and RMU combined with UNDO (alpha=0.25, beta=0) in the WMDP-Bio and WMDP-Cyber domains was evaluated under 7 adversarial fine-tuning schemes covering different data mixtures, question-answer formats, and prompt lengths.
   • Results: Averaged over 4 random seeds, UNDO improves resistance to fine-tuning relearning for both WMDP-Bio and WMDP-Cyber (according to Figure 8); in the MMLU retained evaluation, the UNDO model is on average 4.88% lower (Section 6), and Table 13 shows that as alpha increases to 0.6, MMLU after retraining drops to 0.25.
   • The authors' interpretation: The authors state that UNDO is on the Pareto frontier on WMDP-Bio and competitive on WMDP-Cyber, but on this task it does not clearly surpass the baseline frontier as it does on synthetic tasks; the authors speculate the reason is that limited computational resources caused the distillation corpus to account for only 0.015% of the original pretraining scale.

   Analysis of the Trade-off Between Parameter Perturbation and Compute

   • What was tested: In language and arithmetic tasks, alpha was scanned with beta=0.1 fixed (0.1 to 0.8), and beta was scanned with alpha fixed (0.05 to 1.0), recording the number of distillation steps required to recover to 95% of the teacher's retained performance and the relearning curves (according to Figure 5 and Figure 6).
   • Results: Changes in alpha have a clearly dominant effect on robustness, while the effect of beta's value is flatter; at alpha=0.1, recovering retained capability requires 1 distillation step, and at alpha=0.7 it requires 300 to 500 steps (according to Figure 6(e)).
   • The authors' interpretation: The authors believe the degree of global corruption is the core driver determining computational requirements and robustness, and that users can achieve a continuous trade-off between computational cost and safety robustness by adjusting the perturbation coefficient.

   Other Ablations and Analyses

   • Language task MaxEnt learning rate scan: at lr of 8e-5, retained CE Loss is 4.9895 and forgotten CE Loss is 12.3808 (Appendix B.1).
   • Arithmetic task MaxEnt learning rate scan: at lr of 9e-5, retained accuracy is 0.795 and forgotten accuracy is 0.0125 (Appendix B.1).
   • Arithmetic task RMU learning rate scan: at lr of 8e-6, retained accuracy is 0.615 and forgotten accuracy is 0.0675 (Appendix B.1).
   • Effect of the perturbation parameter beta on the number of distillation steps: at alpha=0.5, beta values of 0.05, 0.1, 0.5, and 1.0 all require 200 distillation steps (Figure 6(e)).
   • Effect of the WMDP noise coefficient alpha on MMLU: 0.55 at alpha of 0, 0.49 at 0.25, 0.37 at 0.35, and 0.25 at 0.6 (Table 13).

   Negative Results and Exceptions

   • In the arithmetic domain, the improvement in relearning resistance from RMU combined with distillation is smaller than that of MaxEnt; the authors state the reason is that RMU's initial retained-forgotten trade-off is worse (retained accuracy only 62%, forgotten accuracy 6.8%, whereas MaxEnt is 80% and 1.3%, page 5 and Appendix B.1).
   • Under the WMDP setting, UNDO causes Gemma-2-2B's MMLU to drop by an average of 4.88%, failing to open up a Pareto frontier gap from the baselines as on synthetic tasks; the authors speculate this is due to undertraining from distilling on only 300 million tokens (0.015% of the original pretraining corpus) (page 9, page 29).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that distillation is computationally expensive, causes a 4.88% MMLU drop on WMDP, and that noise sampling and activation-based distillation remain to be explored.

   Limitations and Future Directions Noted by the Authors

   • Decreased Downstream Retained Performance: In Gemma-2-2B WMDP experiments, UNDO models average a 4.88% drop in MMLU benchmark scores (Section 6 Limitations).
   • Real Benchmark Trade-off Not Widened: In the WMDP setting, other methods achieve similar retained-forget trade-offs, and UNDO does not surpass baseline Pareto frontiers as in synthetic tasks; the authors speculate the reason is insufficient distillation training, with distillation data only 0.015% of the original pretraining corpus scale (Section 6 Limitations and Appendix D.2.7).
   • Significantly Higher Computational Cost than Fine-tuning Unlearning: Compared to traditional fine-tuning-based unlearning methods, the distillation process consumes much more computational resources (Section 6 Limitations).
   • Noise Sampling Diversity to Explore: Parameter perturbation currently mainly uses Xavier initialization sampling; the authors note that other random noise sampling methods warrant further exploration (Section 5).
   • Pretraining Latent Traces Hard to Fully Exclude: Constructing ideal evaluation sets for unseen pretraining data is challenging because the model has seen all tokens in the vocabulary during pretraining; simple keyword or semantic matching cannot ensure the complete absence of latent traces (Appendix F).
   • Distillation Form and Scheduling Exploration: The authors explored noise schedulers and distillation mechanisms based on activation states rather than output logits in preliminary work; these directions can be further studied in the future (Appendix G).

   Experimental Coverage

   • Models under test cover 4 model architectures: Gemma-2-0.1B, Gemma-2-0.3B, Gemma-2-2B, and Phi-1.5 (Sections 3, 6 and Appendices A, F).
   • Evaluation tasks cover synthetic language (English-Korean), synthetic four-operation arithmetic, WMDP (WMDP-Bio and WMDP-Cyber), and the fictional character QA benchmark TOFU (Sections 3, 6 and Appendix F).
   • Adversarial attack evaluation includes worst-case fine-tuning attacks with 10 learning rate sweeps (up to 500 steps), 7 data and format combinations of relearning variants, and INT4 weight quantization attacks (Sections 3, 6 and Appendices D.2.5, F.2).
   • Comparison baseline methods cover 6 unlearning algorithms: GradDiff, MaxEnt, RMU, SAM, RepNoise, and UNDIAL, plus filtered retraining from scratch as a control (Sections 4, 6).
   • The paper does not report evaluation data on reasoning capabilities in non-English major languages for unlearning and distillation, nor does it report consistency scores from external LLMs or human evaluation.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors claim that distillation can block the transfer of harmful latent capabilities, and propose the UNDO framework to balance computational cost and robustness through noise addition and distillation, making unlearned models harder to reverse via fine-tuning.

   This study addresses the deficiency that large language model machine unlearning defenses are difficult to resist fine-tuning relearning attacks, and proposes the UNDO framework for robust capability removal through noise addition and output distillation.

   • Core Problem: Conventional unlearning techniques are highly susceptible to recovering unlearned capabilities under parameter fine-tuning or quantization perturbation. The authors find that even when the model's output fully fits an Oracle teacher that has never seen harmful data, dangerous knowledge still lurks within the network.
   • Method Design: The authors construct the UNDO pipeline of first behavioral suppression, then global parameter noise addition (combining coefficient alpha and noise scale beta), and finally forward KL divergence distillation on unlabeled corpora, turning distillation into a barrier that filters latent capabilities.
   • Key Quantitative Results:
     1. In synthetic language and arithmetic tasks, UNDO exhibits stronger anti-relearning performance as perturbation parameters increase, with relearning curves approaching retraining from scratch on filtered data (per Figure 4, Figure 5).
     2. On the WMDP benchmark, applying UNDO (alpha=0.25, beta=0) to Gemma-2-2B improves resistance against 7 adversarial fine-tuning configurations, with MMLU maintained at 0.49 (original model 0.55, average drop 4.88%, per Table 13 and Section 6).
     3. Under INT4 quantization attacks on the TOFU fictional QA benchmark, GradDiff after distillation has a Forget Truth Ratio of 0.017, while the undistilled model has 0.185 (per Table 14).
   • Conclusion and Implications: The authors argue that there is an essential disconnect between the model's "exhibited behavior" and its "latent capabilities"; distillation can block the transfer of latent representations that are not explicitly activated, providing a practical path for industrial deployment of large models by converting conventional model compression pipelines into safety defense barriers.

   Collapse this answer

Related papers