Unlearning is not deletion: exploring the reversibility of machine unlearning in large language models

1. 01What problem does this paper attempt to solve?

   The authors state that task-level metrics cannot easily distinguish true knowledge elimination from superficial suppression, and propose a representation-level analysis framework for the reversibility of unlearning in large models.

   This paper attempts to investigate whether knowledge is truly eliminated after machine unlearning in large language models, and whether task-level evaluation metrics can reflect the reversibility and destructiveness of unlearning.

   • Application scenario and importance: Large models' memorization of training data can easily lead to survival-sensitive privacy risks, copyright infringement, or output of harmful text; machine unlearning aims to eliminate the influence of specified data at the algorithmic level without retraining the full model.
   • Limitations of existing evaluation: Existing research typically relies on task-level metrics such as unlearning set accuracy and perplexity; the authors point out that such metrics can be misleading, and apparent performance degradation may only represent temporary suppression of information, which can be restored by a small amount of subsequent fine-tuning.
   • Core observation and hypothesis: The authors hypothesize that task-level collapse is not equivalent to internal knowledge elimination, and that variations in unlearning algorithms and hyperparameters cause qualitatively different dynamic shifts in the model's internal representation layers.
   • Main approach of the paper: The authors propose a representation-layer analysis diagnostic toolbox (covering PCA similarity and shift, CKA, Fisher information, and mean PCA distance), establish a four-category unlearning mechanism from the two dimensions of reversibility and catastrophicness, and adopt a restricted relearning protocol as an empirical probe for detecting knowledge retention.
   • Recovery attack setting (Threat Model):
     • Attack goal: Recover knowledge of the target unlearning set Df that has been eliminated or suppressed.
     • Attacker knowledge and capability: Cannot access the original pretraining full corpus; only attempts to recover knowledge through lightweight fine-tuning (relearning) on data of restricted scale (strictly matching the unlearning set size |Df|), prompt attacks, jailbreaks, or low-bit quantization.
     • Victim system: Open-source large language models processed by single or continual unlearning algorithms.

   Collapse this answer

2. 02What related research exists?

   The authors review single-shot forgetting, continual forgetting, and recovery evaluation studies, noting that existing work mostly focuses on behavioral performance and lacks mechanistic analysis at the internal representation level.

   In the introduction and related work, the authors systematically review three categories of related research and clarify the comparison baselines and benchmark datasets:

   Single-Shot Unlearning Methods

   • Gradient-based objective functions: Yao et al. (2024) studied gradient ascent (GA) as well as variants regularized with retention set cross-entropy (GA+GD) or KL divergence (GA+KL); the authors note that pure gradient ascent is often accompanied by large utility degradation.
   • Weight attribution and preference optimization: Jia et al. (2024) proposed WAGLE (GA+GD+WAGLE), which selectively updates important parameters through weight attribution to balance unlearning and utility; Zhang et al. (2024) proposed negative preference optimization (NPO) to penalize fitting of the unlearning set; Li et al. (2025a) proposed the model editing method AlphaEdit, whose properties under sequential requests the authors note remain underexplored; Liu et al. (2024) proposed a prompt manipulation method, which the authors note is a shallow form of unlearning without parameter updates.

   Continual Unlearning Methods

   • Error accumulation under continual requests: Barez et al. (2025), Shi et al. (2025), and Gao et al. (2025) studied sequentially arriving unlearning requests, noting that naive extensions of single-shot methods cause error accumulation, leading to simultaneous collapse of retention and unlearning sets.
   • Parameter decoupling and modular defenses: Hu et al. (2022) adopted LoRA-based orthogonal updates to mitigate degradation; Wuerkaixi et al. (2025) proposed the ALKN framework, countering cumulative degradation through parameter-level intervention and adaptive modules.

   Unlearning Evaluations and Vulnerability Analysis

   • Behavioral evaluation and memory probes: Yao et al. (2024) and Li et al. (2024b) used accuracy and perplexity as general evaluations; Lee et al. (2024) proposed verbatim memory probes, which the authors note struggle to reflect knowledge retention at the semantic level.
   • Robustness and recovery methods: Lo et al. (2024) and Lynch et al. (2024) evaluated lightweight fine-tuning relearning; Patil et al. (2024) and Liu et al. (2023) tested prompt and jailbreak attacks; Zhang et al. (2025) found that low-bit quantization recovers unlearned knowledge; Rezaei et al. (2025) proposed RESTOR to evaluate parameter state recovery; Hu et al. (2025) and Yang et al. (2026) explored the awakening effect of fine-tuning on benign data.

   Baseline methods and benchmark datasets

   • Comparison baselines: Core evaluations include GA, GA+GD, GA+KL, NPO, NPO+KL, and random labels (RLabel); extended evaluations include RMU, UnDIAL, AltPO, PDU, and GA+GD+WAGLE.

   • Benchmark datasets: Simple tasks use arXiv abstracts and GitHub code; complex tasks use NuminaMath-1.5; downstream general utility uses CommonsenseQA, GSM8K, and MATH; out-of-distribution testing uses TOFU and the Traditional Chinese corpus.

   • Positioning of this paper: The authors note that most previous research treated reversibility and catastrophic unlearning as phenomena at the external behavioral level, lacking mechanistic analysis from an internal structural perspective; this paper is the first to systematically analyze the mechanisms of unlearning reversibility from representation-layer geometry and parameter sensitivity.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors construct a representation-layer toolbox combining PCA shift, CKA, and Fisher information along with a restricted relearning protocol, formally distinguishing four unlearning mechanisms from the perspective of perturbation mechanisms.

   The authors propose a unified representation-layer analysis framework, combining four feature and parameter diagnostic tools along with a restricted relearning protocol, to investigate the reversibility of unlearning from internal mechanisms.

   Formal definition of the four-category unlearning mechanism

   • Two-dimensional decoupling: The authors decouple the model state after unlearning along two dimensions: the degree of damage to the retention set (catastrophicness) and the degree of recovery of the unlearning set after relearning (reversibility).
   • Four specific mechanisms:
     1. Reversible catastrophic unlearning: both unlearning set and retention set performance collapse, but can be recovered through relearning;
     2. Reversible non-catastrophic unlearning: only the target unlearning set performance degrades, the retention set remains intact, but the unlearned content is easily recovered by relearning;
     3. Irreversible catastrophic unlearning: unlearning set and retention set performance permanently collapse, and relearning cannot recover them;
     4. Irreversible non-catastrophic unlearning: the ideal state, where target unlearning set knowledge is persistently erased and the retention set is not harmed.
   • Mechanism determination expression: Let Δ_u(T) = E(θ₀, T) − E(θ_u, T) be the performance drop after unlearning, and Δ_r(T) = E(θ₀, T) − E(θ_r, T) be the gap between post-relearning and initial performance. When Δ_u(T_r) ≈ 0 and Δ_r(T_f) ≫ 0, this corresponds to the ideal irreversible non-catastrophic unlearning.

   Representation-layer multi-dimensional diagnostic toolbox

   • PCA Similarity & Shift: Extract activation matrices of unlearned and relearned models at each hidden layer, compute the first-order cosine similarity of principal component directions (representing global alignment), and project onto the first two principal component planes of the initial model to compute two-dimensional mean coordinates (representing local translational shift).
   • Mean PCA Distance: To compress representation shift into a single scalar metric, the authors define the cross-layer average Euclidean distance:

   PCA dist = 1/L ∑_i=1^L √((PC1Δ(i))² + (PC2^upd(i) − PC2^orig(i))²), which quantifies the mean translational shift of activations on the principal component plane across layers.

   • Centered Kernel Alignment (CKA): Compute the linear centered kernel alignment coefficient, comparing the activation Gram matrices before and after unlearning, with values ranging from 0 to 1, used to measure the degree of overlap of activation subspaces.
   • Empirical Fisher Information Matrix (FIM): Estimate the diagonal elements of the empirical Fisher information, characterizing changes in local loss surface flatness and parameter sensitivity caused by unlearning through the mean distribution of squared parameter gradients and their spectral shift.

   Restricted relearning protocol

   • Restricted budget design: To strictly distinguish relearning from full retraining, relearning only performs lightweight fine-tuning on the unlearned model, strictly limiting the training sample size to the same as the unlearning set size |Df|, and does not touch the original pretraining corpus.
   • Comparison of three data sources:
     1. The unlearning set itself (simulating the worst-case recovery condition);
     2. A subset of the domain-aligned retention set (reflecting in-distribution signal recovery);
     3. Completely unrelated out-of-distribution data (testing whether unrelated data produces an awakening effect).
   • Optimization and hyperparameter configuration: The optimizer is AdamW, using a cosine annealing schedule with 10% warmup and decay to 10% of the peak, weight decay of 0.1, and gradient clipping threshold of 1.0.

   Perturbation theory model derivation

   • Low-order and high-order perturbation expansion: The authors model layer-wise updates as a composition of layer-wise perturbations in a feedforward network, noting that under small learning rates or few requests, the network is dominated by low-order perturbations, where high-level logits are affected while deep representation geometry remains intact, leading to performance degradation that is highly reversible.
   • Cross-layer cumulative damage: Large learning rates or many accumulated requests cause perturbations to propagate across layers through high-order interactions, leading to subspace fracture and loss surface flattening, resulting in irreversible collapse.

   Collapse this answer

4. 04What experiments did the paper conduct?

   In experiments across 7 open-source models and multiple types of tasks, the authors state that classical metrics often mask reversibility, while mean PCA distance can provide early warning of irreversible collapse.

   Experimental Setup

   • Models under test: Tested 7 open-source models: Yi-6B, Qwen-2.5-7B, Qwen2.5-3B, Qwen3-8B-Base, Qwen3-14B, Llama-3-8B, and Llama-3-8B-Instruct.
   • Evaluation tasks and datasets: Simple tasks use arXiv abstracts and GitHub code; complex tasks use NuminaMath-1.5; retained-set downstream benchmarks include CommonsenseQA (CSQA), GSM8K, and MATH; distribution tests include TOFU and traditional Chinese corpus taide-bench.
   • Comparison algorithms: Main evaluations cover GA, GA+GD, GA+KL, NPO, NPO+KL, RLabel, with extended evaluations of RMU, UnDIAL, AltPO, PDU, and GA+GD+WAGLE.
   • Metric definitions: Forget accuracy (F.Acc), retain accuracy (R.Acc), forget/retain perplexity (F.Ppl/R.Ppl), MIA AUC (min-k%-prob), PCA similarity and displacement, average PCA distance, linear CKA, and FIM diagonal spectrum.
   • Evaluation protocol and sample size: No LLM-based scoring; relearning budget strictly matches forget set size; Table 4 reports results from four random seeds with 95% confidence intervals.
   • Hardware environment: All experiments run on NVIDIA H100 GPUs.

   Main Results

   Swipe horizontally to read the table

   Setting and method	F.Acc after unlearning (%)	R.Acc after unlearning (%)	F.Acc after relearning (%)	R.Acc after relearning (%)	MIA after unlearning (%)	MIA after relearning (%)
   Original baseline	78.9	65.5	78.9	65.5	70.8	70.8
   GA (LR=3e-5, N=100)	0.0	0.0	2.1	1.8	26.1	74.5
   GA (LR=5e-6, N=100)	9.1	6.2	80.0	64.9	23.2	68.0
   GA (LR=3e-6, N=100)	16.8	14.4	80.8	65.2	25.2	68.6
   GA+GD (LR=3e-5, N=100)	9.7	2.3	2.2	2.6	16.8	68.1
   NPO (LR=3e-5, N=100)	37.8	37.9	57.0	45.6	60.1	70.0
   RLabel (LR=3e-5, N=100)	0.0	0.0	4.3	2.8	65.1	69.5

   (According to Table 3, Yi-6B simple task continual unlearning and full forget set relearning evaluation results)

   • Reversible catastrophic unlearning performance: Under moderate learning rates (e.g., GA at LR=3e-6, N=100), F.Acc drops to 16.8% and R.Acc to 14.4% after unlearning, but relearning recovers F.Acc to 80.8% and R.Acc to 65.2%, indicating that surface collapse does not destroy internal representations.
   • Irreversible catastrophic unlearning performance: Under aggressive hyperparameters (e.g., GA at LR=3e-5, N=100), all metrics approach 0 after unlearning, and after relearning F.Acc is only 2.1% and R.Acc only 1.8%, with average PCA distance reaching 104.58 (Table 4), indicating irreversible representation collapse.
   • Relearning exceeding initial baseline: In gentle settings, relearned F.Acc exceeds the initial baseline (e.g., GA at LR=3e-6, N=100 reaches 80.8% after relearning, higher than the initial 78.9%), which the authors attribute to the unlearning process acting as contrastive regularization.

   Effectiveness Comparison of Alternative Recovery Strategies

   • Test setup: On Yi-6B (GA, LR=6e-6, N=100), evaluate recovery effects of Int4 quantization, 5-shot context prompting, prompt rewriting attacks, jailbreak attacks, and relearning (Table 11).
   • Experimental results: After unlearning, F.Acc is 0.00% and average PCA distance is 31.66; after quantization attack, F.Acc is 0.00% (PCA distance 32.21), after 5-shot prompting 0.01% (PCA distance 30.83), after prompt attack 0.03% (PCA distance 29.14), after jailbreak attack 0.03% (PCA distance 30.04); relearning fine-tuning recovers F.Acc to 78.21% (PCA distance drops to 1.04) (Table 11).
   • Authors' interpretation: The authors note that once a model enters the reversible catastrophic mechanism, inference-time interventions that do not update parameters or only apply mild perturbations cannot awaken representations; explicit parameter updates are required to recover knowledge.

   Relearning Data Source and Sample Efficiency Analysis

   • Test setup: On Yi-6B, test forget set, retain set, and unrelated data with relearning budgets of 10%, 30%, 60%, and 100% (Table 12).
   • Experimental results: With only 10% of the forget set for relearning, F.Acc recovers to 67.28% (PCA distance drops to 8.49); with 10% retain set or unrelated data, F.Acc is only 0.05% and 0.02% (PCA distances 30.57 and 31.02, respectively); even with 100% unrelated data, F.Acc reaches only 65.66% (Table 12).
   • Authors' interpretation: The authors note a clear hierarchy in recovery efficiency across data sources, with the forget set being most sample-efficient, while recovery from retain set or unrelated data is significantly slower.

   Exploring the Boundary of Irreversible Non-Catastrophic Unlearning

   • Test setup: Use GA+GD+WAGLE with parameter-selective updates, combined with restricted retain set or unrelated data relearning (Table 5, Table 14).
   • Experimental results: On Yi-6B (LR=4e-5, N=50), after relearning with unrelated data, forget set F.Acc increases only slightly from 27.8% to 31.5% (PCA distance stays at 24.56), while retain set R.Acc remains at 53.5% (PCA distance only 8.12) (Table 5); similar trends are observed on Qwen3-8B-Base and Llama-3-8B (Table 14).
   • Authors' interpretation: The authors state this condition presents a seemingly irreversible and non-catastrophic unlearning case, but note that achieving persistent knowledge elimination without harming retained utility remains an open challenge.

   Other Ablations and Analyses

   • Complex math task (Qwen-2.5-7B, GA, LR=5e-6, N=6): after unlearning, MATH is 0.0% and GSM8K is 0.0%; after relearning, MATH recovers to 9.3% and GSM8K to 77.8% (Table 7).
   • Single-step unlearning (Yi-6B, GA, LR=3e-6): after unlearning, F.Acc drops to 65.4% and R.Acc to 54.0%; after relearning, F.Acc recovers to 76.6% and R.Acc to 65.2% (Table 2).
   • Cross-lingual and TOFU evaluations (Yi-6B, GA, LR=3e-6): after relearning, forget set PCA distance for traditional Chinese drops from 17.12 to 4.98; for TOFU, PCA distance drops from 0.51 to 0.27 (Table 13).
   • Extended unlearning algorithms (Yi-6B, LR=3e-5): UnDIAL achieves F.Acc of 61.3% after relearning (PCA distance 8.80), AltPO achieves F.Acc of 52.4% (PCA distance 14.90), with better representation retention than GA (Table 15).
   • Probability-based MIA failure phenomenon: under aggressive continual unlearning (Yi-6B, GA, LR=3e-5, N=100), MIA AUC drops to 26.1% after unlearning and rebounds to 74.5% after relearning (Table 3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that the framework has not been validated on very large models and production pipelines and lacks formal privacy guarantees; future work needs to extend to a broader range of representation analysis tools and threshold settings.

   Limitations and future directions noted by the authors

   • Model and scenario scale: The experiments only tested several open-source large models and some tasks and unlearning methods; although the diagnostic framework is model-agnostic, empirical validation on very large models and production-scale pipelines remains to be done (Appendix A.1).
   • Lack of formal privacy guarantees: The restricted relearning protocol and selected metrics clearly reflect representation shift, but are not exhaustive and cannot provide formal privacy guarantees (Appendix A.1).
   • Breadth of representation analysis tools: The research mainly relies on four tools: PCA similarity, PCA shift, CKA, and Fisher information; introducing broader analysis tools such as correlation-based SVCCA is an important direction for future work (Appendix A.1).
   • Openness of threshold quantification: Precisely defining the specific numerical thresholds for unlearning mechanisms (reversible vs. irreversible, catastrophic vs. non-catastrophic) is not straightforward, because these thresholds are influenced by the unlearning method and task complexity (Section 5.4).
   • Hyperparameter sensitivity across model families: Different model families exhibit differences in hyperparameter sensitivity when reaching comparable behavioral mechanisms; exploring how this sensitivity determines the boundary between reversible and irreversible can help guide the design of stable and persistent unlearning algorithms (Section 5.4).

   Experimental coverage

   • Models tested: The experiments cover 7 open-source models: Yi-6B, Qwen-2.5-7B, Qwen2.5-3B, Qwen3-8B-Base, Qwen3-14B, Llama-3-8B, and Llama-3-8B-Instruct (Section 3.1, 3.2, Table 8-10).
   • Tasks and benchmarks: Includes arXiv abstracts, GitHub code, NuminaMath-1.5, CommonsenseQA, GSM8K, MATH, as well as TOFU and the Traditional Chinese corpus taide-bench (Section 3.1, Appendix A.5).
   • Unlearning methods: Includes 11 methods: GA, GA+GD, GA+KL, NPO, NPO+KL, RLabel, GA+GD+WAGLE, RMU, UnDIAL, AltPO, and PDU (Section 3.1, Table 5, Table 15).
   • Relearning evaluation conditions: Sample size strictly matches the unlearning set size; tested unlearning sets (10% to 100%), retention sets, and unrelated data; single-step learning rates range from 3e-6 to 6e-6, continual unlearning includes 6 to 100 requests and learning rates from 3e-6 to 6e-5 (Section 3.1, Appendix A.4.2).
   • Unreported information statement: The paper does not report p-values for statistical tests when reporting task-level metrics, and does not provide specific statistics on the number of samples in the training corpus (Table 2-4).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that task-level metrics in large model unlearning evaluation are misleading, and propose representation-layer diagnostic tools and a four-category mechanism to achieve more trustworthy unlearning evaluation.

   • Paper positioning: This paper studies the reversibility of machine unlearning in large language models at the representation level, addressing the disconnect between surface-level task metric declines and persistent elimination of internal knowledge.
   • Problem addressed: Existing unlearning evaluations commonly rely on task-level metrics such as accuracy and perplexity, but models often recover knowledge with a small amount of fine-tuning after appearing unlearned; traditional evaluations struggle to distinguish true knowledge elimination from superficial temporary suppression.
   • Core method design: The authors propose a representation-level diagnostic toolbox including PCA similarity and displacement, linear CKA, empirical Fisher information, and average PCA distance, constructing a four-category unlearning mechanism taxonomy along the "reversibility" and "catastrophic" dimensions, and design a budget-limited relearning protocol as a probe for representation retention.
   • Main experimental findings:
     1. In single-shot unlearning on a simple task with Yi-6B, GA reduces forget accuracy to 65.4%, but fine-tuning relearning recovers it to 76.6% (Table 2), with layer-wise CKA near 1.0, confirming that single-step unlearning is mostly reversible non-catastrophic suppression;
     2. In continual unlearning, with moderate parameters (GA, LR=3e-6, N=100), forget accuracy drops to 16.8% but relearning recovers to 80.8% (Table 3), while with aggressive parameters (LR=3e-5, N=100), relearning accuracy is only 2.1% and average PCA distance rises to 104.58 (Table 4), marking a shift to irreversible catastrophic collapse;
     3. Alternative recovery methods fail under the reversible catastrophic mechanism: after Int4 quantization and 5-shot prompting, forget accuracy remains 0.00% and 0.01% (Table 11); in contrast, relearning with only 10% of the forget set recovers 67.28% accuracy (Table 12);
     4. With GA+GD+WAGLE combined with unrelated data relearning on Yi-6B, the authors observe a seemingly irreversible and non-catastrophic unlearning case where forget accuracy stays low at 31.5% and retain accuracy remains at 53.5% (Table 5).
   • Authors' conclusions and implications: The authors point out that task-level metrics alone cannot reliably determine unlearning effectiveness; small output-layer perturbations can mask intact internal representations; they suggest future unlearning evaluations should combine task metrics, restricted relearning probes, and representation-level geometric analysis for more credible assessment.

   Collapse this answer

Related papers