Layered unlearning for adversarial relearning

1. 01What problem does this paper attempt to solve?

   The authors hypothesize that single unlearning installs only a shared inhibitor, so they use layered unlearning to induce multiple inhibitors to resist subset relearning.

   Post-training changes are often undone by fine-tuning on subsets; the authors attribute this to a single shared inhibitor and break it apart with layered unlearning.

   • Scenario: Fine-tuning, preference learning, and machine unlearning are used to change the behavior of pretrained LLMs. The authors say these changes are often shallow or fragile and can be bypassed by adversarial prompts or fine-tuning (intro cites Jain et al., 2024; Arditi et al., 2024; Zou et al., 2023; Greenblatt et al., 2024; Che et al., 2024; Deeb & Roger, 2025; Betley et al., 2025).
   • Existing gap: Deeb & Roger (2025) report that fine-tuning on a subset of deleted data can re-elicit "forgotten" information. The authors argue that relearning generalizes because fine-tuning dismantles a single shared mechanism, thereby reversing the entire post-training modification.
   • Hypothesis: The authors say current forgetting methods introduce context-dependent inhibitors. If the full set shares only one inhibitor, an attacker only needs to relearn a subset.
   • Approach: Propose Layered Unlearning (LU). Data is split into k disjoint folds; stage i forgets the union of the first i folds and retains the later folds, inducing inhibitors with different coverage. The motivation comes from Reason's (1990) Swiss cheese defense in depth.
   • Threat model:
     • Goal: Fine-tune on a proper subset of the forgetting set to recover performance on the remaining forgotten folds. The paper calls this adversarial relearning.
     • Capability: The attacker holds MCQ or corresponding corpus data for subset S and fine-tunes on it. The paper does not state white-box, black-box, or no-box.
     • Victim system: A model that has undergone unlearning. LLM experiments consistently use Zephyr-7B-β; forgetting sets come from subsets of WMDP, Years, or MMLU.
     • Judgment: Following Deeb & Roger (2025), after fine-tuning on S, report the highest accuracy across epochs on MCQ for T = F minus S.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into LLM unlearning, adversarial relearning, and sequential unlearning; this paper targets relearning robustness with retention-reducing multi-stage unlearning.

   Section 4 of the paper discusses three groups, with experimental baselines listed separately in Section 3.

   LLM Unlearning

   • Approximate strategies: The paper states that exact unlearning is difficult, and existing work is mostly approximate, including model optimization (Ilharco et al., 2022; Liu et al., 2022; Yao et al., 2023; Eldan & Russinovich, 2023; Jia et al., 2024; Zhang et al., 2024b; Li et al., 2024b) and prompt- or context-based methods (Thaker et al., 2024; Pawelczyk et al., 2023; Liu et al., 2024a).
   • Residue can be re-elicited: The paper states that these models remain vulnerable to adversarial attacks (Schwarzschild et al., 2024; Patil et al., 2024; Lynch et al., 2024), or can be relearned with a small amount of seen data (Hu et al., 2024; Lynch et al., 2024).

   Adversarial Relearning

   • Che et al. (2024): Fine-tuning on a small subset of the unlearned data to recover information from the full set. The paper states that they point out that most existing unlearning methods are vulnerable to such attacks.
   • Deeb & Roger (2025): Informationally distinct examples can also induce relearning; failure is not limited to rote memorization. The evaluation protocol in this paper comes from this work.
   • Proposed defenses: Rosati et al. (2024), Zou et al. (2024), Tamirisa et al. (2025), Sheshadri et al. (2025). The paper states (citing Che et al., 2024) that none of these defenses consistently blocks adversarial relearning. Prior work has used corpus fine-tuning (Che et al., 2024) and MCQ fine-tuning (Deeb & Roger, 2025) separately; the paper states that to its knowledge there is no comprehensive comparison of the two, and that corpus fine-tuning is more natural and also more effective.

   Sequential Unlearning

   • Dou et al. (2025): Removing copyrighted information over time. The paper only describes it.
   • Zhao et al. (2024): Using sequential unlearning to improve unlearning efficiency. The paper states that they do not consider robustness to adversarial relearning.

   Baselines and Benchmarks

   • Baselines: RMU (Li et al., 2024a, representation engineering) and SimNPO (Fan et al., 2025, gradient ascent type). The appendix also includes RMU-Split, which projects each fold to a different random vector.
   • Benchmarks: WMDP, Years, MMLU; synthetic settings are two-dimensional Gaussian mixture classification and three-token bigram.

   The authors position this paper as: analyzing how unlearning order and structure affect robustness to adversarial relearning, and using structured unlearning to mitigate information leakage; this differs from sequential unlearning that only pursues unlearning efficiency.

   Collapse this answer

3. 03How does the paper address this problem?

   Layered Unlearning divides into k folds and unlearns in stages with expanding subsets, while retaining the remaining folds to induce different suppressors.

   Layered Unlearning (LU) does not change the internal loss of the unlearning primitive U, only the unlearning and retention sets at each stage, so that suppressors cover different folds.

   Unlearning Problem and Notation

   • Goal: Remove information of the forget set F from parameters θ, while maintaining performance on the retention set R. With unlimited compute, one would retrain from scratch on D minus F; U is a post-training approximation of this.
   • Interface: U maps θ, F, R, and hyperparameters γ to new parameters. The paper writes U : Θ × X × X × Γ → Θ.

   Layered Procedure

   • Partition: Uniformly randomly split F into k disjoint folds. For the same k and dataset, the partition remains fixed across unlearning and evaluation.
   • Update: At stage i, call U on the parameters from the previous stage, unlearning the accumulated first i folds, and retaining the not-yet-unlearned folds plus the original retention set. That is, \theta_i = U(\theta_{i-1}, \bigcup_{j=1}^{i} F_j, R \cup \bigcup_{j=i+1}^{k} F_j, \gamma_i). The meaning is that the suppression scope expands with stages, while the folds that must still be retained prevent a shared suppressor covering all folds.
   • Two-fold case: First unlearn A, retain B. The authors state this forces the model to activate an IA that only suppresses A. The second stage may activate an IB that only suppresses B, or an IAB that suppresses both A and B. Relearning B will turn off suppressors involving B, but IA remains, so A stays suppressed. Whether the barrier is one-way or two-way depends on the primitive. A three-fold illustration is in Figure 1.

   Primitives in Synthetic Tasks

   • Two-dimensional classification: Input is R^2. Class 1 is a Gaussian mixture, means drawn from U([-50, 50]^2), covariance dominated by σ^2 = 4, with perturbation of magnitude 0.1; class 0 is a uniform background on U([-60, 60]^2). The classifier is logistic regression on a 12×12 RBF grid, with parameter dimension 145 (including bias). Unlearning relabels points in F as class 0, retaining original labels for R. Optimizer is Adam.
   • Bigram: Vocabulary is a, b, r, sequence length 8, ε = 0.05. After a or b, the probability of r is 1−2ε; after r, the probability of a or b is each 1/2 − ε/2. The model is a one-layer, attention-only transformer with parameter dimension 4288. Unlearning flattens the next-token distribution for tokens in F to uniform 1/3, retaining original conditional distributions for R, using standard language modeling loss. Relearning restores the original conditional distribution for that subset.
   • Readouts: In classification, A and B are accuracy of labeling the corresponding sets as class 1; R is joint accuracy of labeling R as class 1 and background as class 0. In bigram, A and B are prediction performance on the corresponding bigrams; R uses total variation relative to uniform over a and b, lower is closer to the retention target.

   LLM Primitives and Success Criteria

   • Primitives: RMU for representation engineering, SimNPO for gradient ascent type. With LU, denoted L-RMU, L-SimNPO. The appendix's RMU-Split projects each fold to a different random vector, to separate LU from "separate random vectors". Procedure illustration is in appendix Figure 5.
   • Models and data: All LLM experiments use Zephyr-7B-β. Forget sets are WMDP (dangerous knowledge MCQ), Years (major events with years), or MMLU subsets. Retention set is MMLU; when unlearning MMLU, the retention set excludes the forgotten categories.
   • Filtering: Only models with retain accuracy drop no more than 10% are kept (appendix Table 6). Appendix G sets the unlearning threshold at 0.35, with the rationale that random guessing on four-choice is expected at 0.25, and gives this upper bound based on 735 questions per dataset, minimum fold of one quarter, and three standard deviations.
   • Relearning evaluation: For all 2^k−2 proper subsets S, fine-tune with Adam on S's MCQ or corresponding corpus, then evaluate 0-shot on the remaining folds' MCQ using the Language Model Evaluation Harness, taking the highest accuracy across epochs as the final forget accuracy. Appendix E: learning rate 10^{-6}, batch size 4, MCQ 8 epochs, corpus 5 epochs. Per-stage coefficients, learning rates, and fine-tuning layers are in appendix G.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors say that on synthetic tasks and Zephyr-7B-β, LU makes early folds harder to recover via relearning on later folds.

   Experimental Setup

   • Models: The only LLM experiment is Zephyr-7B-β. Synthetic experiments use RBF logistic regression and a one-layer attention-only transformer.
   • Data: WMDP is split into 2, 3, and 4 folds; MMLU and Years are split into 3 folds. Appendix C: MMLU forgetting classes are geography, culture, STEM, chemistry, business; retained classes are health, history, law, philosophy, social sciences. Appendix G states each dataset has 735 questions total. Table 1 uses 5 Gaussians per set, random assignment, 10 seeds; Table 2 uses 10 seeds.
   • Baselines: Single unlearning of the same primitive (U, RMU, SimNPO) and RMU-Split. LU variants are U-LU, L-RMU, L-SimNPO, L-RMU-Split. Appendix G states RMU on WMDP uses the official checkpoint from Li et al. (2024a); SimNPO is rerun because the online checkpoint underforgets on its folds.
   • Metrics: MCQ accuracy on folds not involved in relearning after relearning; retained-set accuracy; recovery rate in Appendix F. This ratio divides the rebound of "after relearning minus after forgetting" for two methods, counting forgetting accuracy below 0.25 as 0.25; the paper says lower means more resistant to relearning.
   • Evaluation: 0-shot, Language Model Evaluation Harness. The paper does not use LLM judges, nor does it report human agreement or statistical significance tests.
   • Repetition: Synthetic results are averaged over 10 seeds. The main LLM table does not report multi-seed repetition. Fine-tuning takes the highest validation accuracy across epochs.

   Main Results

   According to Table 3, Zephyr-7B-β, WMDP three folds, accuracy after MCQ fine-tuning:

   Swipe horizontally to read the table

   Method	Relearn C eval A	Relearn C eval B	Relearn A eval C	Relearn B eval A	Relearn A eval B
   RMU	0.43	0.39	0.45	0.41	0.41
   L-RMU	0.26	0.34	0.50	0.31	0.40
   SimNPO	0.52	0.48	0.50	0.54	0.47
   L-SimNPO	0.42	0.45	0.41	0.42	0.41

   • The authors say robustness depends on fold order: an attacker holding fold A can recover more information about B and C than an attacker holding only fold C (Section 3.1). The Figure 3 caption says below the diagonal is lower than above, and calls L-RMU a one-way barrier. In the table, when relearning early folds and evaluating late folds, L-RMU is not always lower than RMU.
   • The authors say SimNPO with LU is also more resistant to relearning, but the improvement is weaker and the barrier is more symmetric (Section 3.1).
   • The starting points are not the same. In Table 7, after forgetting and before relearning, WMDP three folds L-RMU is 0.15 on A and RMU is 0.27. Appendix F normalizes this difference using recovery rate.

   Synthetic Tasks

   • Two-dimensional classification (Table 1): After standard U relearns B, accuracy on A is 0.93 (original model 1.00). The authors say this recovers 93% of task A's original performance. After U-LU relearns B, A is 0.30; the authors say this is only 0.29 above the 0.01 after forgetting. After relearning A in reverse, U-LU's B is 0.96. On the retained set, U-LU after relearning B is 0.86 and U is 0.80.
   • Bigram (Table 2): After U relearns B, A is 0.76, and LU is 0.51 (original model 0.91; after forgetting U is 0.33, LU is 0.34). The authors say U improves the other task by 0.43 on average after relearning A or B, while LU improves it by only 0.17 on average, with no transfer in either direction. The main text also says relearning B recovers 73% of task A's performance, which is not the same formulation as the 0.76 in the table; here the table prevails.
   • Authors' interpretation: The Figure 2 caption says in the standard pipeline, relearning B generalizes near A; after LU, B still rises but no longer generalizes to A. The main text says the greater the task overlap, the stronger the resistance to relearning; clustering Gaussian means and assigning whole clusters reduces resistance to relearning across algorithms. The bar heights in Figures 6–7 are not given numerically.

   Corpus Fine-tuning and Other Datasets

   • Corpus vs. MCQ (Section 3.2, Table 3): The authors say corpus fine-tuning is stronger. They say RMU rises by 5% on average under corpus fine-tuning; L-RMU rises by 10% on average when relearning late folds and evaluating early folds, and 6% in the reverse direction. In the table, L-RMU relearning C and evaluating A is 0.26 for MCQ and 0.36 for corpus; RMU in the same cell is 0.43 and 0.50. SimNPO and L-SimNPO are less affected by corpus: L-SimNPO relearning A and evaluating B has corpus accuracy 0.35, lower than MCQ's 0.41.
   • Years and MMLU: The authors say Appendix E has similar results. Years three folds, relearning C and evaluating A, MCQ: RMU 0.54, L-RMU 0.29 (Table 12). MMLU three folds, same cell: RMU 0.53, L-RMU 0.33 (Table 11). On MMLU, L-RMU can be higher when relearning early folds and evaluating late folds: relearning B and evaluating C has MCQ 0.59, RMU 0.51.
   • Retained set: In Table 6, full MMLU without forgetting is 0.58. On WMDP three folds, RMU, L-RMU, and SimNPO all have retain 0.57, and L-SimNPO has 0.55, all within the filter of no more than 10% drop.

   Other Ablations and Analyses

   • RMU-Split (Table 9): The paper says results do not change compared to the shared random vector. WMDP three folds, relearning C and evaluating A, MCQ: RMU-Split 0.40, L-RMU-Split 0.25.
   • Recovery rate (Table 14, WMDP three folds MCQ): L-RMU vs. RMU, relearning C and evaluating A is 0.06, relearning A and evaluating C is 1.33.
   • Bigram components (Appendix Table 4): When relearning B, using all U gives A of 0.75; swapping in only LU's QK and OV gives 0.52; all LU gives 0.51. The paper says QK or OV is most critical for resistance to relearning. A zero-layer, embedding-only transformer has almost no resistance to relearning.
   • Number of folds (Table 8): WMDP two folds, relearning B and evaluating A, MCQ: RMU 0.45, L-RMU 0.30. Four-fold results are in Table 10.
   • Post-forgetting starting point (Table 7, WMDP three folds): L-RMU is A 0.15, B 0.24, C 0.33; RMU is 0.27, 0.24, 0.33; L-SimNPO is 0.29, 0.32, 0.33.
   • Figure 5 caption: When RMU forgets on WMDP in the order A, B, C, A stays low after the first segment drops, while B and C drop only in the second and third segments, respectively. The text does not give curve values.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out hyperparameter sensitivity, fold count scaling, and computational cost, and leave comprehensive adversarial evaluation and fine-tuning on unrelated data for future work.

   Limitations and Future Directions Noted by the Authors

   • Hyperparameters: LU is sensitive to hyperparameters, and the balance between forgetting and retention must be tuned at each stage. The authors say this reflects the difficulty of separating forgotten representations from retained ones (Section 5).
   • Number of folds: As folds increase, the model must distinguish each pair of folds. The authors say complexity scales as (2k), limiting large k (Section 5).
   • Compute: LU is more computationally expensive. The authors say in exchange, it finds optima that standard forgetting cannot find no matter how long it trains, and suggest lower-cost methods for future work (Section 5).
   • Evaluation combinations: There are 2^k−2 relearning subsets, plus batch size, learning rate, dataset, and forgetting method, making comprehensive adversarial evaluation difficult. The authors leave full analysis to future work and say fixed hyperparameters are already enough to break RMU and SimNPO (Section 5, appendix).
   • Unrelated data: The authors state they do not directly address harmless fine-tuning, where the attacker uses data unrelated to the forgotten content, and only provide intuition for future work (Section 5).
   • Mechanism and generalization: How attention provides robustness in the bigram is left to future work (Appendix B.2). The authors also suggest the layered approach could generalize to alignment: first train to be harmless but unhelpful, then become helpful while remaining harmless (Section 5). The conclusion says LU can serve as a testbed for studying inhibitor mechanisms.

   Experimental Coverage

   • Models: The LLM experiment uses only one model, Zephyr-7B-β (Section 3).
   • Primitives and data: Forgetting primitives are RMU, SimNPO, and RMU-Split; datasets are WMDP (2/3/4 folds), MMLU (3 folds), and Years (3 folds) (Section 3, Appendix E).
   • Attack forms: Adversarial evaluation is MCQ fine-tuning (8 epochs) and corpus fine-tuning (5 epochs) on subsets of the forgetting set, with learning rate 10^{-6} and batch size 4 (Appendix E). The introduction mentions prompt engineering can bypass post-training, and the paper does not report numbers under that attack.
   • Filtering and seeds: Only models with retain drop no more than 10% are included (Section 3). Synthetic experiments report averages over 10 random seeds (Table 1, Table 2); the main LLM results do not report repetitions.
   • Appendix comparisons: Appendices D–F also report relearning accuracy for RMU-Split, 2/4 folds, MMLU and Years, and recovery rate. The paper does not report statistical significance tests.

   Collapse this answer

6. 06Summarize the main content of the paper

   LU uses multi-stage retention and unlearning to construct different suppressors; the authors state it is more resistant to MCQ relearning, but corpus fine-tuning is stronger.

   LU is a k-fold sequential unlearning method designed to mitigate "fine-tuning on a subset of the forgotten data can recover the remaining folds."

   • Problem: The authors argue that existing forgetting mostly installs a single context-dependent shared inhibitor, and one relearning session dismantles the entire modification.
   • Method: Stage i forgets the union of the first i folds while retaining the not-yet-forgotten folds, leaving inhibitors with different coverage. The primitive itself is unchanged. LLM experiments use RMU and SimNPO with the model Zephyr-7B-β; synthetic experiments use two-dimensional classification and a one-layer bigram transformer.
   • Synthetic: With 5 Gaussians per set and 10 seeds, standard U relearning B gives A of 0.93, and U-LU gives 0.30 (Table 1). On bigram, LU relearning B gives A of 0.51, and U gives 0.76 (Table 2).
   • LLM: On WMDP three folds, MCQ, L-RMU relearning C and evaluating A is 0.26, RMU is 0.43; in reverse, relearning A and evaluating C, L-RMU is 0.50, not lower than RMU's 0.45 (Table 3). The authors say the barrier is one-way for RMU and more symmetric and weaker for SimNPO.
   • Corpus attack: The authors say corpus fine-tuning is stronger. On WMDP, it raises RMU by 5% on average and L-RMU by 10% on average when relearning late folds and evaluating early folds (Section 3.2).
   • Conclusion: The authors say LU improves resistance to standard MCQ relearning while exposing the limits of existing methods under corpus attacks; forgetting itself is fragile, and robustness requires layered defense. They also say the results advance the state-of-the-art in machine unlearning and help understand post-training updates.

   Collapse this answer

Related papers