Not All Changes Are Harmful: Update-Matched Unlearning for Large Language Models

1. 01What problem does this paper attempt to solve?

   Subsequent fine-tuning can restore deleted knowledge, and low relearning scores cannot distinguish selective resistance from plasticity loss.

   Subsequent fine-tuning can restore deleted knowledge, and low relearning cannot distinguish selective resistance from loss of plasticity.

   • Scenario: The authors believe LLMs memorize private or harmful useful knowledge, and unlearning should delete it while preserving useful capabilities. Models that are still fine-tuned after release must also be able to learn new tasks; parameter changes may also make deleted knowledge usable again, even if fine-tuning does not target it. The paper gives an example: fine-tuning on similar questions from other authors once restored target names that were no longer output.
   • Existing shortcomings: TAR and AntiDote are trained against simulated updates, but capability objectives are evaluated on models without simulated modifications, checking current behavior rather than post-update plasticity. The authors argue that a model that responds weakly to all updates can pass this check while exhibiting low relearning.
   • Hypothesis: Matching the retention set outputs before the update would treat changes caused by ordinary learning as deviations. If the reference accepts the same update, that change is no longer counted as a difference. The authors propose the reference-timing hypothesis: under similar target suppression, an update-matched reference preserves plasticity better than a pre-update reference.
   • Method: UMU applies the same simulated relearning update to the defended model and to a reference initialized from the same unlearned model, and uses new-task trials to select checkpoints. It can be stacked on existing unlearning methods and has no runtime components at release.
   • Threat model:
     • Goal: Prevent subsequent fine-tuning from restoring deleted knowledge while preserving the ability to learn new tasks.
     • Capability: The paper does not state white-box or black-box. Evaluation retrains the released model: Harmful Attack uses p=0.1 or 0.3 of the forget set, Benign Attack uses instructions without target data. Non-targeted fine-tuning can also cause restoration.
     • Victim system: A single model after merging EraseLoRA; teacher and TaskLoRA are discarded before release.
     • Judgment: Relearning takes the maximum FS during retraining; plasticity takes NS after one epoch on AGNews.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into forgetting evaluation and re-learning-resistant training; the authors place the difference at the reference time.

   In Related Work, the paper divides into unlearning objectives and evaluation, and robust training against subsequent fine-tuning, and points out differences in retention objectives and sources of relearning updates.

   Unlearning objectives and evaluation

   • Three types of deletion: Output layer uses gradient ascent or negative preference optimization (Jang et al., 2023; Chen and Yang, 2023; Yao et al., 2024; Zhang et al., 2024; Fan et al., 2025b; Jia et al., 2024). Representation layer perturbs activations of target data (Li et al., 2024; 2026; Hong et al., 2024). Parameter layer locates or regularizes neurons attributed to the target (Yang et al., 2025a;b).
   • Evaluation and restoration: Immediate evaluation looks at unlearning and utility (Maini et al., 2024, etc.), plus memory and privacy leakage, knowledge extraction, and fine-tuning on other target facts to restore withheld facts. Non-targeted fine-tuning can also trigger restoration (Lucki et al., 2025; Hu et al., 2025), including questions with different fictional authors but shared target syntax (Yoon et al., 2026).
   • Position papers: Thaker et al. (2025) and Shumailov et al. (2024) argue that existing benchmarks underestimate such restoration, and that unlearning alone cannot constrain what a released model can be made to output. The paper accordingly looks at both unlearning under continued fine-tuning and the ability to learn new tasks.

   Training against relearning

   • Simulated updates and harmful fine-tuning: Task blocking was proposed earlier to limit harmful adaptation while preserving useful behavior (Henderson et al., 2023; Deng et al., 2024). Simulated defenses train against explicit fine-tuning trajectories (Tamirisa et al., 2025; Sondej et al., 2025) or generated low-rank perturbations (Sanyal et al., 2026). There are also defenses that perturb or regularize the state before release (Huang et al., 2024a;b; 2025; Rosati et al., 2024), motivated by the fact that a small amount of fine-tuning can undo safety training (Qi et al., 2024; Lermen et al., 2023).
   • Regularization and retention rules: Invariance regularization treats downstream tasks as environments where unlearning should be stable (Wang et al., 2025a); sharpness-aware objectives flatten the loss near the unlearned state (Fan et al., 2025a); distillation or confusion losses strengthen unlearning itself (Xu et al., 2025a; Wang et al., 2025c). TAR and AntiDote's language modeling and reference losses do not include simulated modifications; Sondej et al. (2025) filter unlearning gradients by retention gradients. ILU reports downstream task accuracy separately from static utility and unlearning robustness. The paper notes that these tests distinguish preserving existing performance from preserving the ability to learn new tasks.
   • Differences the paper names: TAR and AntiDote's capability objectives do not check post-update plasticity. Existing defenses' relearning updates come from a separate adversary or auxiliary network; UMU records updates from an ordinary learner relearning the target on the defended model itself.

   Baselines and benchmarks

   • Baseline methods: GD (Maini et al., 2024), RMU (Li et al., 2024), KLUE (Yang et al., 2025b). UMU is added on top of the checkpoints after these methods' unlearning is complete, and the unlearning loss follows the base method.
   • Benchmarks: TOFU's Forget01; FaithUn's 5% forget. New task uses AGNews, benign relearning uses Alpaca.

   The authors position the paper as: comparing student and reference under the same recorded relearning update, and using short new-task learning trials to select candidates; the object of study is unlearning under continued fine-tuning, and the ability to learn new tasks.

   Collapse this answer

3. 03How does the paper address this problem?

   UMU replays the same relearning update to student and teacher, and uses new-task trials to select the released model.

   Update-Matched Unlearning (UMU) does not aim to resist all updates, but rather keeps deleted knowledge suppressed under subsequent updates while allowing learning from subsequent fine-tuning. It consists of update-matched replay and checkpoint selection by learning trials, and the release has no runtime components.

   Problem setup and reference timing

   • State: θ0 is the model after base unlearning, the defended model is θφ=θ0+Eφ, with starting point Eφ=0. An update Δ is relearning if it restores deleted knowledge; if it can still reduce the new-task loss Ltask, it preserves plasticity.
   • Objective: With Δ fixed, the defense solves min_ϕ L_F(θ_ϕ+Δ)+λ_R L_R(θ_ϕ+Δ). Both terms are computed on the same post-update model. L_F is the base unlearning loss; L_R is the KL between the post-update model and a detached reference distribution on retention-set answer tokens; λR>0.
   • Two references: The pre-update reference is θ0, with loss denoted Lold; the update-matched reference is θ0+Δ, with loss denoted Lmatch. They share φ, Δ, and retention tokens, differing only in whether they penalize prediction changes caused by the update itself.
   • Gradient: Appendix A states that the pre-update gradient equals the update-matched gradient plus a term driven entirely by Δ's changes on retention predictions. That term is zero when the update does not change retention predictions; otherwise it pulls predictions back toward the pre-update state. At φ0, where the defense has not yet modified the model, the update-matched gradient is zero. The authors believe the retention set does not contain target knowledge, so this term corresponds to ordinary learning rather than target restoration.
   • One-step direction: For a smooth task loss, in the first-order change of the task loss from a retention step, λR appears only in the step size; the increase or decrease is determined by the sign of the cosine c between the retention gradient and the task gradient. When c<0, any sufficiently small retention step increases the task loss, and reducing λR only makes each step smaller. Figure 2 compares the two references on GD, Llama-3.2-3B-Instruct, TOFU: without defense, the pre-update retention KL rises over ten steps of ordinary learning, while the update-matched one is zero by construction; over 21 states in 100 steps of defense, the authors say cold remains negative and cmatch tends to zero; with one SGD step on the retention loss only, the authors say pre-update increases the task loss, while update-matched is almost unchanged. At step 0, the update-matched cosine is undefined because the gradient is exactly zero.

   Update-matched replay

   • Where updates come from: Separate fine-tuning, even with the same data and seed, follows different trajectories, and the difference would be attributed to the defense; a pre-fixed perturbation is identical on both sides but is not the update the model actually experiences. Each round, fine-tune on target answers on a copy of the current model, recording weight changes Δk after several steps k∈K. The authors cite Tamirisa et al. (2025): a defense that only resists early steps of an attack can be undone by later steps.
   • Replay: The same Δk is added to both sides; the student becomes θφ+Δk, and the frozen reference becomes θ0+Δk. The training objective is L_UMU(ϕ)=avg_{k∈ K}(L_F(θ_ϕ+Δ_k)+λ_R L_match(θ_ϕ+Δ_k)). Terms in the base objective other than the unlearning loss remain unchanged on the model without that update. In Figure 3, the unlearning losses on both sides are the same; the only difference is whether the reference is at θ0 or θ0+Δk.
   • Role of KL: The retention term is the usual KL on output distributions, with the teacher as reference and the student as the defended model; the only change is that the teacher receives the same update. The authors give two points: changes on the retention set from the update are present on both sides, so at φ0, Lmatch and its gradient are zero, penalizing only changes the defense adds; the comparison covers only the retention set, and if the post-update reference restores target answers, those answers are still constrained by the unlearning loss.
   • Stackable: L_F follows the base method, e.g., GD's gradient ascent term or RMU's representation loss. Thus it can be added on top of existing checkpoints where the unlearning loss can be computed on the post-update model.

   Selection by learning trials

   • Motivation: Low restoration or consistency with the reference does not indicate that a checkpoint can still learn; looking only at restoration biases toward candidates that have already stopped learning.
   • Rule: From a pre-declared pool of late candidates, each candidate is briefly fine-tuned on an ordinary new task. A learning trial must show sufficient progress, and unlearning, retention, and existing task performance must pass development-set checks. Training and selection data are internal and disjoint from all evaluation sets.
   • Release: Only the selected candidate's defense weights are merged into the base unlearned checkpoint; the reference and learner are discarded. Algorithm 1 in Appendix B gives the procedure.

   Instantiation and configuration

   • LoRA: The teacher is the frozen base unlearned checkpoint; the student adds trainable EraseLoRA, merged at release. A temporary TaskLoRA is trained with AdamW on a student copy and discarded before release along with the teacher. The authors' rationale: both sides share frozen weights, so TaskLoRA's weight changes are the recorded update and can be added to a same-architecture teacher; LoRA is the lowest-cost way to adapt a released model and can undo safety training.
   • Differentiation: The recorded update and teacher are treated as constants during differentiation; gradients go only to EraseLoRA, not through the learner trajectory, so this is not second-order meta-learning. Each round is one short rehearsal, plus one teacher forward and one student forward per recorded step.
   • Values in Appendix C: Both LoRAs have rank 8, applied to every layer's query, key, value, and output projections, with scaling 16/8. TaskLoRA runs 20 steps per round, learning rate 1e-4, batch size 8; K is 5, 10, 20, with weights each 1/3. The outer loop runs 100 steps with learning rate 1e-4. The main text writes λR in equation (4); Appendix C gives λR=1 and lists λS=1, though the main text's equations do not define λS. Candidates are the last 5 outer-loop checkpoints; learning trials run 10 steps on a selected subset of AGNews. Qualification criteria: development-set FS no more than 1 point above the base checkpoint, development-set RS and US within 1 point of the base, and learning progress at least 90% of the pre-unlearning model. These values are shared across all bases, benchmarks, and backbones. After merging in bfloat16, FS, RS, and US differ by at most 0.1 from when the adapter is attached, and 20 steps of AGNews fine-tuning can reduce training loss.

   Collapse this answer

4. 04What experiments did the paper conduct?

   In twelve settings, UMU simultaneously reduces both types of relearning and improves new task scores; removing the synchronous reference lowers new task scores.

   Experimental Setup

   • Models and Bases: The backbones are instruction-tuned versions of Llama-3.2-3B-Instruct and Qwen3-4B. UMU is added on checkpoints after GD, RMU, and KLUE unlearning, with and without UMU sharing the same starting point. Default is the model before unlearning.
   • Data and Stopping Points: TOFU's author profiles are first injected and then deleted; Forget01 has 40 forget and 3960 retain entries; injection runs for 5 epochs with a learning rate of 1e-5. FaithUn's real-person facts come from pretraining, using 5% forget. Base unlearning stops at a threshold: FaithUn accuracy 0.33, TOFU's ROUGE-L recall 0.1971, which is the average recall of randomly mismatched question-answer pairs. The learning rate is the smallest among 1e-5, 2e-5, 5e-5 that reaches the threshold.
   • Metrics: FS is on the forget set, ROUGE-L recall for TOFU and multiple-choice accuracy for FaithUn; RS is on the retain set; US is on general benchmarks not used by UMU. Harmful Attack is the maximum FS reached when retraining on a proportion p of the forget set of 0.1 or 0.3; Benign Attack is the maximum FS when retraining on 1000 Alpaca entries that do not contain target data. NS is the test accuracy after fine-tuning on AGNews for one epoch, disjoint from the selected subset. Lower is better for FS and both attack scores; higher is better for RS, US, and NS. The paper does not use LLM evaluation.
   • Attack Protocol: Harmful attacks retrain on a proportion p drawn from the eligible pool, and FS is measured on the remaining part. TOFU's eligible pool is all 40 questions, 20 for each of the two fictional authors, so retraining uses 4 or 12 questions and evaluation uses 36 or 28. FaithUn's eligible pool is the forget questions that the published model answered incorrectly. AdamW trains for 5 epochs with learning rates 1e-5, 5e-6, 1e-6, reporting the maximum FS over the epoch and learning rate grid. Benign attacks use learning rates 1e-5, 1e-6, 1e-7. Each configuration is repeated 3 times with different data orders and averaged. Evaluation attacks are re-optimized against the published model and do not reuse the TaskLoRA from training.
   • New Task and Utility: AGNews has 120000 training and 7600 test entries. 6000 training entries are held out, 3000 for selection and 3000 for the diagnostics in Section 3; the remaining 114000 entries, with an effective batch size of 128, give 891 updates per epoch. NS is the accuracy on all 7600 test entries at the last step. Figure 4 is measured at 24 checkpoints, starting at 0, 1, 2, 5, 10, 20, 50, and then every 50 steps. FaithUn's US is the average accuracy on 500 entries each from MMLU, GSM8K, HellaSwag, ARC-Challenge, and WinoGrande, with the same samples for each model; TOFU's US is the average ROUGE-L recall on Real Authors and World Facts.
   • Hardware: All experiments run on a single NVIDIA GeForce RTX 5090. Appendix C gives RMU's layers and coefficients and KLUE's neuron proportions; UMU hyperparameters are shared across all configurations.

   Main Results

   The table below follows Table 1; the value before the arrow is the base, and after the arrow is with UMU. Only 7 settings are listed, including the TOFU Qwen RMU with the highest remaining p=0.1 harmful attack score. TOFU Qwen's GD and KLUE, and FaithUn's KLUE for Llama and Qwen, are omitted. p=0.3 is in Table 1.

   Swipe horizontally to read the table

   Setting	Harmful p=0.1	Benign	NS
   TOFU Llama GD	45.41→24.98	20.98→19.08	82.03→85.16
   TOFU Llama RMU	87.97→52.34	20.33→15.35	79.69→83.98
   TOFU Llama KLUE	43.71→24.04	22.19→18.95	81.25→84.38
   TOFU Qwen RMU	89.07→53.00	20.89→15.87	82.03→85.55
   FaithUn Llama GD	48.13→11.11	33.33→9.88	80.86→84.77
   FaithUn Llama RMU	52.63→15.79	14.29→5.26	78.12→83.20
   FaithUn Qwen GD	26.00→5.56	21.83→3.70	83.59→86.72

   • Unlearning and Retention: The authors state that UMU maintains unlearning and repairs the retention damaged by base unlearning. On TOFU, FS decreases slightly for each base, such as GD on Llama from 19.32 to 18.45; on FaithUn, FS stays at 0.0 for all methods. RS rises where the base pulled it down, such as GD on FaithUn from 81.03 to 87.99 (Llama) and 69.35 to 79.29 (Qwen3); the authors state that on TOFU, KLUE has the largest RS gain, with Llama rising from 86.86 to 92.54. For RMU on TOFU in Table 1, RS goes from 93.70 to 93.56 (Llama) and 94.43 to 94.30 (Qwen), not rising. All six TOFU US values are lower than the corresponding base, and the authors do not separately explain US.
   • Relearning: The authors state that in all twelve settings, both attack scores drop together, so base unlearning suppresses rather than deletes; UMU trains against updates that would re-expose the target. Section 5.2 states the average drops are 9.9 for benign and 28.8 for p=0.1 harmful; the abstract states harmful and benign drop by 30.4 and 9.9 on average, without noting which p corresponds to 30.4. The authors state that harmful attacks are the threat directly targeted by training, and the benign drop shows resistance extends to updates not rehearsed. The highest remaining p=0.1 harmful score is on TOFU's RMU+UMU.
   • Plasticity: All three bases learn AGNews worse than the pre-unlearning model; adding UMU brings them close to that model. The authors state that on Llama, RMU drops the most, 7.0 on TOFU and 8.6 on FaithUn, and also recovers the most on FaithUn, at 5.1. The authors state that GD+UMU's 85.16 (TOFU Llama) and 86.72 (FaithUn Qwen3) are within 1.6 and 1.2 of the pre-unlearning model, respectively. The authors attribute the cause to: if the reference does not receive updates, all changes on retained data count as errors; in UMU, changes from the update itself no longer count as errors. The authors also state that part of the NS rise comes from the learning trial, and the two parts are separated in Section 5.3.

   Ablation: Replay and Selection (Table 2)

   • What Was Tested: Only on GD, across four dataset and model settings, removing replay and removing the learning trial separately. When replay is removed, the teacher is taken before the replay update, i.e., a pre-update reference. All variants share the starting point and evaluation protocol.
   • Results: On TOFU Llama, removing replay gives a p=0.1 harmful score of 23.89, while full UMU gives 24.98; NS is 78.91, which the authors state is 6.3 lower than full, and also lower than GD without UMU (82.03). On FaithUn Qwen, removing replay gives an NS of 81.25, which the authors state is 5.5 lower than full, and state that in all four settings this NS is lower than GD without UMU. Removing selection gives attack scores within about 1 point of full; FaithUn Qwen's p=0.3 is 8.33, while full is 9.26. The authors state that its NS is 2.0 to 2.7 lower than full in every setting.
   • Authors' Interpretation: When replay is removed, the pre-update reference treats all post-update changes as deviations, and the cost falls on NS; attack scores do not show this because a model that cannot learn also cannot recover. Looking only at recovery drifts toward candidates that are least trainable. The authors consider both directions to support the reference-timing hypothesis.

   Learning and Recovery on the Same Trajectory (Figure 4, Table 3)

   • What Was Tested: The published Llama-3.2-3B-Instruct, comparing GD, UMU without replay, and full UMU, fine-tuned on AGNews for one epoch (891 updates), while recording FS on TOFU Forget01.
   • Results: The authors state that the three models start within 1 point of each other, at 61.32, 61.76, and 60.93. UMU trails GD at step 20, surpasses it at step 50, and ends at 85.16, while GD ends at 82.03; the no-replay variant trails GD at step 50 and ends at 78.91. The authors state that GD's FS rises in nine of ten intervals, from 19.32 to 22.18; they also state that UMU rises only 0.88 (18.45 to 19.33), and the no-replay variant rises 0.32. Table 3 gives absolute FS at 11 proportion points within a one-epoch budget; GD's endpoint is also 22.18, but the 30%, 60%, and 90% points are lower than the previous column. The three UMU variants end between 18.93 and 19.33, each within 1 point of its start.
   • Authors' Interpretation: The authors state that the difference is in how fast learning happens, because a retention reference that penalizes the update itself also slows updates that acquire the new task. The authors state that the flattest curve in Figure 4(b) belongs to the lowest curve in (a), and that UMU combines the steepest learning curve with an almost equally flat recovery curve, and state that this selectivity comes from replay.

   Other Ablations and Analyses

   • Appendix E: On FaithUn, base unlearning lowers GD's RS by 8.2 (Llama) and 11.7 (Qwen3). The authors offer one interpretation for UMU recovering this part and state that the teacher's RS under replayed updates was not measured, so the interpretation is unverified.
   • Appendix E: On TOFU, forgotten and retained authors use the same profile template for injection. The authors use this to explain KLUE's lowest RS and UMU's largest RS gain. The authors believe that on Qwen3, the smaller NS loss after base unlearning is because a larger and deeper network leaves more capacity untouched by the same update.
   • Appendix F gives generation examples for GD and GD+UMU under p=0.3 harmful attacks. The authors state that before the attack, both answer fluently and replace the target entity; after the attack, GD recovers the label content, while GD+UMU still gives the replacement answer.
   • Figure 2's axes are optimizer steps, along with Retain KL, Gradient cosine, and Task ΔCE. The text gives rises and falls and signs, but not endpoint values for the curves. The diagnostic setting is GD, Llama-3.2-3B-Instruct, TOFU.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that forgetting quality inherits the base method, and both rehearsal and attack use low-rank adaptation.

   Limitations and future directions noted by the authors

   • Inherited unlearning quality: UMU inherits the unlearning quality of the underlying unlearning method (Section 6).
   • Low-rank adaptation: The rehearsal procedure and the evaluated attacks use low-rank adaptation (Section 6).
   • Not reusable across checkpoints: Each base unlearning checkpoint currently requires its own rehearsal (Section 6).
   • Future plans: The authors plan to study full-parameter and adaptive fine-tuning attacks, and whether rehearsal-derived updates can transfer or be reused across checkpoints (Section 6).

   Experimental coverage

   • Main experiments: Backbones are Llama-3.2-3B-Instruct and Qwen3-4B; benchmarks are TOFU Forget01 and FaithUn's 5% forget; bases are GD, RMU, KLUE, for a total of twelve settings (Sections 5.1–5.2, Table 1).
   • Attack protocol: Harmful retraining uses p=0.1 and 0.3 of the forget set, benign retraining uses 1000 Alpaca examples; each attack configuration is repeated 3 times and averaged (Appendix D). The paper does not report standard deviations or statistical tests.
   • Ablations and trajectories: Ablations of replay and learning trials are on GD, across four dataset and model settings (Table 2). The same trajectory is for Llama-3.2-3B-Instruct, the GD family, AGNews, and TOFU Forget01 (Figure 4, Table 3). Figure 2's diagnosis is for GD, Llama-3.2-3B-Instruct, TOFU.
   • Adaptation form: The conclusions state that rehearsal and the evaluated attacks use low-rank adaptation; Appendix D writes that evaluation retraining uses AdamW, without specifying whether the evaluated attacks use LoRA. In the instantiation, EraseLoRA and TaskLoRA are LoRA (Section 4.3, Appendix C).
   • Comparison scope: Experimental baselines are GD, RMU, KLUE (Table 1). Related work discusses TAR, AntiDote, and ILU; the paper does not report comparative experiments with these three. The utility set is 500 examples each from five general benchmarks, plus TOFU's Real Authors and World Facts (Appendix D). Appendix E states that the teacher's RS under replayed updates was not measured.

   Collapse this answer

6. 06Summarize the main content of the paper

   UMU uses a synchronously updated reference to separately relearn resistance and plasticity, and the authors state that both improve simultaneously across twelve settings.

   UMU is a training procedure added on top of existing forgetting methods, making models that will still be fine-tuned after release resistant to re-learning while still able to learn new tasks.

   • Problem: Subsequent fine-tuning can restore deleted knowledge, even when the fine-tuning does not target that knowledge. Looking only at low re-learning cannot distinguish selective resistance from loss of plasticity. A pre-update reference treats useful changes brought by the update itself as deviations. The authors propose the reference-timing hypothesis: under similar target suppression, a reference synchronized with the update better preserves plasticity.
   • Approach: On a student copy, perform short re-learning on target answers, record weight changes over several steps, and replay the same update on a frozen teacher and student. Only EraseLoRA is trained, so that the post-update forgetting loss decreases and the post-update teacher is matched on the retention set. Late checkpoints are selected after short new-task trials and merged; the teacher and TaskLoRA are discarded, and the released artifact has no runtime components.
   • Main results: Across the twelve settings in Table 1, relative to GD, RMU, and KLUE, harmful and benign attack scores decrease and NS increases. The authors report that at p=0.1, harmful attack score decreases by 28.8 on average, benign attack score by 9.9 on average, and NS increases by 3.5 on average; the abstract writes the average harmful decrease as 30.4. On TOFU, RMU+UMU still has p=0.1 harmful scores of 52.34 (Llama) and 53.00 (Qwen). On FaithUn, all methods keep FS at 0.0.
   • Comparisons: Removing replay still keeps attack scores low, sometimes even lower than full UMU, but NS is lower than GD without UMU (Table 2). In Figure 4, the authors state full UMU has the steepest AGNews learning curve, while FS only rises from 18.45 to 19.33.
   • Authors' conclusion: Robust forgetting should prevent deleted knowledge from returning without making the model resistant to learning itself. The authors note the method inherits the base model's forgetting quality, rehearsal and the evaluated attacks use low-rank adaptation, and each checkpoint needs its own rehearsal.

   Collapse this answer

Related papers