What should LLMs forget: Knowledge-association reinforcement learning unlearning
What Should LLMs Forget: Reinforcement Unlearning of Knowledge Associations
The authors propose KARU, a reinforcement forgetting framework that takes knowledge association (KA) as the forgetting target, and claim that on RWKU its forgetting score is relatively reduced by up to 30.5% compared with RULE.
- LLM forgetting must delete specific knowledge while retaining general capability, but the authors argue that existing methods optimize uniformly over the entire response and do not clearly specify the knowledge to forget, resulting in coarse forgetting and possible residue in the predictive distribution.
- The authors redefine the unlearning target as knowledge association (KA), i.e., the conditional predictive distribution between the query context and the target knowledge, and propose the RL framework KARU: token-level reward KATR, distribution-level Logits Flattening, and proximity-weighted Adaptive Neighborhood Sampling.
- On RWKU (Llama3-8B), the authors report that KARU's aggregate forgetting score is relatively reduced by 21.8% (PPO) and 30.5% (GRPO) compared with RULE, with retention and general capability comparable; the trend is consistent on 3B/7B models and on MUSE (ICLM-7B).
- The authors point out that the framework relies on NER to extract knowledge tokens and adopts a two-stage training pipeline, with end-to-end unlearning being a future direction. The experiments cover two benchmarks, RWKU and MUSE, 4 models, and two algorithms, PPO/GRPO; the paper does not report the number of training repetitions.
- Evaluated models
- Llama3-8B-InstructLlama3.2-3B-InstructQwen2.5-7B-InstructICLM-7B
- Benchmarks
- RWKUMUSEMMLUBBHAlpacaEval
- Metrics
- Forget score (FB/QA/AA, ROUGE-L)Retain score (邻域 FB/QA)GenReaFluNo Verb Mem.No Know Mem.Utility Preserv.
The researchers propose KARU, an LLM reinforcement learning unlearning framework that targets knowledge association (KA) as the unlearning objective. Existing methods usually optimize the entire response and do not explicitly locate the knowledge to be forgotten, resulting in coarse and uncontrollable unlearning; KARU combines token-level reward construction, distribution-level regularization, and proximity-aware retention sampling to finely suppress target knowledge and mitigate collateral damage. On multiple LLMs and representative benchmarks, its unlearning effectiveness consistently outperforms existing baselines while maintaining comparable retention performance and general capabilities.
Paper analysis
What problem does this paper attempt to solve?
The authors ask what LLM forgetting should actually forget, and propose taking knowledge association rather than the entire response as the forgetting target.
Core question: what exactly should an LLM forget during unlearning—the authors argue it should be the knowledge association between the query and the target knowledge, not the entire generated response.
- Problem scenario: LLM pretraining memorizes private, copyrighted, or sensitive information; unlearning aims to remove the influence of specific knowledge while preserving overall capability, and the GDPR's "right to be forgotten" is one motivation (according to the authors).
- Shortcomings of existing methods: the authors argue that existing RL unlearning methods treat all generated content alike, and the optimization signal is dispersed over irrelevant tokens, leading to imprecise unlearning; forgotten knowledge may still be encoded in the predictive distribution and can be easily extracted or reactivated; the retain objective weights samples uniformly, ignoring differences among samples in susceptibility to collateral degradation.
- Core observation: after unlearning, changes in the predictive distribution are concentrated on knowledge tokens (Figure 1 left); based on this, the authors propose that knowledge association (KA)—the conditional predictive distribution between the query context and the target knowledge—is the fundamental unlearning target.
- Paper's approach: propose KARU (Knowledge Association Reinforcement Unlearning), a unified RL framework that approximately optimizes the KA objective from three perspectives—gradient, distribution, and retention: token-level reward KATR, distribution-level Logits Flattening, and proximity-aware retain sampling ANS.
- Scope: the paper explicitly focuses on entity-centric factual knowledge unlearning scenarios (Sec. 3.1).
What related research exists?
Related work is divided into two groups: paradigms of LLM forgetting and LLM reinforcement learning; RULE is the SOTA baseline the authors position.
LLM unlearning methods
- Gradient-based: Jang et al. (2022) reverse-optimize the original training objective on forget samples; GA (Maini et al., 2024) performs gradient ascent on the forget set loss.
- Preference optimization-based: NPO (Zhang et al., 2024) and SimNPO (Fan et al., 2025) formulate unlearning as preference learning.
- Parameter/distribution-based: including task arithmetic (Ilharco et al., 2023), knowledge editing (Meng et al., 2022), and methods that explicitly optimize the predictive distribution (Entesari et al., 2025, etc.).
- Fine-grained objectives: TNPO (Wang et al., 2025) and TIF (Zhou et al., 2026) move toward token-level intervention; CEU (Entesari et al., 2025) points out the limitation of only suppressing the target token probability without considering the predictive distribution; PALU (Zhai et al., 2026) combines local intervention on sensitive segments with entropy flattening. The paper points out that extending such fine-grained objectives to diverse model-generated outputs without relying on predefined response-level or human-annotated segment-level supervision remains challenging; KARU addresses this by jointly optimizing token-level and distribution-level objectives around KA.
LLM reinforcement learning
- Policy optimization algorithms: PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), etc. are widely used for post-training.
- Reward granularity: ORM evaluates only the final response, PRM provides intermediate-step supervision (Lightman et al., 2023), but finer granularity usually requires additional reward models or annotations.
- RL unlearning: RULE (Zhang et al., 2026) is an RL unlearning method with response-level rewards; the paper points out that existing RL unlearning methods mainly rely on sequence-level rewards, assigning the same supervision signal to all tokens and failing to identify the tokens that carry knowledge.
Baselines and benchmarks: the comparison baselines are GA, NPO, SimNPO, PALU, RT, and RULE (GA/NPO/SimNPO also have +GDR/+KLR variants); the benchmarks are RWKU (Cao et al., 2024) and MUSE (Shi et al., 2025). The authors position this paper as: taking KA as the unlearning target and unifying token-level and distribution-level objectives into an RL framework, distinct from RULE's response-level rewards.
How does the paper address this problem?
KARU takes KA as the forgetting target and approximately optimizes it in RL with three components: KATR, LF, and ANS.
Overall idea: formalize the unlearning target as knowledge association (KA)—the conditional predictive distribution at the knowledge token position
K_\theta(z_{t_0}) \triangleq \pi_\theta(\cdot \mid z_{t_0}), i.e., the model's predictive distribution at the first knowledge token position—and then approximately optimize, under an RL framework, the objective of "suppressing KA on the forget set and maintaining KA on the retain set."Problem setting and formalization
- Ideal objective: minimize the difference between Kθ and the reference model Kref on forget samples, while constraining the knowledge token probability on retain samples to be no lower than a threshold (Eq. 3); the reference model is unavailable, so approximation is needed.
- Empirical basis: after unlearning, the distribution shift is concentrated on knowledge tokens (Figure 1 left); knowledge tokens account for a small proportion of the response—below 10% on RWKU and below 17% on MUSE (Table 4).
KATR: Knowledge Association Token Reward
- Derivation source: taking the difference measure in Eq. 3 as KL divergence, its gradient is isomorphic to the token-level policy gradient, and the log coefficient can be interpreted as the ideal token-level reward (Eq. 8).
- Reward design: on forget samples, knowledge tokens are given −1 and refusal tokens +1; on retain samples the reverse (knowledge tokens +1, refusal tokens −1); the rest are 0 (Eq. 12). The authors call it a sign-level approximation of Eq. 8, avoiding estimating the magnitude of the unavailable reference model.
- Token partitioning: knowledge tokens are extracted offline from the forget set using BERT-Large-NER and post-processed; refusal tokens come from a predefined refusal template (Appendix B).
LF: Logits Flattening
- Motivation: the authors believe KATR does not explicitly constrain the predictive distribution, and probability mass may shift among related alternatives while still remaining concentrated.
- Entropy proxy principle: confabulated knowledge (not learned by the model) has higher entropy at the knowledge token position than memorized knowledge (Figure 1 right: 3.96 vs. 1.44), so entropy is used as a reference-free distribution-level proxy; in Table 5, the Shannon entropy separation SNR is 1.94, higher than the 1.51 of top-1 probability.
- Implementation: maximize entropy only at knowledge token positions (Eq. 9); full-vocabulary entropy is too expensive, so it is approximated with Top-k probability mass, with an error bound of O(ε log |V|) (Lemma 4).
ANS: Adaptive Neighborhood Sampling
- Observation: collateral degradation has a proximity effect—retain samples closer to the forget set in representation space show larger ROUGE-L drops (Figure 4).
- Weighting scheme: sorted by Mahalanobis distance, the nearest top α proportion of retain samples are weighted-sampled with linearly decreasing weights, and β controls the maximum weighting strength (Eq. 10); Mahalanobis achieves ROC-AUC 1.000 on forget/retain separation, higher than Euclidean 0.696 and cosine 0.827 (Figure 9).
- Stability: the authors report that the neighborhood structure barely changes during training—95% of retain samples change in relative distance by less than 2.8%, and the ranking Spearman correlation is 0.998, supporting one-time offline computation.
Overall pipeline
- Two stages: first use the forget set for refusal SFT initialization (following RULE's rejection-tuned model), then perform RL training (PPO or GRPO), with the total objective being the KATR reward term minus the LF regularization weighted by λLF (Eq. 11, Algorithm 1).
- Hyperparameters (Table 6): LR 2e-6; on RWKU, λLF=1e-2, Top-k is 50 (PPO)/100 (GRPO), αWS 15.0/10.0, βWS 1.0/3.0.
What experiments did the paper conduct?
Evaluated on RWKU and MUSE across 4 models and two algorithms, PPO/GRPO, KARU's unlearning score is lower than RULE's.
Experimental setup
- Models under test: RWKU mainly uses Llama3-8B-Instruct, with Llama3.2-3B-Instruct and Qwen2.5-7B-Instruct for cross-model validation; MUSE uses ICLM-7B.
- Benchmarks and metrics: RWKU evaluates from three aspects—forgetting (FB/QA/AA probes, ROUGE-L, lower aggregate unlearning score is better), retention (neighborhood FB/QA, higher is better), and general capability (Gen on MMLU 5-shot, Rea on BBH, Flu on AlpacaEval); MUSE evaluates No Verb Mem., No Know Mem., and Utility Preserv. (ROUGE-L).
- Baselines: GA, NPO, SimNPO (including +GDR/+KLR variants), PALU, RT, RULE; both KARU and RULE are initialized from RULE's rejection-tuned model and optimized with PPO/GRPO.
- Evaluation method: all metrics are automatic metrics such as ROUGE-L; the case study includes 1–5 human ratings (Table 8). The paper does not report the number of training repetitions or statistical significance tests (the distribution analysis in Appendix A includes paired t-tests).
Main results
RWKU, Llama3-8B-Instruct (per Table 1, lower unlearning score is better, higher retention is better):
Swipe horizontally to read the table
Method Forgetting All ↓ Retention All ↑ Gen Rea Flu Original 76.9 87.6 63.3 40.5 706.6 GA 68.4 79.8 63.5 38.8 707.7 NPO 31.5 72.1 62.8 37.9 693.1 SimNPO 44.7 77.0 62.7 38.5 700.2 PALU 38.0 76.6 63.0 35.6 703.6 RT 58.7 77.9 63.6 38.5 680.7 RULE (PPO) 30.3 61.4 63.3 38.9 641.5 RULE (GRPO) 32.5 71.9 63.3 41.0 702.0 KARU (PPO) 23.7 67.7 63.5 41.0 717.7 KARU (GRPO) 22.6 71.1 63.0 42.1 707.7 - Forgetting: the authors report that KARU's aggregate unlearning score decreases relative to RULE by 6.6 points under PPO and 9.9 points under GRPO, corresponding to relative reductions of 21.8% and 30.5%; the abstract's "up to 30.5%" refers to the GRPO setting.
- Retention: KARU (GRPO)'s retention score is 1.1% lower than RULE (GRPO)'s, while KARU (PPO)'s is higher than RULE (PPO)'s; the authors state that forgetting and retention are in tension, and the goal is the best trade-off rather than maximizing retention.
- Trade-off: on Pareto frontier AUC, KARU-GRPO is 0.559, higher than RULE-GRPO's 0.416, NPO's 0.473, and PALU's 0.411 (Figure 6 top).
Cross-model generalization
- Llama3.2-3B (Table 2): KARU's Forgetting All is 30.1 (PPO)/27.0 (GRPO), and RULE's is 32.5/33.8; the authors report relative improvements of 7.4% (PPO) and 20.1% (GRPO).
- Qwen2.5-7B (Table 2): KARU is 30.4/29.0, and RULE is 44.3/43.5; the authors report relative reductions of 31.4%/33.3%, with modest changes in retention.
MUSE results
- Setup: ICLM-7B, with metrics No Verb Mem., No Know Mem. (lower is better), and Utility Preserv. (higher is better) (Table 7).
- Results: KARU (GRPO) is 6.9 / 0.7 / 60.0, and RULE (GRPO) is 7.3 / 1.4 / 55.6; KARU (PPO) is 7.8 / 0.0 / 30.3, and RULE (PPO) is 2.2 / 2.6 / 13.6. GA and SimNPO are 0.0 on all three.
- Exception: on MUSE, KARU (PPO)'s No Verb Mem. (7.8) is higher than RULE (PPO)'s (2.2), i.e., forgetting is weaker; the authors emphasize that KARU has the highest utility preservation and competitive forgetting.
Mechanism and quality analysis
- Entropy analysis: Figure 7 reports the entropy of each method at knowledge token positions; KARU is higher than Original, RT, and RULE on each RWKU subset (300 QA samples per subset); the authors state that KARU pushes predictions toward a high-uncertainty state.
- Supervision concentration: Appendix B.1 analyzes validation queries using more than 800 RWKU queries; the share of supervision that KATR assigns to knowledge tokens is higher than a uniform reference, whereas under RULE supervision is the same at all positions (Figure 8).
- Case study: in Table 8, KARU receives human ratings of 5/5 on two forget queries; GA and RULE leak the answer, NPO confuses entities, and SimNPO exhibits repetitive degradation.
Other ablations and analyses
- Component ablation (Table 3, RWKU/Llama3-8B): under PPO, Forgetting All for Baseline→+KATR→+LF→+ANS is 30.3→27.7→24.0→23.7; under GRPO it is 32.5→25.6→22.1→22.6.
- ANS effect on retention: under PPO, adding ANS raises Retention All from 61.6 to 67.7; under GRPO it rises from 65.3 to 71.1 (Table 3).
- Pareto AUC (GRPO): Baseline 0.416→KATR 0.504→KATR-LF 0.504→KARU 0.559 (Figure 10).
- Hyperparameter sensitivity (Figure 11): λLF, Top-k, αWS, and βWS are swept on four RWKU subsets; the authors state that the trade-off is stable over a relatively wide range.
- Overhead: KARU takes about 1.43 minutes per step, and RULE about 1.40 minutes (Appendix F.4, H20 GPU).
What are some points that can be further explored?
The authors point out two limitations: knowledge token extraction and the two-stage pipeline; the experiments cover two benchmarks and four models.
Limitations and future directions pointed out by the authors
- Knowledge localization relies on NER: The framework currently relies on named entity recognition to extract knowledge tokens; the authors believe it can be improved with more accurate knowledge localization (Limitations and Future Work in the Conclusion).
- Two-stage pipeline: The current approach uses a two-stage pipeline of SFT initialization plus RL training; the authors consider developing an end-to-end forgetting framework a promising future direction (same as above).
Scope of the experiments
- Models tested: Llama3-8B-Instruct, Llama3.2-3B-Instruct, Qwen2.5-7B-Instruct (RWKU) and ICLM-7B (MUSE), 4 in total (Sec. 5, Appendix F.1).
- Benchmarks and algorithms: Two benchmarks, RWKU and MUSE; two RL algorithms, PPO and GRPO, both initialized from RULE's rejection-tuned model (Sec. 5).
- Forgetting scenario: The paper explicitly focuses on entity-centric factual knowledge forgetting (Sec. 3.1).
- Evaluation method: The main metrics are automatic metrics of the ROUGE-L type; human evaluation appears only in the two cases in Table 8 (1–5 points), and the paper does not report the consistency of human ratings.
- Not reported in the paper: The number of training repetitions and statistical significance tests for the main results (the distribution shift analysis in Appendix A reports a paired t-test with p<0.001).
Summarize the main content of the paper
KARU redefines the forgetting target as knowledge association; on RWKU/MUSE, its forgetting is better than RULE while retention is comparable.
This is an LLM forgetting method paper, and its core claim is: the fundamental target of forgetting is not the entire generated response, but the knowledge association (KA) between the query context and the target knowledge.
- Problem: The authors argue that existing RL forgetting methods use sequence-level rewards that treat all tokens uniformly, so the forgetting signal is diluted, and knowledge may remain in the predictive distribution.
- Method: Under an RL framework, KARU approximately optimizes the KA objective with three components—KATR assigns ±1 token-level rewards to knowledge/refusal tokens; Logits Flattening maximizes the Top-k entropy at knowledge token positions; ANS weights sampling of neighboring retained samples by Mahalanobis distance. The pipeline is refusal SFT initialization plus PPO/GRPO training.
- Main results: On RWKU, Llama3-8B, KARU's aggregate forgetting score is 23.7 (PPO)/22.6 (GRPO), lower than RULE's 30.3/32.5; the authors state relative reductions of 21.8%/30.5% (Table 1); retention and general capability are comparable.
- Generalization: The trend is consistent on Llama3.2-3B and Qwen2.5-7B, with relative reductions of 31.4%/33.3% on Qwen2.5-7B (Table 2); on MUSE, KARU (GRPO)'s Utility Preserv. reaches 60.0, higher than RULE's 55.6 (Table 7).
- Exception: On MUSE, PPO, KARU's No Verb Mem. (7.8) is higher than RULE's (2.2).
- Authors' conclusion: KARU consistently improves the forgetting–retention trade-off across different RL algorithms and models; limitations are reliance on NER to extract knowledge tokens and the two-stage pipeline, and the authors plan to explore end-to-end forgetting in the future.