LEAF-OPD: Learning to Unlearn from Privileged Context via On-Policy Distillation
LEAF-OPD: Learning to Forget from Privileged Contexts via On-Policy Distillation
The authors propose LEAF-OPD, which uses a frozen teacher with privileged context to perform token-level distillation on student trajectories to achieve LLM unlearning; on RWKU, Forget Quality is 34% lower than the second-best method.
- LLM unlearning must remove the target knowledge while preserving model utility. The authors argue that existing methods mostly only teach the model "what not to answer" without explicitly teaching it "how it should answer," and that RLVR-type methods incur additional sampling overhead.
- LEAF-OPD has two stages: first, use Supervision Signal (SS) to evaluate the strength of teacher supervision with privileged context, and when insufficient, strengthen it with lightweight SFT; then distill the token-level distribution of this frozen teacher on the student's self-generated trajectories to a student without context, using different distillation objectives for forget and retain data respectively.
- Evaluated on three benchmarks: TOFU, RWKU, and MUSE-News. The authors report that on TOFU, FQ, MU, and F-RL are all the best among the evaluated methods, with F-RL at 0.0027; on RWKU, average FQ is 34% lower than the second-best method; on MUSE-News, it satisfies all three criteria and has the highest KnowMemr.
- The authors note that KnowMem performance on MUSE is relatively weak, possibly because training uses only text continuation data. The experiments cover 3 models and 3 benchmarks; the paper does not report the number of training repetitions or variance, and code and data are planned for release.
- Evaluated models
- Llama2-7BQwen3-8BLlama3-8B-Instruct
- Benchmarks
- TOFURWKUMUSE-News
- Metrics
- Forget Quality (FQ)Model Utility (MU)Forget ROUGE-L (F-RL)Retain ROUGE-L (R-RL)Retain Quality (RQ)VerbMemKnowMemSupervision Signal (SS)
Researchers propose an LLM unlearning method, LEAF-OPD, in which a frozen teacher model provides token-level supervision on student-generated responses: on forget data the teacher guides unlearning with privileged context (instructing it not to leak the forget targets), and on retain data it uses no privileged context to maintain model capability; the student never accesses privileged context in either case and learns how to respond to knowledge that should be forgotten. Addressing the problem that teacher supervision often covers only the first few tokens of student trajectories, the paper proposes the Supervision Signal (SS) metric to measure correction supervision strength and uses lightweight SFT to strengthen the teacher's context-conditioned refusal behavior. On three benchmarks, TOFU, RWKU and MUSE-News, LEAF-OPD achieves the best Forget Quality, Model Utility and Forget ROUGE-L on TOFU, and on RWKU its Forget Quality is 34% higher than the second-best method while maintaining competitive retention performance.
Paper analysis
What problem does this paper attempt to solve?
The authors argue that existing LLM unlearning methods only suppress the target knowledge without teaching the model how to respond, and propose distillation from privileged context to teach it explicitly.
Core problem: how to remove target knowledge from an LLM while preserving utility, and explicitly teach the model how it should respond when encountering that knowledge, rather than only suppressing it.
- Problem setting: LLM training corpora contain private information, copyrighted content, and potentially dangerous knowledge, which the authors say has given rise to LLM unlearning—selectively removing knowledge in forget data while preserving model utility as much as possible.
- Shortcomings of existing methods (according to the paper): GA-type methods continuously push the model away from its original distribution, harming utility; methods that add retain data only constrain "what should not be answered" without teaching "what should be answered"; RLVR-type methods (RULE, ReRULE) require additional response sampling and are computationally expensive; training-free methods (such as GUARD) do not change parameters, which the authors say raises the question of "whether knowledge is truly removed or only suppressed at inference time."
- Core observation: the effectiveness of training-free methods shows that, given proper contextual guidance, LLMs can generate safe responses; therefore, "unlearning behavior under privileged context" can be internalized into the parameters of a context-free model via On-Policy Self-Distillation (OPSD).
- The paper proposes: the LEAF-OPD algorithm—a frozen teacher provides token-level supervision on the student's online trajectories under privileged context; it also proposes the Supervision Signal (SS) metric to measure teacher supervision strength, and when supervision is weak, uses lightweight SFT to turn a "bad teacher" into a "good teacher."
- Method positioning: this is not an attack or defense paper and does not involve a threat model; it targets the training scenario in which the model owner actively removes knowledge, with the teacher and student controlled by the same party.
What related research exists?
Related work falls into two groups, LLM unlearning and on-policy/context distillation; this paper extends the OPSD paradigm to the unlearning task.
LLM Unlearning (parameter-update type)
- GA (Yao & Xu, 2024): directly maximizes the loss on forget data; the paper notes that it continuously pushes away from the original distribution and harms utility. NPO (Zhang et al., 2024a) introduces a reference model, and SimNPO (Fan et al., 2026) removes this dependence.
- FLAT (Wang et al., 2025) and SGA (Pang et al., 2025): introduce f-divergence loss adjustment and label smoothing, respectively.
- RULE and ReRULE (Zhang et al., 2025; Pang et al., 2026): learn refusal behavior via RLVR; the paper notes that such methods require additional response sampling and are computationally expensive.
LLM Unlearning (training-free type)
- GUARD (Deng et al., 2025): does not change parameters and suppresses target knowledge at generation time; the paper notes that this raises the question of whether the knowledge is truly removed.
On-Policy and Context Distillation
- Context distillation (Snell et al., 2022): internalizes behavior induced by additional context into the model.
- MiniLLM (Gu et al., 2024) and GKD (Agarwal et al., 2024, etc.): the former studies reverse-KL distillation, and the latter trains on student-generated sequences to reduce training-inference mismatch.
- OPSD, OPCD, etc. (Zhao et al., 2026; Ye et al., 2026; Penaloza et al., 2026): use privileged information to supervise the student and internalize context-conditioned behavior; this paper directly extends the OPSD paradigm.
Baselines and benchmarks: the experiments compare against baselines including GA, GD, KL, LLMU, SGA, PO, DPO-RT, NPO/NPO-RT, SimNPO, FLAT, RULE, ReRULE, Task Vectors, WHP, ICUL, and Prompt-based Unlearning, using three benchmarks: TOFU, RWKU, and MUSE-News.
The authors position this paper as extending OPSD to LLM unlearning, differing from existing methods in that it distills context-conditioned unlearning behavior into the parameters via token-level supervision and does not require a task-specific reward function.
How does the paper address this problem?
Two stages: first use the SS metric to evaluate and strengthen a privileged-context teacher, then online-distill the teacher's token-level distribution to a context-free student.
Overall idea: transfer the teacher behavior of "refusing when given privileged context" to a "no-context" student via on-policy token-level distillation, using two distillation objectives for forget data and retain data respectively. The pipeline of LEAF-OPD is shown in Figure 2.
Problem Formulation
- Given a forget dataset Df and a retain dataset Dr, the goal is to change the original model π0 into πθ so that it no longer exhibits the knowledge in Df while preserving utility on Dr as much as possible.
- The general objective is to minimize the weighted sum of the forget term and the retain term: minθ Lforget(θ; Df) + λ Lretain(θ; Dr); different methods instantiate the two terms differently.
Stage 1: Finding a Good Teacher (Supervision Signal)
- Using factual trajectories generated by the original student π0 that contain the target knowledge, compare the difference ∆t in the log probability of each token for the same teacher with and without privileged context; a negative ∆t indicates that the context suppressed that token, i.e., provided corrective supervision.
- Define the Supervision Signal: SS(Δ, s, e) = −1/(e−s+1)∑t=se Δt, which measures the average strength of supervision over the interval [s, e]; higher means stronger opposition to the factual trajectory.
- Two specialized metrics: SS1 covers the entire trajectory and measures overall strength; SS0.5 covers the second half of the trajectory and measures the persistence of supervision.
- Criterion: a teacher is considered good when the means of SS1 and SS0.5 on the forget training data both exceed 1; when insufficient, lightweight SFT is applied to the teacher to strengthen its refusal behavior under privileged context. SFT data amounts: 20 samples for TOFU, 240 synthetic samples for RWKU, and 100 drawn from 2,666 source records for MUSE (Table 8).
Stage 2: On-Policy Self-Distillation from the Privileged Teacher
- The teacher π⟨T⟩ is frozen and the student π⟨S⟩ is updated; the student first generates trajectories online, and the teacher then provides token-by-token supervision along those trajectories. The teacher obtained in stage 1 also serves as the student's initial policy.
- Forget objective: for forget samples, the teacher additionally sees a privileged context cf specifying the unlearning behavior, and the KL divergence between the teacher and student distributions at each token is minimized (Lf).
- Retain objective: for retain samples, the same frozen teacher without context serves as the retain teacher, constraining the student to maintain the teacher's token distribution (Lr) to preserve utility.
- Differences from OPSD: adds the teacher evaluation and strengthening of stage 1, and the separated distillation objectives for forget/retain data in stage 2.
Implementation Details (Appendix C.2)
- The student is trained with a LoRA adapter (rank 64, scaling 128), and the teacher is frozen; optimizer AdamW, BF16 precision.
- TOFU: stage 1 SFT for 3 epochs, lr 1×10⁻⁴; stage 2 distillation for 1 epoch, with forget data oversampled 45 times. RWKU: stage 2 trained for 2 epochs, lr 5×10⁻⁵. MUSE-News: stage 1 trains a teacher LoRA (rank 16) and merges it, with half of the forget samples prefixed with up to 20 real continuation tokens; stage 2 trains for 3 epochs and constructs 2,727 continuation-prefix pairs.
- KL direction: the main experiments use reverse KL (β=1); Appendix B discusses the difference between forward and reverse KL from the perspective of gradient structure.
What experiments did the paper conduct?
Compares against more than ten baselines on TOFU, RWKU, and MUSE-News, and conducts ablations on KL direction, teacher quality, and retain supervision.
Experimental Setup
- Models under test: TOFU uses Llama2-7B and Qwen3-8B; RWKU uses Llama3-8B-Instruct; MUSE-News uses the original checkpoint provided by MUSE (Llama-2-7B).
- Benchmarks and scale: TOFU contains 200 fictitious authors, each with 20 question-answer pairs, and the main experiments use the forget01 split (1% of authors as the forget set); RWKU contains 200 celebrities and 13,131 multi-level probes; MUSE-News is news article unlearning.
- RWKU training data: RWKU is a zero-shot benchmark and does not provide a training set; the authors use GPT-5.6-Luna to generate forget and retain training data for each target entity.
- Metrics: TOFU reports FQ (KS test p-value, higher is better), MU (higher is better), F-RL (lower is better), R-RL (higher is better); RWKU's FQ uses ROUGE-L recall on three probe types FB/QA/AA (lower is better), RQ uses FB/QA probes on neighboring entities (higher is better); MUSE reports VerbMemf, KnowMemf (lower is better) and KnowMemr (higher is better).
- Baselines: GA, GD, KL, LLMU, SGA, PO, DPO-RT, NPO/NPO-RT, SimNPO, FLAT, RULE, ReRULE, Task Vectors, WHP, ICUL, Prompt-based Unlearning; RWKU additionally adds GDR/KLR regularization variants.
- Other: training-free baselines (ICUL, Prompt-based) are evaluated under oracle routing, i.e., assuming perfect discrimination between forget and retain queries; the paper does not report the number of training repetitions.
Main Results
TOFU (forget01, per Table 1):
Swipe horizontally to read the table
Method Llama2-7B FQ↑ MU↑ F-RL↓ R-RL↑ Qwen3-8B FQ↑ MU↑ F-RL↓ R-RL↑ Original LLM 4.4883e-06 0.6239 0.9851 0.9818 0.0068 0.6346 0.9326 0.9135 Retained LLM 1.0000 0.6267 0.4080 0.9833 1.0000 0.6470 0.3856 0.8965 GA 0.0068 0.5990 0.4817 0.9204 0.0541 0.4367 0.5279 0.5407 NPO-RT 0.0030 0.5994 0.5049 0.9270 0.0541 0.4399 0.5458 0.5442 FLAT (Pearson) 0.0030 0.6304 0.4825 0.9284 0.0541 0.4693 0.4860 0.5447 ICUL 0.0005 0.6239 0.4772 0.9818 0.0541 0.6346 0.4619 0.9135 LEAF-OPD 0.0143 0.6892 0.0027 0.9382 0.0971 0.6422 0.0027 0.6756 RWKU (Llama3-8B-Instruct, per Table 2, lower FQ is better, higher RQ is better):
Swipe horizontally to read the table
Method FQ FB FQ QA FQ AA FQ All RQ FB RQ QA RQ All Original 85.6 70.3 74.7 76.9 93.1 82.0 87.6 GA 72.0 64.6 68.5 68.4 85.0 74.7 79.8 NPO 46.6 39.0 35.3 40.3 79.2 70.9 75.1 SimNPO 42.1 36.1 42.2 40.1 82.8 70.3 76.5 RULE(GRPO) 28.0 16.8 38.3 27.7 76.2 71.3 73.7 ReRULE(GRPO) 26.5 13.1 33.2 24.2 79.5 68.8 74.1 LEAF-OPD 13.1 5.9 29.2 16.0 79.3 75.0 77.1 - The authors state that LEAF-OPD achieves the best FQ, MU, and F-RL among the unlearning methods under test on both TOFU models, with F-RL of only 0.0027; R-RL exceeds all parameter-update baselines but is lower than training-free methods—the authors note that the latter benefit from a classifier that can perfectly discriminate forget/retain queries, whereas LEAF-OPD does not require one.
- The authors state that on RWKU, LEAF-OPD's average FQ is 34% lower than the second-best ReRULE, and its average RQ is higher than RULE and ReRULE, ranking in the top 50% of all methods under test.
MUSE-News Results
- What was measured: news unlearning on Llama2-7B, with the criterion that both VerbMemf and KnowMemf do not exceed the Retained LLM and KnowMemr is nonzero (Table 3).
- Results: LEAF-OPD satisfies all three criteria (VerbMemf 5.2, KnowMemf 25.6, KnowMemr 32.5); among methods satisfying the criteria it has the highest KnowMemr, 0.7 higher than the second-best FLAT (31.8). GA and NPO have KnowMemr of 0.0 and do not satisfy the criteria.
- Authors' interpretation: LEAF-OPD performs relatively weaker on QA-type KnowMem metrics than on continuation-type VerbMem, which the authors suggest may be attributable to using only text continuation data for unlearning on MUSE.
Ablations (TOFU, Table 4)
- KL direction: with a good teacher, both forward (β=0) and reverse (β=1) KL are effective; with a bad teacher, forward KL has a clearly lower F-RL (0.0392 vs. 0.7356 on Llama2-7B) but still falls short of the good teacher. The authors attribute the relative advantage of forward KL to its gradient structure being able to exploit sparse supervision at a few token positions (Appendix B).
- Removing retain supervision: LEAF-OPDβ=1,no-retain decreases MU on both models; on Qwen3-8B, FQ rises to 0.5786 but MU drops to 0.0166, which the authors describe as accompanied by an almost complete collapse in utility.
- KL only on the first token: both first-token-only variants perform poorly (F-RL of 0.9168 and 0.4488 respectively on Llama2-7B), which the authors state does not solve the weak-supervision problem of a bad teacher.
Other Ablations and Analyses
- Runtime (Table 6, Qwen3-8B): LEAF-OPD takes 20 minutes, longer than fine-tuning baselines such as GA (5), NPO (8), and FLAT (10); the authors state that compared with RLVR-type methods, stage 2 requires only 1 rollout per prompt rather than n, but the table does not give runtime numbers for RULE.
- Stage 1 SFT only (Table 7, Llama2-7B): with the good teacher used alone, F-RL is 0.9323, far higher than the 0.0027 of full LEAF-OPD; the authors state that the effect comes mainly from the full framework rather than stage 1 SFT itself.
- SS metric (Figure 4): the authors state that lightweight SFT simultaneously improves the bad teacher's SS1 and SS0.5 on the forget training data of all three benchmarks; panel (c) of the figure gives the mean SS before and after SFT on each benchmark.
- TOFU Forget05/Forget10 (Appendix D, Table 9, Llama2-7B): LEAF-OPD has MU of 0.6726/0.6773 and R-RL of 0.9480/0.9363, but its FQ is lower than NPO (on Forget05, NPO is 0.3281 and LEAF-OPD is 9.590×10⁻¹⁰).
What are some points that can be further explored?
The authors note that the relatively weak KnowMem on MUSE may stem from the form of the training data, and state that the SS metric may be applicable to broader distillation frameworks.
Limitations and Future Directions Noted by the Authors
- Relatively weak KnowMem on MUSE: the authors write that LEAF-OPD's performance on QA-type KnowMem metrics is less than ideal relative to VerbMem, which they suggest may be attributable to using only text continuation data for unlearning on MUSE (Section 5.4).
- Scope of the SS metric: the authors state that SS is used in this paper to evaluate the strength of LEAF-OPD's teacher supervision, but that the metric may apply to broader on-policy distillation frameworks aimed at unlearning or changing target behavior patterns (Section 1, contributions).
- Code and data release: the authors plan to make code and data public after the paper is accepted (Reproducibility Statement); they have not yet been released.
Scope of Experiments
- The models under test are Llama2-7B, Qwen3-8B, and Llama3-8B-Instruct, 3 in total; the benchmarks are TOFU (main experiments on forget01, with forget05/forget10 added in the appendix), RWKU, and MUSE-News, 3 in total.
- RWKU's forget/retain training data are generated by GPT-5.6-Luna (the benchmark itself does not provide a training set); the lightweight SFT data are synthetic samples: 20 for TOFU, 240 for RWKU, and 100 for MUSE (Table 8).
- Evaluation metrics are mainly ROUGE-L, KS test p-values, and memorization metrics; no LLM-judge scoring is used; training-free baselines are evaluated under an oracle routing setting that assumes perfect discrimination between forget/retain queries.
- The paper does not report the number of repetitions, the number of random seeds, or the variance or confidence intervals of results for each experiment.
- The runtime comparison (Table 6) covers only 9 methods on Qwen3-8B and does not include runtime numbers for RLVR methods such as RULE/ReRULE.
Summarize the main content of the paper
LEAF-OPD online-distills the unlearning behavior of a privileged-context teacher into the student's parameters, achieving a competitive unlearning-utility trade-off on three unlearning benchmarks.
An LLM unlearning training method paper addressing the problem of "only suppressing knowledge without teaching the model how to respond."
- Problem: the authors argue that GA-type methods harm utility, retain regularization only constrains "what should not be answered," RLVR-type methods have high sampling overhead, and it is questionable whether training-free methods truly remove knowledge.
- Method: LEAF-OPD has two stages. Stage 1 uses the Supervision Signal (SS1 and SS0.5) to measure the token-level supervision strength of a teacher with privileged context over student trajectories, and applies lightweight SFT to strengthen it when insufficient; stage 2 freezes the teacher and performs token-level KL distillation along trajectories generated online by the student, using the teacher with context for forget data and the same teacher without context for retain data, with the student never seeing the context.
- TOFU results: FQ, MU, and F-RL are the best among the unlearning methods under test on Llama2-7B and Qwen3-8B, with F-RL of 0.0027 on both models and MU of 0.6892 on Llama2-7B (Table 1).
- RWKU results: average FQ of 16.0 on Llama3-8B-Instruct, which the authors state is 34% lower than the second-best ReRULE, and average RQ of 77.1, higher than RULE and ReRULE (Table 2).
- MUSE-News results: satisfies all three criteria, with KnowMemr of 32.5, the highest among methods satisfying the criteria (Table 3); the authors note that its KnowMem is relatively weak compared with VerbMem, possibly due to using only text continuation data.
- Key ablations: with a bad teacher, forward KL outperforms reverse KL but still falls short of the good teacher; removing retain-side supervision greatly reduces MU; distilling only on the first token performs poorly; with stage 1 SFT only, F-RL is as high as 0.9323, and the authors state that the effect comes from the full framework.
- Authors' conclusion: privileged-context supervision plus on-policy token-level distillation is an effective approach to LLM unlearning, reducing reproduction of target knowledge while maintaining competitive utility across multiple settings under test.