Examining machine unlearning in AI safety from an adversarial perspective

1. 01What problem does this paper attempt to solve?

   The authors question whether safety unlearning truly deletes dangerous knowledge from the weights, and use white-box attacks to examine its essential difference from safety fine-tuning.

   Core question: does machine unlearning truly remove dangerous knowledge from LLM weights, or does it, like refusal-based safety training, merely "mask" the knowledge.

   • Problem setting: LLM pretraining corpora contain dangerous information, and safety fine-tuning can be bypassed by jailbreaks; unlearning methods such as RMU claim to reduce accuracy on dangerous knowledge benchmarks to random level, but prior work has found that the knowledge can still be recovered.
   • Existing shortcomings (according to the paper): the authors believe that existing evaluations are mostly black-box, only verifying that knowledge cannot be extracted from outputs, which does not match the goal of "deleting from weights" and may yield misleading success conclusions; the original RMU paper reports that white-box jailbreaks such as GCG are ineffective against it.
   • Core hypothesis: the authors propose that if there exists at least one method—without updating weights, or updating weights with data that has almost no mutual information with the target knowledge—that can recover considerable accuracy, then from a safety perspective unlearning is not successful.
   • Paper contributions: the authors state that this is the first comprehensive white-box evaluation of safety-oriented unlearning methods, comparing RMU and NPO with a DPO safety training baseline, and proposing multiple adaptive recovery methods.
   • Threat model:
     • Attacker goal: recover the dangerous knowledge measured by WMDP (biosecurity, cybersecurity) from the unlearned model, measured by WMDP accuracy.
     • Knowledge: white-box access to the unlearned model; and the assumption of access to the original model before unlearning, or obtaining an equivalent model via fine-tuning.
     • Capabilities: can modify model weights, intervene in activation space at inference time, and optimize input prompts.
     • Victim system: Zephyr-7B-β unlearned with RMU or NPO, or safety-trained with DPO; the authors note that the original RMU setting assumes only black-box access, and the white-box setting is stronger than its threat model.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: safety training and jailbreak, LLM unlearning, evaluation of unlearning robustness, and three concurrent works.

   Safety training and jailbreak

   • DPO (Rafailov et al., 2024) and PPO (Ouyang et al., 2022): train models to refuse dangerous questions; the circuit breakers of Zou et al. (2024b) use representation engineering to orthogonalize the directions of unwanted concepts.
   • Jailbreak methods: prompt-only strategies (Shah et al., 2023; Huang et al., 2023), white-box optimized prompts (GCG of Zou et al., 2023; Andriushchenko et al., 2024), ablating model activations (Arditi et al., 2024, which found that refusal is mediated by a single direction in activation space).

   LLM unlearning

   • The gold standard of unlearning (Cao and Yang, 2015; Bourtoule et al., 2021): make the model indistinguishable from a model retrained after deleting the target data.
   • RMU (Li et al., 2024): a safety-oriented unlearning method proposed together with the WMDP benchmark, and the main evaluation target of this paper; NPO (Zhang et al., 2024) is a general unlearning algorithm, which the authors adapt for dangerous knowledge removal.
   • The authors point out that the field lacks standardized metrics: evaluation criteria vary across benchmarks, from perplexity and multiple-choice accuracy to ROUGE after prefix injection.

   Evaluation of unlearning robustness

   • Lynch et al. (2024): by probing internal representations, knowledge can be extracted from both original and unlearned models at comparable rates.
   • The original RMU paper (Li et al., 2024) and Hu et al. (2024): found that fine-tuning can easily undo unlearning protection.
   • Gonen and Goldberg (2019) (word embedding debiasing) and Sharma et al. (2024) (diffusion model unlearning): the authors cite them to illustrate the importance of careful evaluation and white-box methods.

   Concurrent work

   • Deeb and Roger (2024): three methods including RMU are not robust to fine-tuning attacks using data from the unlearning distribution; the authors consider this supportive of this paper's conclusions.
   • Doshi and Stickland (2024): used black-box techniques such as few-shot to test RMU and LLMU, recovering a moderate amount of knowledge; the authors state that they recovered most of the knowledge in a white-box setting.
   • Che et al. (2024): tested the sensitivity of 8 capability removal techniques to off-the-shelf attacks; the authors state that this paper differs from it in emphasizing the adaptivity of adversarial evaluation.

   Baselines and benchmarks: This paper uses DPO as the safety training baseline, with WMDP (1,273 biology questions, 1,987 cybersecurity questions) and MMLU as benchmarks. The authors position this paper's difference from the above work as: proposing white-box extraction methods that do not modify weights or use only a few samples, and emphasizing adaptive evaluation.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors use six types of white-box knowledge recovery methods to attack RMU, NPO, and DPO, testing whether knowledge still exists in the weights after unlearning.

   Overall approach: under a white-box setting, apply a set of recovery methods ranging from representation engineering to prompt optimization to the unlearned model; except for fine-tuning, none modify the weights, thereby determining whether the knowledge is still stored in the weights.

   Problem setting and evaluation method

   • Unlearning setting: Unlearning relies on a forget set (the domain to be removed, such as enhanced pandemic pathogens) and a retain set (knowledge to be retained, such as general biology), with data from the WMDP biology and cybersecurity corpora.
   • Success criterion: Dangerous knowledge is measured by WMDP multiple-choice accuracy; to probe latent knowledge under refusal, the option with the highest probability among A/B/C/D is taken directly as the answer. MMLU measures general capabilities.
   • Protected models: RMU uses a public checkpoint (Zephyr-7B-β fine-tuned on WMDP and WikiText); NPO (including the NPO+RT variant with retain loss) and DPO were fine-tuned by the authors on WMDP. Preference data was generated by GPT-4 from the corpus, with dangerous questions using one of 80 refusal strings as chosen and the correct option as rejected; training data was mixed at forget/retain/OpenAssistant 50:25:25.

   Finetuning

   • Design: Use LoRA to fine-tune the unlearned model on three types of data: the forget set, the retain set (disjoint from the forget set), and WikiText (little overlap with dangerous knowledge), with sample sizes from 5 to 1000.
   • Rationale (given in the paper): The retain set and WikiText have low mutual information with the target knowledge, so the model cannot acquire new knowledge from them; if accuracy is restored, it indicates the knowledge was already in the weights.

   Orthogonalization

   • Design: Compute a "forget direction" separately for each transformer block: collect the outputs of each block of the original model and the unlearned model on the forget preference data, compute the direction using difference in means, and remove that direction layer by layer at inference.
   • Variant without the original model: Use the representations of the unlearned model on WikiText and the MMLU validation set as benign representations for differencing (Appendix E.3); tokens with z-score greater than 3 are removed as outliers before computation.

   Logit Lens

   • Design: Project the residual stream representation output by each transformer block onto the vocabulary, answer with the highest probability among A/B/C/D, and evaluate layer by layer; Appendix G.3 also evaluates projections of attention outputs, intermediate representations, and MLP outputs.

   Enhanced GCG

   • Design: Adapted from FLRT, optimizing a universal adversarial prefix rather than a suffix, with the goal that the model does not recognize the dangerous knowledge from the start (once RMU detects dangerous context, it continuously injects noise into the residual stream); tokens earlier in the prompt are given greater loss weight.
   • Key parameters: Optimized on 6 WMDP questions that the original model answers correctly and the unlearned model answers incorrectly; Zephyr-7B-β is used as the "malicious model" to compute the representation matching loss, taking the first 10 generated tokens as ttarget and the next 25 as tmatch; the adversarial string is at least 100 tokens; for RMU, the representations of its unlearned layers 5, 6, and 7 are used, with weights decreasing linearly from 2 to 1.

   Set difference pruning

   • Design: Reproduce Wei et al. (2024b): use SNIP scores to measure neuron importance for dangerous knowledge on the WMDP forget set and utility scores on MMLU, take the set difference to identify neurons contributing only to dangerous knowledge, and prune them.
   • Appendix method: Appendix H additionally proposes an adaptive character perturbation targeting RMU (greedily inserting perturbations by the cosine similarity between tokens and the forget direction); Table 4 in Appendix E.1 summarizes the resources required by each method (whether the original model, forget set, weight or activation access is needed).

   Collapse this answer

4. 04What experiments did the paper conduct?

   On WMDP-Bio, all six method categories recover most of the knowledge: RMU direction ablation reaches 64.7%, exceeding the unprotected baseline of 64.4%.

   Experimental Setup

   • Models tested: Zephyr-7B-β (unprotected baseline), Zephyr_RMU (public checkpoint), and the authors' self-trained NPO and DPO versions of Zephyr-7B-β; Appendix H additionally tests Mixtral-8x7B-Instruct_RMU.
   • Benchmarks: WMDP-Bio (1,273 multiple-choice questions), WMDP-Cyber (1,987 questions), MMLU (57 tasks); the main results report WMDP-Bio, which the authors say is because the gap between the original and unlearned models on WMDP-Cyber is small and the results are more volatile (Appendix F).
   • Metrics: overall accuracy on the entire benchmark; the appendix uses perplexity to analyze generation quality.
   • Method variants: Logit Lens reports the best layer among all layers; Finetuning reports the best result among 5 samples from the forget set (Table 1 note).
   • Repetition and review: accuracy is the argmax of option probabilities, with no LLM review; the paper does not report multiple repetitions or confidence intervals (except for the perturbation experiment in Appendix H, which reports 95% confidence intervals).

   Main Results

   According to Table 1 (WMDP-Bio and MMLU accuracy, %):

   Swipe horizontally to read the table

   Method	Unprotected	RMU	NPO	DPO
   Default decoding	64.4	29.9	29.5	27.9
   Logit Lens	66.2	31.8	38.6	48.2
   Finetuning	-	62.4	47.4	57.3
   Orthogonalization	-	64.7	45.1	50.7
   Enhanced GCG	-	53.9	46.0	49.0
   Pruning	-	54.0	40.4	50.4

   • The authors believe that unlearning methods, like safety training, can be undone by the same kinds of techniques, and that knowledge is mainly masked rather than deleted.
   • The authors state that NPO has lower accuracy than RMU and DPO under most attacks, showing signs of "deep unlearning," but its MMLU drop is larger (under default decoding, NPO is 52.1 and RMU is 57.1).
   • Logit Lens is the only method that is largely ineffective against RMU: the authors state that RMU reduces Logit Lens accuracy to near random level, whereas DPO still reaches 48.2%, indicating that unlearning removes knowledge from the residual stream more effectively.

   Fine-tuning Sample Size Experiment (Figure 2)

   • What was tested: the effect of sample size (5–1000) and dataset (forget/retain/WikiText) on WMDP-Bio accuracy during LoRA fine-tuning.
   • Results: with only 10 samples from the retain set, most capability is restored: NPO 52.7%, DPO 57.0%, RMU 61.6%, with MMLU dropping by less than 2 percentage points; 1000 retain samples fully restore all three methods; 5 samples from the forget set restore RMU to almost the original level.
   • Authors' interpretation: both safety training and unlearning can be undone by fine-tuning on unrelated data; the authors believe this suggests that unlearning is also expressed through shallow features, and that knowledge is restored disproportionately quickly.

   Supplementary Experiment on Direction Ablation (Appendix F.3, Table 7)

   • What was tested: computing the unlearning direction using benign representations of the unlearned model on the WikiText or MMLU validation set instead of using the original model.
   • Results (WMDP-Bio): RMU reaches 54.6 with the mmlu direction and 52.5 with the wikitext direction, both lower than 64.9 with the original model's representations; NPO reaches 49.4 with the wikitext direction, higher than 45.7 with the original model; DPO is 56.1 with both the mmlu and original-model directions.
   • Authors' interpretation: considerable knowledge can be restored without access to the original model; for RMU, ablating only the layer-7 direction still restores 56.7%, and ablating only the forgotten layers 5, 6, and 7 restores all knowledge.

   Adaptive Perturbation Against RMU (Appendix H)

   • What was tested: greedily inserting character perturbations based on the cosine similarity between tokens and the layer-7 unlearning direction, until all token similarities fall below a threshold.
   • Results: at a threshold of 0.5, Zephyr_RMU differs from the unprotected baseline on WMDP-Bio by only 2.2 percentage points; the perturbed prompts also improve recovery for DPO and NPO; transferred to Mixtral-8x7B-Instruct_RMU, accuracy improves by up to 29% (Figure 12). The perplexity of RMU on responses generated by the original model drops from about 1600× under the unconditional setting to about 16× after perturbation (Table 12).
   • Authors' interpretation: the authors state that RMU is sensitive to keywords and injects noise in a heavy-side function manner (Appendix J: in prompts containing "virulence," the cosine similarity exceeds 0.8 starting from the 23rd token), and the perturbation can bypass detection.

   Other Ablations and Analyses

   • Proportion of key neurons for pruning: 2.0% of weights for NPO, 0.9% for RMU, 2.4% for DPO; after pruning, WMDP accuracy increases by at least 10 percentage points for each method.
   • Single-layer direction ablation: for NPO and DPO, single-layer ablation reaches at most 38 (layer 15), lower than layer-by-layer ablation; the authors state that each layer can detect dangerous inputs to some extent.
   • First principal component as the ablation direction: RMU recovers 53.7% under the chat template and only 34.1% without the chat template; for NPO and DPO it does not improve WMDP-Bio and MMLU drops by about 10 percentage points.
   • WMDP-Cyber (Table 6): the recovery magnitude is smaller, e.g., RMU fine-tuning 41.7 (baseline 27.7); full-layer ablation for NPO causes catastrophic forgetting (MMLU 25.6), and ablating only the layer-15 direction gives WMDP-Cyber 35.0 and MMLU 55.4.
   • Chat template sensitivity (Table 9): whether DPO training uses the chat template causes WMDP-Bio to fluctuate greatly between the two evaluation environments (e.g., Checkpoint 1: 28.7 with template, 46.8 without template); the gap for adversarial prefixes between the two environments is up to 20 percentage points.
   • Perplexity analysis (Appendix I): among the 24 adversarial prefixes, when accuracy is below 0.4, low perplexity correlates with high accuracy, but the prefix with the highest accuracy does not have the lowest perplexity; the authors believe this calls into question the practice of judging jailbreak quality by keyword matching or automated review.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out shortcomings such as the NPO implementation harming general capabilities, and propose improvement directions such as combining representation engineering with gradient ascent.

   Limitations and future directions noted by the authors

   • Cost of the NPO implementation (Discussion): The authors state that the current NPO implementation still causes a larger drop in MMLU and general capabilities.
   • Future direction one (Discussion): Future work could study combining representation engineering with gradient ascent to strengthen existing unlearning methods.
   • Evaluation of and alternatives to the RMU strategy (Discussion): The authors state that RMU's noise injection strategy is insufficient for open-source models, consider orthogonalizing harmful knowledge directions a more promising approach, and caution that noise-based unlearning may produce misleading preference results under GCG evaluation.
   • Other possible directions (Discussion): The authors mention that the method proposed by Hong et al. (2024) for precisely locating and modifying knowledge vectors may be effective.
   • Evaluation of TAR (Discussion): The authors state that the concurrent work TAR can defend against some fine-tuning attacks but has no effect on others.
   • The ECO analysis is preliminary (Appendix A.2): The potential bypass of the ECO detector is only a preliminary analysis and red-team draft, with no experiments conducted.

   Scope of experimental coverage

   • The base model tested is a single one: Zephyr-7B-β; RMU uses a public checkpoint, while NPO and DPO were trained by the authors; Appendix H additionally conducts perturbation transfer experiments on Mixtral-8x7B-Instruct_RMU.
   • The main results focus on WMDP-Bio; WMDP-Cyber results are in Appendix F, where the authors state that the gap between the original and unlearned models on this benchmark is small and the results are more volatile.
   • Knowledge recovery methods cover six categories: Finetuning, Orthogonalization, Logit Lens, Enhanced GCG, Pruning, and the perturbation in Appendix H (designed only for RMU).
   • Preference data was generated by GPT-4 and gpt-4o-mini-2024-07-18, with 10,000 samples per corpus; DPO/NPO training hyperparameters underwent a limited search (Appendix D, Table 2).
   • The paper does not report the number of repetitions or statistical significance tests for the main experiments; the optimization budget of Enhanced GCG (number of iterations, number of queries) is not given in the main text.

   Collapse this answer

6. 06Summarize the main content of the paper

   The white-box evaluation shows that RMU, like NPO and DPO, can have dangerous knowledge recovered by multiple methods; the authors believe unlearning mainly masks rather than deletes.

   This is a white-box adversarial evaluation paper on machine unlearning methods for AI safety, comparing RMU, NPO, and a DPO safety training baseline.

   • Problem: Unlearning claims to thoroughly remove dangerous knowledge from weights; the authors question whether it is fundamentally different from refusal-based safety training, and propose a criterion: if considerable WMDP accuracy can be recovered without updating the weights, or by updating the weights with data having almost no mutual information with the target knowledge, then unlearning is not successful.
   • Method: Under a white-box setting (able to modify weights, intervene in activations, and access the original model), six categories of recovery methods are applied: few-shot LoRA fine-tuning, layer-wise direction ablation, Logit Lens, Enhanced GCG prefix optimization adapted for unlearning, set difference pruning, and adaptive character perturbation targeting RMU.
   • Main results (WMDP-Bio, Table 1): RMU baseline 29.9%, reaching 64.7% via direction ablation (exceeding the unprotected baseline of 64.4%), 62.4% via fine-tuning on 5 samples from the forget set, 53.9% via Enhanced GCG, and 54.0% via pruning 0.9% of weights; NPO and DPO are likewise partially recovered by various methods.
   • Fine-tuning experiment (Figure 2): only 10 unrelated samples from the retain set restore RMU to 61.6%, DPO to 57.0%, and NPO to 52.7%, with MMLU dropping by less than 2 percentage points.
   • Exception: Logit Lens is near chance level for RMU; the authors state that unlearning does remove knowledge from the residual stream more effectively; NPO recovers the least under most attacks, which the authors consider a sign of deep unlearning, but at a greater cost to general capabilities.
   • Authors' conclusion: existing unlearning methods mainly mask knowledge rather than delete it, and share limitations with safety training; black-box evaluation is insufficient to determine whether unlearning is successful, and adaptive white-box evaluation is needed.

   Collapse this answer

Related papers