Collapse Irrelevant Representations (CIR) ensures robust and interference-free large language model unlearning

1. 01What problem does this paper attempt to solve?

   Forgetting hits shared general representations, so fine-tuning can restore it; CIR first collapses common subspaces and then updates.

   Existing forgetting hits overly general representations; dangerous knowledge is not removed, and fine-tuning can restore it.

   • Scenario: The authors state that pretraining causes models to learn capabilities usable for bioterrorism and cybercrime, as well as information about their own safety controls. RLHF and DPO only make the model disuse these capabilities; jailbreaks or benign fine-tuning can still re-invoke them.
   • Existing shortcomings: The authors state that dedicated forgetting methods can also be reversed by fine-tuning attacks and other adversarial methods. The introduction states that once general performance interference reaches 0.1%, forgetting becomes vulnerable to attacks; Section 3.1 operationalizes this as retain loss not exceeding 100.1% of its initial value, and uses it to explain the near-zero robustness observed in Deeb & Roger (2024) when retain loss interference of 5% is allowed.
   • Core observation: Weight updates are computed from activations and module output gradients, most of which are not representations specific to the fact being forgotten. In Figure 3, when forgetting one capital fact, the update cosine similarity with another capital fact reaches 84%; mismatched facts do not interfere in the same direction.
   • This paper's approach: Proposes Collapse of Irrelevant Representations (CIR), which uses PCA to find and collapse the common representation subspace, then computes the forgetting update, and switches to a breaking loss that directly acts on MLP outputs.
   • Attack in evaluation: The attacker fine-tunes on facts different from the evaluation facts but in the same category to recover the target facts; the paper states it does not assume the attacker has the full forgetting set. The forgotten model is Llama-3.1-8B, and post-attack WMDP accuracy is used for measurement.

   Collapse this answer

2. 02What related research exists?

   Related work includes activation interference, ablation, and meta-learning; baselines are Circuit Breakers and Gradient Difference.

   The paper organizes related work by forgetting approach, reversibility, and low-mutual-information attacks.

   • Backpropagation-only alignment: DPO (Rafailov et al., 2024) is classified by the authors as only disabling, not removing, unwanted capabilities, based on Lee et al. (2024). The introduction places RLHF in the same category.
   • Interfering with activations or ablating directions: Zou et al. (2024), Rosati et al. (2024), and Li et al. (2024) interfere with intermediate activations. Wang et al. (2024), Wu et al. (2023), Uppaal et al. (2024), and Suau et al. (2024) locate harmful neurons or activation directions and then ablate them. The authors cite Lo et al. (2024)'s result that after ablation, the model can quickly relearn using semantically similar neurons.
   • Meta-learning: Tamirisa et al. (2024), Sondej et al. (2025), and Henderson et al. (2023) simulate how attackers can relearn unwanted knowledge in order to counter it in advance.
   • Reversibility and low-mutual-information attacks: The authors state that existing forgetting techniques are easily reversible, citing Łucki et al. (2025) and Lynch et al. (2024), via fine-tuning, jailbreaks, few-shot prompting, disabling refusal mechanisms, or out-of-distribution inputs. Deeb & Roger (2024) recover supposedly forgotten facts by training on a completely separate set of facts; this paper's fine-tuning attack follows that setup.

   Baselines and Benchmarks: Gradient Difference (Liu et al., 2022) maximizes cross-entropy on the forgetting set and minimizes loss on the retain set; Circuit Breakers (Zou et al., 2024) pushes the cosine similarity between residual stream activations and initial activations to no less than 0. Evaluation uses WMDP-Bio and WMDP-Cyber; the retain set is the biology subset of FineFineWeb, or the computer science and technology subset; general performance uses WikiText loss.

   The authors position this paper after these works: instead of hard filtering in weight space, it first removes common representations in activations and gradients before updating; and it replaces Circuit Breakers' cosine loss and residual stream intervention point.

   Collapse this answer

3. 03How does the paper address this problem?

   CIR first collapses common components in MLP activations and gradients with PCA, then disrupts MLP outputs with a dot-product loss.

   CIR does not directly compute updates on activations and gradients containing shared representations; instead, it first collapses common subspaces, then uses the purified vectors for the outer product; the forgetting loss is changed to a dot product applied to MLP outputs.

   Why filter on activations and gradients

   • Interference and reversibility: The non-interference phase is defined as retain loss below 100.1% of its initial value; beyond that is the interference phase. In 50 runs in Figure 2, the authors report that forgetting after the threshold can usually be fine-tuned back, while before the threshold it remains robust. This 100.1% was tuned to be the highest threshold at which this relationship still holds.
   • Low-rank stripes: The update is obtained by activation times gradient, approximately low-rank, with interference appearing as whole rows and columns. In Figure 4, unmasked disruption/transfer is 58%, weight-masked is 33%, and whole-column and whole-row masked is 5%.
   • Superposition: Element-wise ablation still cannot distinguish relevant from irrelevant directions. The authors attribute this to representation superposition (Elhage et al., 2022), so they instead project out common directions and approximate irrelevance by frequency: representations common across many training texts are treated as unrelated to the fact to be forgotten.

   Collapsing common subspaces

   For each trained MLP, take input activations and the module output gradients from backpropagation, first subtract the mean direction; the mean is treated as the 0th principal component and removed first during collapse. The remaining principal components are

   a_pure=a′−∑_i=1^k(a′· PC_i)PC_i

   where a' is the activation with the mean direction removed; gradients are processed similarly. The authors take 24 activation principal components and 36 gradient principal components per Figures 6a, 6b, and report a plateau between 12 and 48, with removing the first few activation principal components being critical. Principal components are recomputed each forgetting epoch; they can be estimated on any data, with the authors reporting best on the forgetting corpus, and the same forward and backward pass can be reused. Only MLPs are intervened on, not attention.

   Loss

   On the logits side, negative cross-entropy, negative entropy, and clamping the target token logit to not go below 0 were tried. On the representation side, Circuit Breakers clamps the cosine similarity with the initial activation on the residual stream to not go below 0. The authors change two things: cosine can also be lowered by adding a random direction, so they switch to dot product; and they do not hit the residual stream but the MLP output before it is added to the residual stream. The final forgetting loss is

   L=ReLU(m· m_orig)/|m_orig|²̅

   that is, the dot product of the current MLP output with the original output, clipped at 0 with ReLU, divided by the average of the squared norm of the original MLP output, to avoid later layers with larger norms dominating the loss. Representations are not disrupted at the BOS position. On the retain set, the norm of the change in the residual stream relative to the original activation is penalized, following Circuit Breakers. The loss acts on layers 6–12 (the 32-layer Llama 8B in the paper); the authors report that only the forward and backward passes of the first 12 layers are needed.

   Update procedure

   Algorithm 1: each epoch iterates over forgetting samples, extracts and caches each MLP's activations and gradients; only if principal components already exist, collapse and do einsum updates, optionally training one retain batch. After the epoch, PCA is done on the cache and cleared. Per the pseudocode, the first epoch only collects and computes principal components. The authors follow the practice of normalizing the norm of the forgetting update to a fixed value, which is the forgetting rate.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On Llama-3.1-8B, the authors state the post-attack drop is 80x the baseline for Bio and 30x for Cyber.

   Experimental setup

   • Model: Main experiments use Llama-3.1-8B. Forgetting sentences are generated by gpt-4.1. Interference examples in Appendix A use Llama-3.2-1B. Appendix D uses claude-4.1-opus to see if question text can predict non-robustness, and requires not hinting specific patterns to it.
   • Data: Further filtered from Deeb & Roger's WMDP-Deduped: answers shorter than 60 characters, removing none/all of the above, and removing questions where Llama-3.1-8B accuracy is worse than random, yielding 144 bio and 203 cyber questions. Each question generates 20 sentences, with 3 used for training. Questions are split 20/80 into development and held-out sets, with results reported on the held-out set: 112 bio and 165 cyber. The retain set is the biology subset of FineFineWeb, or the computer science and technology subset.
   • Baselines and stopping lines: Gradient Difference and Circuit Breakers. General performance uses WikiText loss. CIR stops when it exceeds 100.1% of its initial value. Baselines drop accuracy at most 1 percentage point at the same threshold, so the authors put the baseline stopping line at 103%, calling it a 30× concession, and retune hyperparameters.
   • Attack: After stopping, 100 epochs of fine-tuning. Per Deeb & Roger's split: forgetting uses 100% of data, relearning uses 80%, evaluation uses the remaining 20%; relearning facts are different from evaluation facts but same distribution.
   • Metrics: WMDP accuracy. Post-attack accuracy takes the peak of the smoothed relearning curve, because some attacks are longer than optimal; Appendix C states 10-epoch binning for denoising. Figure 6 accuracy is at temperature=1. The paper does not state the temperature for main figures, nor does it use an LLM to score WMDP answers.
   • Tuning and hardware: Single 40GB A100; caching activations and gradients up to 48GB RAM. For each method, first manually choose a high but safe retain learning rate, then search forgetting rate: 5 runs each, 3 per order of magnitude, taking the one that does not diverge and has the highest post-attack accuracy. If the interference threshold is not reached, forgetting runs at most 200 epochs; one Gradient Difference run is allowed longer. Specific learning rate values are not given in the paper.

   Main results

   Condition	Authors report	Source
   WMDP-Bio, CIR vs best baseline	Post-attack accuracy drop is 80× the baseline; WikiText interference 30× smaller	Section 6, Figure 1b
   WMDP-Cyber, CIR vs baseline	Post-attack accuracy drop is 30× the baseline; robustness 30× higher under 0.1% budget	Section 6, Figure 5
   Abstract's combined claim	Post-attack accuracy drop exceeds Circuit Breakers by 30×; general performance interference 30× smaller; less than 3 GPU seconds per fact	Abstract
   No retain training, 0.1% budget	Bio post-attack accuracy 51%, Cyber 41%	Section 6
   Baseline stops at WikiText 100.1%	Accuracy drop at most 1 percentage point	Section 5

   • The authors report CIR has a wide range of usable hyperparameters; the other two methods have forgetting and retain terms pulling against each other. Figure 5 caption states Circuit Breakers shows sudden regression, with retain loss offsetting previous gains, and relearning from the lowest accuracy is less robust. Figure 1b caption states CIR achieves near-ideal robustness to relearning.
   • The authors report Gradient Difference is poor, mainly because retain-set training cannot suppress WikiText interference, and often cannot suppress it on retain-set evaluation splits either.
   • When CIR's allowed interference is increased to 1% (baselines use 3%, CIR usually cannot reach that threshold), the authors report the extra gain in post-attack accuracy is disproportionately low. A footnote gives another possibility: accuracy may stop decreasing once at random level, but the probability of generating harmful answers (not shown) is still decreasing, so forgetting is still happening, just not robustly.

   Interference threshold and transfer across facts

   • 50 runs: Figure 2 uses CIR with various losses, learning rates, projection counts, sometimes without retain training, all on Llama-3.1-8B and WMDP-Cyber, with attacks all converging. The authors report attacks can at most restore WMDP accuracy to the level at the 0.1% retain-loss threshold. Here 0.1% is retain loss, not the same measurement as the WikiText stopping line in the main experiments.
   • Masking: Figure 4 compares update slices on a single capital fact; unmasked, weight-masked, and whole-column and whole-row masked disruption/transfer are 58%, 33%, and 5% respectively.
   • Other facts and languages: Figure 3 (middle-layer MLP, token before the answer) shows other capital facts at 84%, 84%, 64%; unrelated facts at 32%, 29%, 6%, 6%; wrong pairings at -5%, 19%, -24%, 1%. Appendix A on Llama-3.2-1B reports that changing the question wording still causes interference, with more distant facts around 8%; languages with similar word forms transfer about 50%, with Russian and Portuguese weaker.

   Layers, principal components, and disruption methods

   • Figure 6 reports WMDP-Cyber accuracy after attack convergence, temperature=1, no retain training. Figure 6c at 0.5% allowed interference: circuit breaking only is 55.2, 57.2, 57.3, 56.5, 57.0; CIR plus circuit breaking is 51.0, 43.7, 52.3, 52.9, 51.4; CIR plus MLP breaking is 47.4, 37.4, 44.5, 48.2, 41.0, with layer ranges [0, 6] to [24, 30] respectively. The authors report layers 6–12 are most effective, and hitting MLP outputs improves the forgetting–interference tradeoff by 40%; the introduction's corresponding claim is selectivity improved by 40%.
   • The authors choose 24 activation principal components and 36 gradient principal components, and report a plateau between 12 and 48. Figure 6 note: projecting 1 component only removes the mean, not a true PC, saving computation but performing poorly. Figure 6a is 0.2% allowed interference, CIR with Circuit Breakers loss, layers 6–15; 6b is 0.5% allowed interference. The main text does not separately list the selected cell in these two grids.
   • Figure 8 compares dot product, minimizing activation norm, and cosine similarity on CIR, Llama-3.1-8B, layer 6 activation norms. The authors report that with the same forgetting amount, dot product has less interference, and cosine methods increase activation norm. The main text does not give exact coordinates for each point.

   Other ablations and analyses

   • Table 2 (best forgetting runs, with per-epoch evaluation): CIR is Bio 5 minutes/16 epochs, Cyber 5 minutes/10 epochs; Circuit Breakers is 16 minutes/95 epochs, 45 minutes/161 epochs; Gradient Difference is 106 minutes/409 epochs, 14 minutes/31 epochs.
   • Excluding evaluation, CIR on Bio held-out set is 251 seconds/112 facts, i.e., 2.24 seconds/fact; Cyber is 201 seconds/165 facts, i.e., 1.22 seconds/fact. Elsewhere the paper reports runs with forgetting plus relearning take 15–120 minutes.
   • Using more than 3 sentences per question hurts forgetting; the authors think it may be because later sentences are lower quality.
   • The authors report that directly clamping the target token logit to not go below 0 is better than negative cross-entropy and negative entropy, preventing recall but not generalizing to recognition in multiple choice; hitting intermediate representations removes both recall and recognition.
   • Appendix D: among the 21 WMDP-Cyber questions where the attack improves accuracy, the correct answer is the longest option in 52%, and the remaining 7 questions are written as 14%. Section 8 states the proportion is 52%, with the remaining questions at 14%. The two statements are not exactly identical.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that WMDP questions being guessable, fact scale, and tendency forgetting remain future work.

   Limitations and Future Directions Noted by the Authors

   • Guessable Questions (Section 8): The authors suspect some non-robustness arises because some WMDP questions can be guessed without the answer. In WMDP-Cyber, among questions where the attack improves accuracy, the correct answer is the longest option 52% of the time, versus 14% otherwise. The authors state that fixing this could push post-attack accuracy further toward random.
   • Number of Facts (Section 8): This study only targets facts in WMDP. Scaling to full biological and cybersecurity forgetting would require an order of magnitude more facts.
   • Where Data Comes From (Section 8): The authors state the bottleneck is a lack of high-quality forgetting data; existing biological and cybersecurity forgetting corpora are mostly harmless text; better data requires substantial expert effort. Public release poses safety risks, and both creation and use require coordination by specialized institutions.
   • Tendencies Rather Than Knowledge (Section 8): The common-is-irrelevant assumption holds when forgetting knowledge because relevant representations are fact-specific and relatively rare. For forgetting tendencies such as power-seeking or deception, harmful representations are often common in training text, and collapsing targets need to be finer than PCA. The authors leave this to future work.

   Experimental Coverage

   • The tested model for main results is Llama-3.1-8B; the fact-interference examples in Appendix A additionally use Llama-3.2-1B.
   • The held-out set is 112 WMDP-Bio questions and 165 WMDP-Cyber questions; the forgetting set is 3 sentences per question; the retain set is the corresponding FineFineWeb subset; the general performance stopping line uses WikiText loss.
   • Comparisons are Gradient Difference and Circuit Breakers. CIR stops at 100.1% of WikiText loss; the baseline stops at 103%.
   • The robustness experiment is 100 epochs of fine-tuning, with facts different from the evaluation facts but in the same category (Section 5). Figure 2 includes 50 CIR variants, and all attacks converge.
   • The paper does not report absolute post-attack accuracies for Figure 1b and Figure 5; the main text gives relative multiples computed by the authors; Appendix C reports, for each method, the run that did not diverge and had the highest post-attack accuracy. The learning rate values for the hyperparameter search are not given in the paper.

   Collapse this answer

6. 06Summarize the main content of the paper

   CIR forgets after collapsing common representations; the authors report that with less interference, fine-tuning finds it harder to recover WMDP facts.

   CIR removes common representations before computing weight updates, to forget dangerous facts and reduce the chance that fine-tuning can learn them back.

   • The authors attribute failure to general representations shared by harmful and harmless capabilities: these representations also appear in the attacker's training data, and once damaged, they can be repaired. They define the phase where retain loss is below 100.1% of its initial value as non-interference, and report that only forgetting in this phase remains robust to fine-tuning.
   • The method is to perform PCA on the input activations and output gradients of the trained MLPs, collapse the mean and common principal components, then do the outer product. The forgetting loss is changed to a dot product of the MLP output relative to the original output, clipped with ReLU, and divided by the average of the squared norm of the original output; the retain loss constrains the residual stream.
   • The main experiments only test Llama-3.1-8B. The authors report that with a setting where WikiText interference is 30× smaller than the baseline, the post-attack accuracy drop on WMDP-Bio is 80× that of the best baseline, and on WMDP-Cyber it is 30×. The abstract combines the two categories as exceeding Circuit Breakers by 30×, with less than 3 GPU seconds per fact.
   • With the same 0.1% WikiText budget and no retain training, post-attack accuracy is 51% for bio and 41% for cyber. If the baseline also stops at 100.1%, accuracy drops at most 1 percentage point, so the experiments give the baseline a 103% stopping line.
   • The authors conclude that only by separating harmful and harmless capabilities at the representation level can forgetting both resist relearning and minimally damage general performance. They also suspect WMDP has guessable questions, and believe full-domain forgetting lacks high-quality data, and PCA is insufficient for forgetting tendencies.

   Collapse this answer

Related papers