Sensitive knowledge protection in LLM unlearning: agent threats and layered-aware defenses

1. 01What problem does this paper attempt to solve?

   A legitimate unlearning request can, through knowledge superposition, incidentally delete unrequested sensitive entities; the paper formalizes this as PUA and proposes the defense SAUP.

   Core problem: representations of different entities in an LLM may share parameter subspaces (knowledge superposition), so deleting one entity also damages other entities, and legitimate machine unlearning requests can therefore be exploited to delete sensitive entities that were never requested.

   • Scenario and importance: LLM providers face deletion needs for privacy, copyright, factual correction, and safety, and machine unlearning can remove specified entities without retraining. The authors argue that the collateral damage of unlearning is not merely a utility problem but an overlooked security threat that can lead to misinformation or social bias (e.g., the model misdescribing a brand, demographic group, or medical concept).
   • Shortcomings of existing methods: the authors state that existing unlearning research focuses mainly on whether deletion is effective and whether utility is preserved, and that standard verification only confirms that the requested entity has been deleted; the entity named in the request need not be similar to the affected sensitive entity, and the model often still answers fluently but with incorrect content, so the damage is silent.
   • Core observation: entities are not encoded in isolation; representations of different entities can overlap in the same parameter subspace (knowledge superposition), and this coupling does not require surface or obvious semantic relatedness, so it can affect entities that appear unrelated to the request.
   • Threat model:
     • Goal: selectively damage the knowledge of a sensitive target set T without explicitly requesting deletion of the target entities, rather than broadly degrading model capability.
     • Knowledge: in the white-box setting (open weights), internal model representations can be read to precisely identify superposed entities; in the black-box setting (unlearning API only), only the model can be queried, and superposition is estimated with the help of a same-family proxy model.
     • Capability: can submit requests from the deletable set U only through the legitimate deletion interface, and cannot directly modify model parameters or retrain.
     • Victim system: deployed LLMs offering entity-level unlearning services and their providers.
   • What the paper proposes: it formalizes the above threat as the Proxy Unlearning Attack (PUA) and proposes a provider-side defense framework, Superposition-Aware Unlearning Protection (SAUP), which detects threatened entities before unlearning, decouples shared representations and selectively recovers after unlearning, and does not modify the unlearning algorithm itself.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: LLM machine unlearning methods, attacks that exploit unlearning, and studies of the non-locality of unlearning; the authors state that PUA requires no pre-deployment poisoning or backdoors.

   LLM machine unlearning methods

   • Gradient ascent class—achieves unlearning by maximizing the loss on the target knowledge [4]; preference optimization class models unlearning as steering the model away from undesired outputs [9]; modular design avoids dependence on the retain set [10].
   • LLM-Eraser [11]—localizes knowledge to specific parameters and selectively prunes them; other work uses retrieval-augmented generation to decouple internal knowledge from outputs, redirecting rather than deleting at the parameter level [12].
   • Unlearning evaluation research—mainly focuses on whether the target knowledge is deleted and whether utility is preserved; some work notes that deletion may depend on the question form, and that knowledge remains recoverable under equivalent phrasings [13].

   Attacks that exploit machine unlearning (the paper divides them into three types by harm)

   • Activating backdoors via unlearning: Liu et al. [17] poison training data so that subsequent unlearning requests activate a pre-implanted backdoor; Arazzi et al. [18] still induce a backdoor in a clean setting with no explicit trigger in the forget set; Shang et al. [19] exploit attention sinks so that knowledge is conditionally recovered under specific triggers after unlearning; Lu et al. [20] extend this to federated learning.
   • Adversarial manipulation of unlearning: Qian et al. [21] use constructed deletion requests to induce targeted misclassification in traditional deep models; Zhou et al. [22] attack by optimizing the geometry of the loss surface; Marchant et al. [23] use poisoning to weaken certified unlearning guarantees; Ye et al. [24] use training-time data repetition to simultaneously break multiple safety properties of unlearning.
   • Privacy leakage and reconstruction after unlearning: comparing the original and unlearned models enables inversion and reconstruction attacks [25]–[27] and amplifies membership inference [28]; on the defense side there are representation-level unlearning and posterior-difference auditing [29], [30].

   Non-locality of unlearning and knowledge superposition

   • Some studies find that deleting a target affects knowledge related to its inferences [14] or shared by the forget/retain sets [15], and other work reduces the over-unlearning caused by the unlearning process itself [16]. The paper notes that this line treats collateral damage as a quality and evaluation issue, and that the knowledge involved is related to or shared with the target.
   • Knowledge superposition research [31]–[33] notes that networks encode features with overlapping directions; the knowledge editing field has used similar whitening methods to study superposition [6].

   Baselines and benchmarks: the attack control is a random proxy request matched to the scale of the PUA request (Table II); the defense control is the output-level detection-and-recovery baseline PDDR (Tables V, VI). The target datasets are CounterFact, UMLS, SBIC, and the deletable set comes from Wikidata 5M entities.

   The authors' positioning of this paper: the authors state that existing attacks either require pre-deployment adversarial preparation (poisoning, backdoor implantation, data repetition) or exploit comparisons between the states of the original and unlearned models; PUA requires no pre-deployment access to training data or model internals, is carried out entirely through the legitimate request interface, targets sensitive entities that never appear in any request, and elevates the collateral damage of unlearning from a utility issue to a targeted security risk.

   Collapse this answer

3. 03How does the paper address this problem?

   PUA uses KSD to select, at the representation level, proxy entities that superpose with the sensitive target and submits them for unlearning; SAUP adds detection and selective recovery before and after unlearning.

   Overall approach: first use Knowledge Superposition Detection (KSD) to quantify, at the representation level, the degree of superposition between any two entities; the attacker uses this to select proxy entities and submit legitimate unlearning requests (PUA); the defender uses the same tool to estimate threatened entities before unlearning and perform selective recovery after unlearning (SAUP), without modifying the unlearning algorithm itself.

   Problem formalization

   • The unlearning operator Unlearn(θ, Q) should make the knowledge measure K(·, e) of the requested entities e∈Q smaller while leaving other entities unchanged; superposition breaks this selectivity: deleting Q also lowers the knowledge of e′ that superposes with q∈Q.
   • The attack objective is written as max_{S⊆ U, S∩ T=∅} ∑_T∈𝒯 [K(θ, T) − K(Unlearn(θ, S), T)], i.e., maximizing the damage to the sensitive target set T without ever requesting the targets; the authors state that this objective defines the threat outcome, and that in practice PUA approximates it through superposition detection and does not measure damage after unlearning.

   KSD: Knowledge Superposition Detection (Algorithm 1)

   • Representation extraction: for an entity e, instantiate a shared probe template set P, run the model forward, read the hidden states of the entity-name token at the post-attention layer normalization of each transformer layer, and average across probes to obtain per-layer representations. The probe templates are a required input, and the two entities must be measured under the same templates.
   • Whitening: raw representations are anisotropic, so ordinary cosine similarity is unreliable; compute the covariance independently for each layer and apply a whitening transform so that pairwise similarity becomes a meaningful signal. The authors note that similar whitening has been used in superposition research in knowledge editing.
   • Superposition determination: compute the whitened per-layer cosine similarity ρ(l); if ρ(l) ≥ τ at any layer, the two entities are judged to superpose; record the set of superposition layers and the coverage count c (the number of layers exceeding the threshold). The experiments set τ = 0.9, counting only strong and consistent superposition.

   PUA: proxy selection and request submission

   • Three steps of proxy selection: run KSD between T and the deletable set U; filter by coverage count, keeping candidates with cmin ≤ c(T, s) ≤ cmax (the lower bound removes accidental associations from only one or two layers, and the upper bound removes hub entities that overlap with almost all entities); take the top-k by peak whitened cosine as the proxy set S. In evaluation, only targets with at least k candidates after filtering are kept, so that each attack has a proxy set of size k.
   • Two access levels: in the white-box setting, run KSD directly on the target model; in the black-box setting, run KSD on a proxy model from the same family and transfer the selected proxies. The authors state this is feasible because models trained on similar data form similar superposition structures.
   • Request submission: submit S as an ordinary unlearning request; since superposition does not track semantic or lexical similarity, the proxies need not be similar to the target, and the request itself does not expose the target.

   SAUP: pre-unlearning damage estimation

   • The provider maintains a protected entity set P; upon receiving a request Ur, it runs KSD between Ur and P and collects the protected entities that superpose with at least one entity in Ur as the vulnerable subset V. Any protected entity that superposes with Ur risks collateral deletion, and the provider need not know the requester's intent.
   • Record, for each pair of entities between V and Ur′ (the subset of the request that superposes with V), the set of superposition layers; also maintain a retain set R unrelated to both P and Ur, used in the recovery stage to protect unrelated entities.

   SAUP: post-unlearning selective recovery

   • Constrained recovery objective: single-stage joint optimization L_SAUP = λ_V L_V + λ_R L_R − λ_U L_U + λ_dec L_dec. LV restores the entities in V; LU enters with a negative sign, performing gradient ascent on the requested entities to prevent recovery from incidentally reviving them; LR is the retain replay regularizer; Ldec is the superposition decoupling term. The authors state that a two-stage scheme of decoupling first and then recovering is insufficient, because the recovery gradients re-couple the directions just separated.
   • Decoupling objective Ldec: for each pair (v, u), expand the superposition layers detected by KSD into a neighborhood of radius δ (because residual connections propagate activation changes across adjacent blocks), and on these layers penalize the cosine alignment between the whitened representations of v and u, using stop-gradient to anchor the vulnerable representation and move only the requested representation, driving the two toward orthogonality.
   • Projected gradient update: when the recovery gradient gV is positively correlated with the maintenance gradient gU, project out of gV the component that would also restore Ur′; when the gradients are opposite, keep the full gV. Recovery is organized by KSD unit (one v-u pair and its superposition layers), sampling one unit and one retain batch per step, and applying the projected combined update in a single stage.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Across two model families × three knowledge domains, PUA lowers the target Judge score by 0.15–0.57 and is black-box transferable; SAUP restores the target and keeps the proxies suppressed.

   Experimental setup

   • Models under test: LLaMA-3.1-8B-Instruct and Qwen-2.5-3B/7B/14B-Instruct; the main tables report LLaMA-3.1-8B and Qwen-2.5-7B, and 3B and 14B serve as same-family proxy models in the black-box experiments.
   • Data: the deletable set U is sampled from Wikidata 5M entities, taking three nested subsets of 10K/50K/100K; targets come from three datasets: CounterFact (general facts), UMLS (biomedical terms), and SBIC (socially sensitive knowledge). Each entity must pass a probe confirming that the baseline model scores at least 0.5 on each of the three probes to be included.
   • Entity groups: target group T (never submitted for deletion), proxy group S (entities marked by KSD as highly superposed with T, whitened cosine ≥ 0.9 in at least one layer, i.e., the entities the attacker actually submits), and retain group R (generic entities from U, disjoint from T and S, split into training and held-out parts as a selectivity control). In the defense experiments, the protected set P is the entire scenario dataset.
   • Unlearning methods: Gradient Ascent (GA) and Negative Preference Optimization (NPO), both receiving the same proxy set S; the unmodified model is denoted Baseline.
   • Metrics and judging: each entity is probed along three dimensions: definition, category, and association; the LLM Judge Score is rated by an independent judge model, Qwen-2.5-32B-Instruct, on factual correctness with {0, 0.5, 1}; Ref-NLL is the per-token negative log-likelihood of the evaluated model on the baseline reference answer (undefined for the Baseline row, recorded as –). Metrics are averaged over the three dimensions to the entity, then averaged over the groups (T/S/R).
   • Main setting: white-box access, |U|=100K; the paper does not report the number of repetitions or statistical significance tests.

   Main results

   According to Table I (each cell is Judge / Ref-NLL; Ref-NLL is undefined for Baseline):

   Swipe horizontally to read the table

   Model	Scenario	Method	T	S	R
   LLaMA-3.1-8B	CounterFact	Baseline / GA / GA+SAUP	0.842/– / 0.498/78.744 / 0.800/0.476	0.830/– / 0.377/101.814 / 0.590/16.088	0.760/– / 0.588/30.674 / 0.687/1.158
   LLaMA-3.1-8B	UMLS	Baseline / GA / GA+SAUP	0.830/– / 0.505/9.980 / 0.810/0.102	0.810/– / 0.522/8.741 / 0.655/4.686	0.743/– / 0.670/1.289 / 0.660/1.052
   LLaMA-3.1-8B	SBIC	Baseline / GA / GA+SAUP	0.850/– / 0.281/55.379 / 0.781/0.036	0.787/– / 0.382/54.167 / 0.460/7.721	0.743/– / 0.650/7.862 / 0.587/1.271
   Qwen-2.5-7B	CounterFact	Baseline / GA / GA+SAUP	0.840/– / 0.695/2.455 / 0.770/0.888	0.828/– / 0.670/2.727 / 0.690/1.778	0.725/– / 0.648/1.858 / 0.640/0.960
   Qwen-2.5-7B	UMLS	Baseline / GA / GA+SAUP	0.925/– / 0.683/1.105 / 0.848/0.092	0.828/– / 0.718/1.032 / 0.703/2.321	0.735/– / 0.685/0.351 / 0.670/0.971
   Qwen-2.5-7B	SBIC	Baseline / GA / GA+SAUP	0.899/– / 0.623/4.286 / 0.839/0.108	0.817/– / 0.635/4.708 / 0.671/2.338	0.732/– / 0.678/1.280 / 0.677/0.706

   (Table I also contains NPO and NPO+SAUP rows, omitted here; under NPO, LLaMA's T across the three domains is 0.485/0.442/0.314, and Qwen's is 0.685/0.567/0.626.)

   • The attack degrades unrequested targets: the authors report that the target group degrades in all six model–domain settings, with the largest being a Judge drop of 0.569 for LLaMA+SBIC+GA (0.850→0.281); LLaMA targets drop by 0.33–0.57 and Qwen by 0.15–0.36.
   • The damage is selective: the retain group R is affected much less (LLaMA 0.588–0.700, Qwen 0.638–0.685); the authors consider R's small decline to come from the unlearning method itself rather than the attack.
   • Damage tracks KSD superposition rather than request size: in Table II, averaged over all main settings, PUA-selected proxies reduce T by 0.331 and move R by only 0.082 (selective damage 0.249), whereas size-matched random proxy requests reduce T by only 0.052 (selective damage 0.025).

   Black-box transfer (Table III)

   • What was measured and the results: with Qwen-2.5-7B as the target, the attacker selects proxies on the same-family Qwen-2.5-3B or 14B (|U|=100K). With the 3B proxy, the target reduction ∆T is 0.090–0.168, below the white-box reference (0.145–0.358); with the 14B proxy, transfer is stronger, exceeding the white-box reference under GA on CounterFact (∆T=0.160 vs white-box 0.145) and UMLS (0.253 vs 0.242), and approaching it on SBIC (0.243 vs 0.276).
   • Authors' interpretation: a small proxy model is already sufficient for the attack to succeed, so representation-level access to the target is not necessary; a stronger same-family proxy model can match or exceed white-box results.

   Attacker budget and unlearning strength (Figure 4–6)

   • Deletable set size: Figure 4 shows that at |U|=10K, KSD has already found proxies for 63 SBIC and 273 UMLS targets, rising to 86 and 1069 at 100K; the authors state that the attack is feasible without a large budget, and that a larger budget mainly expands the number of reachable targets.
   • Proxies per target: Figure 5 shows that on LLaMA+SBIC the target reduction rises from 0.117 with 1 proxy to 0.417 with 16, while Qwen+SBIC saturates at 8 proxies (0.117–0.133); the authors state that per-target severity saturates once the target's superposition directions are covered.
   • Unlearning strength: in Figure 6 (SBIC, LLaMA, white-box, |U|=100K), under GA, ∆Judge(T) rises from 0.066 at 1 epoch to 0.855 at 8 epochs, and under NPO from 0.087 to 0.784; R's ∆Judge reaches 0.135 at GA 7 epochs and stays below 0.10 under NPO. The authors state that reducing strength also weakens the intended deletion of S, leaving providers in a dilemma; some small target reductions in the main table are because weak unlearning updates do not sufficiently delete the proxies, and do not mean the attack failed.

   Defense evaluation (Table I +SAUP rows, Table IV–VII)

   • Recovery effectiveness: on LLaMA, SAUP raises T in all six cells while maintaining a positive T−S gap, with a six-cell average of T=0.778, S=0.562, gap 0.216, R=0.654; on Qwen it raises T in all six cells, with an average T−S gap of 0.134 (T=0.805, S=0.671, R=0.684), the smallest being 0.080 for CounterFact+GA. On Ref-NLL, LLaMA targets drop from 55.379 to 0.036 (SBIC+GA), among others, while proxy Ref-NLL remains high.
   • Cost of proxy reversal: Table IV shows that recovery partially reverses unlearning of the proxies, with S rising by 0.213 (GA) and 0.291 (NPO) on LLaMA+CounterFact; on Qwen, four cells rise by no more than 0.020, with a maximum of 0.075 (SBIC+NPO). The authors state that SAUP turns targeted failure into a measurable trade-off between target protection and strictly completing proxy unlearning.
   • Comparison with output-level detection PDDR: in Table V, PDDR misses 29.9% (UMLS), 24.6% (SBIC), and 35.2% (CounterFact) of KSD-marked targets under the definition probe, and still misses 16/194, 3/61, and 49/267 on the union of all probes. In Table VI (LLaMA+NPO), SAUP's T−S gap is larger than PDDR-style recovery in all three domains (UMLS 0.165 vs 0.103, SBIC 0.291 vs 0.080, CounterFact 0.155 vs 0.052); PDDR has higher raw T on UMLS and CounterFact (0.805, 0.797) but S also rises to 0.702 and 0.745.
   • Recovery objective ablation: in Table VII (SBIC+NPO, LLaMA, fixed target subset), with LV only, T=0.883 but S returns to 0.583 (gap 0.300); without Ldec, S=0.450 and the gap is 0.367; full SAUP gives T=0.800, S=0.367, gap 0.433; without LR the gap is 0.450 but R drops to 0.517.

   Other ablations and analyses

   • Superposition visualization (Figure 7–9, SBIC, LLaMA, 30 T–S pairs linked by KSD): in Baseline, the entity-name representation match similarity is 0.9996 and the QA aggregate is 0.2628; after PUA the QA aggregate rises to 0.3166 (GA)/0.2815 (NPO); after SAUP it drops to 0.0340/0.0343, and entity names drop to 0.0412/0.0867. The control relations T–R and S–R are 0.1296 and 0.1261 in Baseline (GA), and after SAUP all three fall into the same low range. The authors state that superposition exists before the attack and is specific to KSD-linked pairs.
   • Lexical and semantic overlap (Table VIII, 10078 KSD-linked proxy pairs in total): 3-gram Jaccard lexical overlap averages 0.072 (UMLS 0.025, SBIC 0.016), label-embedding semantic similarity averages 0.657, and KSD superposition averages 0.991. The authors state that KSD does not select obvious near-synonym neighbors, and that screening only semantically similar requests would miss many vulnerable links.
   • Qualitative analysis (Figure 10–11, Table IX–X): under PUA, the model gives descriptions of unrelated entities or repeats loops for target entities, and in safety-related questions shifts from refusal to compliance with fabricated reasons; SAUP restores baseline answers and refusal behavior. The authors state that the damage is silent and the model usually still answers fluently.
   • Negative results and exceptions: on Qwen, SAUP's T−S gap is overall smaller than on LLaMA (0.134 vs 0.216); the reversal of proxy unlearning by recovery is largest on LLaMA+CounterFact (0.213–0.291); the paper gives no explanation for these exceptions beyond the quoted text above.

   Collapse this answer

5. 05What are some points that can be further explored?

   The paper has no dedicated limitations section; the experiments cover two model families, three knowledge domains, and two unlearning methods, and black-box transfer is validated only within the same family.

   Limitations and future directions noted by the authors

   • The paper does not specifically discuss limitations (there is no Limitations or Future Work section).
   • In VII-C the authors note that SAUP changes the provider's operating point: it turns targeted failure on protected entities into a measurable trade-off between target protection and strictly completing the proxy unlearning request, and recovery partially reverses the submitted proxy unlearning (Table IV).
   • In VII-B the authors note that the provider faces a dilemma: lowering unlearning strength also weakens the due deletion of the requested entities, while raising strength simultaneously strengthens both the intended unlearning and the collateral damage.
   • In VII-C the authors note that output-level detection (PDDR) can only identify visible degradation and cannot replace representation-level KSD detection.

   Experimental coverage

   • The models tested are LLaMA-3.1-8B-Instruct and Qwen-2.5-3B/7B/14B-Instruct, four in total; the main results table reports two, LLaMA-3.1-8B and Qwen-2.5-7B (Table I).
   • The knowledge domains cover three target datasets: CounterFact, UMLS, SBIC; the deletable set comes from Wikidata 5M entities, with sizes of 10K/50K/100K (VII-A).
   • The unlearning methods cover two, GA and NPO; SAUP claims to apply to any unlearning algorithm that accepts entity-level requests, but the experiments pair it only with these two (VII-A).
   • Black-box transfer is validated only within the Qwen family: target Qwen-2.5-7B, proxy models Qwen-2.5-3B and 14B (Table III); the paper does not report cross-family transfer experiments.
   • The paper does not report SAUP's computational overhead, runtime, or the specific values of hyperparameters (λV, λR, λU, λdec, δ, learning rate), nor does it report the number of repetitions or statistical significance tests.

   Collapse this answer

6. 06Summarize the main content of the paper

   The paper formalizes the proxy unlearning attack PUA and proposes the provider-side defense SAUP, verifying on two model families and three knowledge domains that the attack is effective and can be selectively recovered.

   An attack and defense paper on the security of LLM machine unlearning.

   • Problem addressed: different entities in an LLM may share representation subspaces (knowledge superposition), so deleting one entity also damages other superposed entities; a legitimate unlearning request can therefore be exploited to delete sensitive entities that were never requested, while standard verification checks only the requested entity and the damage is silent.
   • Method summary: the attack PUA uses KSD (whitened per-layer representation cosine similarity, threshold τ=0.9) to select, from the deletable set, proxy entities superposed with the sensitive target, and submits them as ordinary unlearning requests; in the black-box setting it uses a same-family proxy model to select proxies and transfer. The defense SAUP marks threatened protected entities with KSD before unlearning, and after unlearning performs single-stage recovery with a constrained objective comprising four terms—target recovery, unlearning maintenance, retain replay, and superposition decoupling—and uses gradient projection to avoid reviving the requested entities, without modifying the unlearning algorithm.
   • Main results:
     • In the main setting (white-box, |U|=100K), the LLaMA-3.1-8B target group's Judge score drops by 0.33–0.57 (SBIC+GA from 0.850 to 0.281), and Qwen-2.5-7B drops by 0.15–0.36, while the retain group drops much less (Table I).
     • Averaged over all main settings, PUA proxies reduce the target by 0.331, whereas size-matched random proxies reduce it by only 0.052 (Table II).
     • In the black-box setting with Qwen-2.5-3B as the proxy model, the target reduction ∆T is 0.090–0.168; the 14B proxy exceeds the white-box reference on CounterFact and UMLS (Table III).
     • SAUP on LLaMA averages T recovery to 0.778 across six cells while maintaining a T−S gap of 0.216; on Qwen the average gap is 0.134 (Table I).
     • The output-level detection baseline PDDR misses 24.6%–35.2% of KSD-marked targets (definition probe), and its recovered T−S gap is smaller than SAUP's in all three domains (Table V, VI).
   • Authors' conclusions and implications: the authors state that legitimate unlearning requests can be maliciously exploited through knowledge superposition, and that even benign users may inadvertently delete entities superposed with sensitive entities; LLM providers need representation-level detection and recovery mechanisms when handling unlearning requests. The authors also note that SAUP turns targeted failure into a measurable trade-off between target protection and strictly completing proxy unlearning, and that recovery partially reverses proxy unlearning.

   Collapse this answer

Related papers