You Can't Have Both: Concept Entanglement Limits Diffusion Model Unlearning

Core question: can concept unlearning in text-to-image diffusion models both thoroughly suppress a target concept and avoid damaging semantically similar neighbor concepts? The authors argue it cannot, and aim to show that this trade-off is determined by representation geometry.

• Problem setting and importance: concept unlearning requires removing undesirable concepts (object categories, artistic styles, specific identities) from a trained diffusion model while preserving the model's general generation ability; privacy and copyright concerns have driven this research. The paper focuses on object category concepts, on the grounds that benchmarks and detectors for such concepts are the most mature.
• Shortcomings of existing methods (per the paper): most existing methods define erasure via target tokens, prompts, or local parameter subsets, and can suppress prompts that directly mention the target, but struggle with concept leakage through associated context (e.g., not mentioning horse but mentioning jockey and racetrack still generates a horse); increasing erasure strength then propagates along shared representations, causing collateral damage to semantic neighbors.
• Core observation: semantically related concepts (horse/pony/donkey) occupy mutually overlapping regions in the cross-attention activation space, whereas isolated concepts (castle) are separated (Figure 1); the authors therefore argue that leakage and collateral forgetting are two dual symptoms of the same geometric fact.
• What the paper proposes: (i) quantify the overlap of concept activation regions with an entanglement coefficient κ̂; (ii) under two empirically validated assumptions, prove the lower bound γ≥κ(1−ε)/L (Theorem 4.6) and that when κ>0, perfect erasure and perfect retention cannot both be achieved (Corollary 4.8); (iii) validate this κ-scaled trade-off on 13 methods, and propose evaluating unlearning methods by the Pareto frontier.
• Evaluation-paradigm assumption: the authors point out that current evaluations assume UA (unlearning accuracy) and RA (retain accuracy) can be optimized independently, whereas their experimental evidence shows this assumption does not hold (Section 3).

Closed-form editing

• UCE (Gandikota et al., 2024) and RECE (Gong et al., 2024)—modify the cross-attention projections through analytical updates, without gradient training; RECE additionally searches for an embedding that can regenerate the target and removes it in closed form.

Fine-tuning methods

• ESD (Gandikota et al., 2023)—uses negative guidance to align the prompt of the target token to a neutral anchor; subsequent work improves precision by editing cross-attention representations (Wang et al., 2025) or updating salient parameters (Fan et al., 2023; Wu and Harandi, 2024).
• CPE (Lee et al., 2025)—trains a residual attention gate with an anchoring loss; TRUST (Kori et al., 2026)—selectively fine-tunes target-related neurons under Hessian regularization.
• AdvUnlearn (Zhang et al., 2024) and STEREO (Srivatsan et al., 2025)—add adversarial training to defend against prompt-based regeneration attacks. The paper notes that such methods still define erasure in terms of target tokens or local parameters, making it difficult to handle leakage from associated context, and that under aggressive erasure the edits propagate along shared representations and damage neighbors.

Inference-time methods

• Do not change weights and intervene during generation: manipulating text embeddings (Li et al., 2024, i.e., SEOT), projecting prompt tokens out of the target subspace (SAFREE, Yoon et al., 2025), adding safety guidance during denoising (SLD, Schramowski et al., 2023), inserting lightweight concept-blocking adapters (Lyu et al., 2024), ablating sparse autoencoder features (SAeUron, Cywiński and Deja, 2025). The paper notes that inference-time intervention itself does not equal preservation: such methods span the entire range from strong erasure with high neighbor damage to strong preservation with residual leakage.

Preservation-oriented methods

• Explicitly aim to reduce collateral damage: CPE anchors non-target concepts, TRUST limits the scope of updates, SAFREE adaptively filters; there are also adversarial preservation (Bui et al., 2024), Receler (Huang et al., 2024), MACE (Lu et al., 2024), meta-unlearning (Gao et al., 2025). The paper evaluates CPE, TRUST and SAFREE, stating that they reach more favorable operating points but do not escape the trade-off.

Baselines and benchmarks: the objects compared in the paper are the 13 unlearning methods above (UCE, RECE, ESD, SalUn, EDiff, CPE, TRUST, AdvUnlearn, STEREO, SEOT, SAeUron, SLD, SAFREE), all applied to Stable Diffusion v1.4; no existing annotated test set was used, and all evaluation images were generated by the models from four types of controlled prompts.

Positioning: the authors state that erasure damaging utility has been widely documented, preservation-oriented methods aim to mitigate it, and localization work (Zarei et al., 2026) identifies where concept representations are located, but none of them models target-neighbor overlap or derives the limits of unlearning; this paper's claim is narrower and structural—robust erasure necessarily damages related concepts because of representation overlap, and it quantifies this with κ, proves it is unavoidable, and turns it into a frontier-based evaluation protocol.

Overall approach: concepts are formalized as regions in the cross-attention activation space, concept activation region, and the entanglement coefficient κ measures the proportion of overlap between the target region and other concept regions; it is proved that any ε-robust erasure must cause a preservation error of at least κ(1−ε)/L on non-target concepts, thereby reframing unlearning as a Pareto problem.

Problem setup and formalization

• Concept activation region: fixing the probe layer ℓ∗ and denoising step t∗, R_c(ℓ^∗) = {Φ^ℓ∗_θ(p) ∣ O_X(x_θ(p), c)=1}, i.e., the set in activation space of prompts that can generate images containing concept c; O_X is the oracle that determines whether an image contains concept c.
• Entanglement coefficient: κ(cu, ρ) is defined as the volumetric proportion of the target's ρ-neighborhood region that intersects other concept regions (Definition 4.1); κ=0 means complete decoupling, and when κ>0 any edit that moves activations within Rcu perturbs other concepts.
• Two objectives: ε-robust unlearning requires that after erasure, the probability of generating the target concept under any prompt does not exceed ε (Definition 4.2); γ-concept preservation requires that the expected activation displacement under the non-target prompt distribution be less than γ (Definition 4.3).

Core theorem

• Two assumptions: Assumption 4.4 requires the generation probability to be L-Lipschitz continuous with respect to activation distance; Assumption 4.5 requires any ε-robust edit to move activations within Rcu by at least ∆>0. The authors state that both assumptions were empirically validated (Appendices D, E).
• Main theorem: γ ≥ κ(c_u,ρ)·(1−ε)/L (Theorem 4.6), i.e., the lower bound on preservation error grows linearly with the overlap κ; Corollary 4.8 states that when κ>0 no method can achieve both ε=0 and γ=0.
• Scope: the authors state that the theorem does not assume how θ̂ is obtained, and holds equally for activation intervention, closed-form editing, gradient fine-tuning (e.g., SalUn, TRUST) and inference-time methods (Remark 4.7).

Estimation of κ̂ and neighbor discovery

• Neighbor discovery: for each target, 10 semantic candidates and 3 unrelated controls are taken, and candidates whose centroid cosine distance is below the 25th percentile of pairwise distances in that pool are kept as neighbors N(cu); controls must fall outside N(cu) (Definition C.1).
• κ̂ estimation: the proportion of target activation samples that fall within the median concept intra-distance ρcu of some neighbor sample (sample-level coverage based on cosine distance, not centroids); the authors state this is a lower-bound estimate of κ and is stable to the choice of neighbor set and threshold (horse's κ̂ stays at 0.81 as the threshold varies from the 10th to the 50th percentile).
• Probe settings: by default, activations are extracted at mid block.attentions.0, step 25 of 50-step DDIM; the authors state that the κ̂ ordering is stable across six candidate layers (rs≥0.80), and mid-block was chosen because its bimodal separation is clearest (Appendix C.2).

Evaluation protocol

• Prompt families: four types of prompts are constructed for each target—direct, which names the target directly; indirect, which uses associated context but excludes the target's word stem; neighbor, which names neighbor concepts; control, which names unrelated concepts; associated context words are mined from LAION caption text matching the target (LAION images are not used).
• Metrics: UA (the proportion of direct prompts not classified as the target), IRR (the proportion of indirect prompts classified as the target, as a worst-case proxy for ε), IRA, NP (the preservation ratio on neighbor concepts relative to the original model, the main trade-off metric), CP (the preservation ratio on control concepts), DamageGap = CP − NP.
• Determination method: all images are top-1 classified by openai/clip-vit-base-patch32 over a fixed 65-concept vocabulary; the authors report that this classifier achieves micro-averaged TPR 97.0% and FPR 0.55% on 500 annotated images (Table 6).

Experimental setup

• Models tested: CompVis Stable Diffusion v1.4 as the main model; cross-architecture experiments on SDXL (base-1.0) and FLUX.1-schnell (Appendix M).
• Target concepts: 5 targets in the main experiments, κ̂∈[0.27, 0.89]—horse 0.81, dog 0.89, cat 0.77, bear 0.82 (high-κ̂ animals) and castle 0.27 (low-κ̂ building); additionally snake, fish, tree, butterfly, car (κ̂=0.37 to 0.83) were evaluated with 4 methods (Appendix J.6).
• Methods tested: 13, in three categories—closed-form (UCE, RECE), fine-tuning (ESD, SalUn, EDiff, CPE, TRUST, AdvUnlearn, STEREO), inference-time (SEOT, SAeUron, SLD, SAFREE).
• Generation settings: 50-step DDIM, guidance scale 7.5, fixed seed; for each (method, target) pair, 100 direct images, 100–235 indirect images, 50 images per neighbor candidate (500 in total), 50 images per control (150 in total) (Appendix H.3).
• Evaluation method: openai/clip-vit-base-patch32 top-1 classification over a fixed 65-concept vocabulary; the authors report TPR 97.0% and FPR 0.55% on 500 annotated images (Table 6); the paper states that manual verification was performed on several random instances, without reporting agreement numbers.

Main results

According to Table 1 (SD v1.4, average over five targets, mean±std, SAeUron covers only the four animal targets):

Swipe horizontally to read the table

Method	Type	UA↑	IRR↓	NP↑	CP↑	DamageGap↓
UCE	Closed-form	80.6±18.4	10.5±23.4	88.6±25.1	99.3±3.1	10.7±25.4
RECE	Closed-form	99.4±0.9	3.0±4.0	56.1±10.6	93.5±2.0	37.4±10.2
ESD	FT	80.7±11.1	20.9±14.2	80.6±13.5	97.0±1.6	16.4±14.5
TRUST	FT	94.2±2.2	6.5±3.5	76.4±9.1	98.0±5.0	21.6±12.2
STEREO	Robust FT	99.8±3.1	3.5±3.5	18.0±12.0	70.8±27.4	52.8±22.7
SEOT	Inf.	82.6±13.2	24.5±13.8	85.1±13.8	98.3±1.3	13.6±13.0
SLD	Inf.	99.6±0.9	0.5±0.9	58.8±16.8	89.1±4.1	30.3±16.3
SAFREE	Inf.	91.4±11.4	8.4±10.2	64.7±16.0	85.7±12.2	21.0±19.4

(The five rows for SalUn, EDiff, CPE, AdvUnlearn and SAeUron are omitted; among them SAeUron has average NP 59.0 and IRR 27.6, one of the preservation-end methods.)

• No method escapes the trade-off: the authors state that none of the 13 methods achieves both UA>90 and NP>80; the methods with UA≥99.4 (RECE, CPE, SLD, STEREO) have NP between 18.0 and 71.9.
• Mechanism category does not determine the operating point: the authors state that the inference-time method SLD achieves the lowest IRR (0.5) but NP 58.8, while the also inference-time SEOT preserves neighbors (NP 85.1) yet leaks (IRR 24.5); algorithm design determines where a method falls on the frontier, and concept entanglement determines the frontier itself.
• The trade-off is structural: the authors state that DamageGap spans more than 40 points across methods (UCE 10.7 to STEREO 52.8), method identity is the primary determinant of the operating region, and concept identity modulates severity.

Scaling relationship between κ̂ and damage (Section 5.3)

• NP decreases monotonically with κ̂: STEREO's NP decreases from 31.96 for castle (κ̂=0.27) to 27.60 for cat, 14.71 for horse, 13.75 for bear, and 1.81 for dog (κ̂=0.89), Spearman rs=−1.0 (right table of Figure 3); the authors state this is consistent with the linear κ dependence in Eq. (8).
• Activation displacement directly verified: at up blocks.1.attentions.1, STEREO's neighbor displacement was measured rising from 75.9 for castle to 257.0 for dog (rs=0.90); for the five fine-tuning methods, displacement and κ̂ are all positively correlated, with four having rs≥0.70 and ESD strictly monotonic (rs=1.00) (Table 14).
• Single Lipschitz constant test: Ltight back-solved from Eq. (8) varies by only 7.8% across concepts for STEREO ([0.0032, 0.0035]), while other methods vary by 24%–57% (Table 15); the authors state this matches the prediction that STEREO operates near the lower bound.
• Additional targets: on the five added targets, the method-average NP decreases monotonically with κ̂ from 65.0 to 58.8 (rs=−1.0), and within-method correlations are all negative (−0.2 to −0.7); the authors note that these correlations are descriptive and not individually statistically significant.

Cross-architecture and hypothesis validation

• SDXL: ESD's NP decreases monotonically with κ̂SDXL—cat 90.9 (κ̂=0.65), castle 85.0 (0.72), dog 77.1 (0.92), rs=−1.0 (Table 17); erasure is weakest for dog, the most entangled (UA 77 vs cat 99), and the authors state the ordering is not driven by erasure strength.
• FLUX: MMDiT has no comparable cross-attention block, and the proxy κ estimate saturates near 1 on both targets and unrelated controls; the authors state that κ-scaling on MMDiT remains untested; however, on FLUX ESD's NP drops to 35.6–40.5 while CP stays at 95.3–98.9, reproducing neighbor-selective damage (Table 18).
• Hypothesis validation: when the six methods erase horse, the target displacement is about 6–16, and for all methods neighbor displacement is greater than control (gap 0.9–6.9) (Appendix D, Figure 5); the prompt perturbation experiment observed no points violating Lipschitz, with empirical constant L̂≈0.174 (Appendix E, Figure 6).

Other ablations and analyses

• Probe layer sensitivity: the κ̂ ordering has rs≥0.80 across six cross-attention layers (Table 3).
• Timestep sensitivity: the κ̂ ordering at steps 10 and 40 relative to step 25 has rs=0.93 and 0.96, but absolute values change (castle is 0.77 at step 10) (Table 4).
• Text encoder comparison: in SDXL's CLIP text space, 9 of 10 concepts have κ̂=1.0, and FLUX's T5 has κ̂ all 0 on the five targets; the authors state this is a descriptive comparison rather than a causal decomposition (Appendix L).
• Exception: STEREO still causes NP 31.96 and DamageGap 64.03 on the low-κ̂ castle; the authors state this reflects that the theorem is a lower bound rather than an upper bound, and that STEREO's subspace suppression is κ-blind (Appendix J.5).

Limitations and future directions noted by the authors

• Limited evaluation scope (Section 7): the 13 methods were evaluated only on 5 targets in SD v1.4, and the other 5 targets only with 4 methods; cross-architecture evidence is limited—SDXL covers only 3 concepts with 1 method, and on FLUX only neighbor-selective damage can be shown while κ cannot be resolved; κ-scaling on MMDiT remains open.
• Concept types not covered (Section 7): the targets are all object concepts; artistic style, identity and harmful-content concepts were not tested.
• The theoretical constants are effective constants (Section 7): L covers the entire network (including skip connections), and is estimated only on the prompt variations in Appendix E.
• κ̂ is a lower-bound proxy (Section 7): its absolute value depends on the probe layer, denoising timestep and candidate pool; the authors' conclusions rely on its ordering, and the ordering is stable across the tested settings.
• The text-space comparison is not a causal decomposition (Section 7): it cannot be used to determine the source of entanglement.
• Future directions (Remark F.1 and Appendix K): characterizing the displacement geometry more finely is an interesting future direction; a systematic estimation of L that distinguishes saturated and unsaturated regimes is a future direction.

Experimental coverage

• The models tested are Stable Diffusion v1.4, SDXL and FLUX.1-schnell, 3 in total; the main experiments are on SD v1.4, 13 methods × 5 targets, plus another 5 targets × 4 methods (Section 5.1, Appendix J.6).
• Cross-architecture evidence: on SDXL, 3 concepts × 1 method (ESD); on FLUX, 3 concepts × 1 method (ESD) (Appendix M).
• Image counts per (method, target) pair: 100 direct, 100–235 indirect, 500 neighbors in total, 150 controls in total; a few runs used non-standard image counts (Appendix H.3, note to Table 1).
• Classifier reliability was evaluated on 500 annotated images (100 per target); the paper states that manual verification was performed but does not report agreement numbers between humans and the classifier (Appendix I.1, I.4).
• The paper does not report statistical significance tests between unlearning methods; Appendix K notes that n=5 concepts limits the power of significance tests for correlations of a single method.

An analytical paper on the limits of concept unlearning in diffusion models: the authors claim that two failure modes of existing methods—concept leakage under indirect prompts and collateral forgetting of semantic neighbors—are not algorithmic flaws but dual symptoms of overlapping concept activation regions.

• Problem to be solved: Concept unlearning requires suppressing a target concept (e.g., horse) while preserving related concepts (e.g., donkey); the authors point out that current evaluations assume UA and RA can be optimized independently, but this assumption does not hold when concepts are entangled.
• Method highlights: Concepts are formalized as regions in the cross-attention activation space, and the entanglement coefficient κ̂ (the proportion of samples whose target region is covered by neighbors) measures overlap; under two empirically validated assumptions—Lipschitz continuity and minimal displacement—the paper proves γ≥κ(1−ε)/L (Theorem 4.6), and that when κ>0, perfect erasure and perfect preservation cannot both be achieved (Corollary 4.8).
• Main results: On SD v1.4, none of 13 methods simultaneously achieves UA>90 and NP>80; STEREO averages UA 99.8 and IRR 3.5 but NP only 18.0, while SEOT preserves neighbors (NP 85.1) but has IRR 24.5 (Table 1).
• κ-scaling validation: STEREO's NP decreases monotonically as κ̂ rises from 0.27 (castle) to 0.89 (dog) (31.96→1.81, rs=−1.0); neighbor activation displacement increases with κ̂ (rs=0.90); on SDXL, ESD's NP ordering is consistent with κ̂SDXL (rs=−1.0); on FLUX, neighbor-selective damage is reproduced but κ cannot be estimated.
• The authors' conclusion: For entangled concepts, perfect unlearning is the wrong goal, and methods should be evaluated according to the Pareto frontier given by the theorem; κ̂ can serve as a pre-deployment diagnostic, benchmarks should jointly report erasure and preservation, and method design should be κ-aware, restricting displacement to the directions required for erasure.