Not All Steps Matter: Time-Adaptive Spectral Control for Concept Removal in Diffusion Models
Not All Steps Matter: Temporally Adaptive Spectral Control for Concept Removal in Diffusion Models
Translation in progress — untranslated text is omitted.
TASN is a training-free concept removal method for text-to-image diffusion models that achieves lower attack success rates than uniform SN on five red-team benchmarks while recovering most of its image quality loss. Time-resolved spectral analysis shows that concept energy is temporally hierarchical: content decays fastest, objects are intermediate, and style persists longest; TASN accordingly adjusts suppression strength and spectral components at each timestep. Ablations confirm that the gain comes from time-directed suppression rather than weaker suppression, and the method generalizes across concept categories and architectures.
- Existing training-free spectral erasure methods apply the same intervention at every denoising step; the authors argue this wastes intervention at low-concept-energy steps and harms image quality, and that more precise concept removal is needed.
- First perform timestep-resolved spectral analysis and find that concept energy decays by category (content fastest, style most persistent); then propose TASN accordingly, making suppression strength proportional to concept energy and tracking only the spectral directions active at each step.
- On SDv1.4, TASN has lower ASR than uniform SN on all five red-team benchmarks (e.g., P4D 10.93% vs. 18.54%), and CLIP recovers from 30.64 to 31.37; the authors state that causal windows and equal-budget controls support that the gain comes from temporal localization.
- The paper does not specifically discuss limitations. Experimental coverage: the main experiments use only SDv1.4 (30-step DDIM), the temporal hierarchy is validated on SDXL; per-class evaluation uses 20 prompts per concept; the paper does not report human evaluation of generated images.