ACM MM 2024 paper EvilEdit: backdoor attack on text-to-image diffusion models within one second

The ACM MM 2024 paper EvilEdit proposes a training-free and data-free backdoor attack method for text-to-image diffusion models. By directly editing the key and value projection matrices of the cross-attention layers, it aligns trigger words with the backdoor target. On Stable Diffusion v1.5, injection is completed within one second on a single GPU, with an attack success rate of 100%, no samples required, only 2.2% of parameters modified, and an FID difference of only 0.13 between normal generation quality and the original model. The method includes a protected whitelist and a visual target attack module, and can inject 5 backdoors simultaneously; after 1500 steps of fine-tuning, the attack success rate of the backdoors remains above 80%. The paper states that a white-box attacker can upload a backdoored model to a public platform via domain spoofing, causing users to generate unsafe images or contaminating downstream training data. Code is open-sourced: https://github.com/haowang-cqu/EvilEdit。