CLIP-Fed: A Defense Method Against Backdoor Attacks in Federated Learning Guided by Vision-Language Pre-trained Models
A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learning
AI summary
The researchers propose CLIP-Fed, which uses the zero-shot capability of vision-language pre-trained models to defend against backdoor attacks in federated learning. It aligns the global model with CLIP using a prototype contrastive loss and KL divergence, and constructs a server dataset without client samples using a multimodal large language model and frequency analysis. On CIFAR-10 and CIFAR-10-LT, the attack success rate drops to 2.03% and 1.35% on average, and the main task accuracy improves by 7.92% and 0.48% on average.