The effectiveness of backdoor mitigation for vision-language models depends on the pretraining objective
Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objective
Translation in progress — untranslated text is omitted.
The paper finds that the effectiveness of backdoor mitigation for vision-language models depends on the pretraining objective: stronger pretraining objectives bring higher zero-shot classification performance and also make backdoor behavior harder to remove. The authors train multimodal models with multiple pretraining objectives on CC3M (3 million) and CC6M (6 million) data points, then use CleanCLIP to remove the poisoning, and find that even with extensive hyperparameter tuning, CleanCLIP cannot effectively remove the poisoning under stronger pretraining objectives. CleanCLIP is the current state-of-the-art multimodal backdoor mitigation method, and the authors note that this finding has important reference value for practitioners who train models on large-scale web-collected data and are concerned about backdoor threats.
- Multimodal models are pretrained on web-scraped data and are susceptible to poisoning backdoors; the SOTA detoxification method CleanCLIP had previously been validated only on models pretrained with the MMCL objective, and the authors ask whether it remains effective for stronger pretraining objectives.
- On CC3M/CC6M, CLIP models were pretrained with two objectives, MMCL and MMCL+SSL, and poisoned, then cleaned by fine-tuning with CleanCLIP; the post-cleaning zero-shot accuracy and ASR were compared, and analyses were done on stopping criteria, non-ideal cleaning data, and several alternative detoxification methods.
- The authors report: MMCL-pretrained models largely retain accuracy after cleansing, whereas MMCL+SSL models on CC6M must lose 45% relative accuracy to reach ASR ≤ 5%; mixing just 5 poisoned samples into the cleansing data weakens cleansing, and deep clustering, heavy regularization, and shrink-and-perturb all fail to detoxify.
- The authors state that detoxifying from a strong target model remains an open problem and suggest that future defenses be tested under multiple pretraining objectives; the experiments are limited to CLIP-type dual encoders (ResNet-50 and ViT), two backdoors (BadNet and label-consistent), CC3M/CC6M data, and Imagenet zero-shot evaluation.