Feedback-Based Modality Mutual Search (FMMS): Attacking Vision-Language Pre-trained Models

Researchers propose a new attack paradigm for vision-language pre-trained (VLP) models, FMMS, which uses a modality mutual loss (MML) to push matched image-text pairs apart and randomly pull mismatched pairs together in feature space, and iteratively optimizes adversarial examples with feedback from the target model. The authors state that this is the first work to explore the multimodal adversarial boundary using feedback from the target model. On the Flickr30K and MSCOCO image-text matching tasks, FMMS significantly outperforms state-of-the-art baselines.