Paying less attention to function words gives vision-language models robustness for free

To address the trade-off between robustness and performance, the researchers propose Function-word De-Attention (FDA), which improves the robustness of vision-language models against cross-modal adversarial attacks by reducing the model's attention to function words. The authors observe that function words may trigger VLM vulnerability under cross-modal adversarial attacks; accordingly, FDA computes the original cross-attention and the function-word cross-attention separately within attention heads, and subtracts the latter from the former. Experiments cover 2 SOTA baselines, 6 attacks, 2 downstream tasks, 3 datasets, and 3 models; in the retrieval task, the attack success rates of the 3 models decrease on average by 18%, 13%, and 53%, respectively, with performance dropping only 0.2%, 0.3%, and 0.6%; in the visual grounding task, the attack success rate decreases by 90%, while performance improves by 0.3%. The paper also verifies FDA's scalability, generalization, and zero-shot performance, with code open-sourced at https://github.com/michaeltian108/FDA。