On Success and Simplicity: Revisiting Transferable Vision-Language Attack Pipelines
On Success and Simplicity: A Second Look at Transferable Vision-Language Attack Pipeline
The researchers propose SimVLA, a simpler transferable adversarial attack pipeline targeting vision-language pre-trained models (VLPMs). The authors identify three previously overlooked issues in prior attack pipelines, stemming from improper cross-modal interaction and excessive operations, and propose improvements leveraging domain knowledge such as cross-modal word identification. Experiments on four datasets and three downstream tasks show that on the Flickr30k text-image retrieval dataset, SimVLA's R@1 transferability is 8.01%-14.71% higher than SOTA baselines, while consuming only about 35.73% of the time and 46.26% of the maximum GPU memory. The code is open-sourced on GitHub (https://github.com/RYC-98/SimVLA）。