Stage-wise Attention-Guided Region Ranking for Adversarial Attacks on Large Vision-Language Models

The paper proposes SAGA (Stage-wise Attention-Guided Attack), a black-box region ranking framework that uses fixed attention maps from open-source LVLMs to guide perturbation updates, without needing access to the target model's parameters, gradients, or attention maps. The authors state that cross-modal attention can identify regions sensitive to adversarial perturbations, and starting from dominant hotspots and gradually shifting the update scope to less salient regions can improve targeted attack effectiveness. Across ten closed-source and open-source LVLMs, SAGA achieves the highest attack success rate and the best overall imperceptibility, and the code has been open-sourced on GitHub.