Turning a Blind Eye: Hijacking Attention with Adversarial Attacks to Make Large Vision-Language Models Blind to Safety Instructions
Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
The researchers propose an attention-guided visual jailbreak method against large vision-language models (LVLMs) that bypasses rather than suppresses safety alignment by directly manipulating attention patterns. The method introduces two auxiliary objectives: one suppresses the model's attention to alignment-related prefix tokens, and the other anchors generation to adversarial image features; the authors claim this push-pull design reduces gradient conflict by 45%. On Qwen-VL the attack success rate reaches 94.4% versus a baseline of 68.8%, with 40% fewer iterations; under a stricter perturbation budget of ε=8/255 it maintains a 59.0% success rate versus 45.7% for the standard method. Mechanism analysis shows that successful attacks reduce the model's attention to the system prompt by 80%, and the authors call this failure mode safety blindness, i.e., the model generates harmful content not because it overrides safety rules but because it cannot retrieve them.