SteerProbe: Learning Black-Box Attacks to Bypass Safety Steering in Vision-Language Models

The researchers propose SteerProbe, an output-only black-box attack that learns to select effective rewrites for unseen requests to bypass activation steering defenses in vision-language models (VLMs). The authors test representative steering defenses with text, visual, and joint rewrites that preserve the original intent, find that these rewrites can bypass the defenses, and give a local analysis condition showing that rewrites can cross an alternative safety boundary under arbitrary changes in local steering corrections. Across three VLM backbones, two benchmarks, and three steering defenses, SteerProbe, under a budget of 500 calibration queries per endpoint and benchmark, raises the harmful rate of all defended settings from an average of 7.43% to 18.36%. The authors therefore argue that robustness on original benchmark inputs is insufficient to characterize the safety of steering defenses, that rewrite robustness should be an important evaluation dimension, and that steering mechanisms need to maintain safety under intent-preserving multimodal variations without losing normal utility.