A Unified Evaluation Framework for Physical Adversarial Robustness in the Era of Edge VLMs

AI summaryThe researchers propose a unified evaluation framework that uses conditional attack success rate (cASR) to measure the robustness of vision-language models and object detectors to physical adversarial patches, finding that the fragility of compact visual grounding VLMs can be comparable to that of object detectors. This metric is computed only on images where the model can still localize under a matching gray occluder, and failures are divided into three categories: missed detection, substitution, and displacement. The study finds that failure modes are mainly related to model output format rather than architecture family; among the three categories, only missed detection can be noticed by downstream systems, while displacement and substitution return high-confidence, normally formatted boxes and are potentially more dangerous. Under distribution shift, VLMs that were originally dominated by missed detection shift more toward displacement and substitution failures. Code and data are open-sourced at https://github.com/ericliang1/physical-adversarial-robustness。