Beyond Standard Benchmarks: Systematically Auditing the Robustness of Vision-Language Models to Natural Semantic Variations Across Diverse Tasks
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
AI summary
A study systematically audits the natural adversarial robustness of CLIP, robust CLIP, BLIP2 and SigLIP2, finding that robust CLIP may amplify natural adversarial vulnerability. The evaluation uses typographic attacks, ImageNet-A and natural-language-induced adversarial examples, covering zero-shot image classification, semantic segmentation and visual question answering. CLIP models show significant performance degradation on natural-language-induced adversarial examples; interpretability analysis is used to locate failure modes.