Beyond False Stability: High-Noise Drift Gating for Test-Time Adversarial Defenses of Vision-Language Models

The researchers propose a training-free, plug-and-play high-noise drift gating mechanism that triggers existing test-time defenses for vision-language models such as CLIP only when adversarial-like instability is detected. The mechanism exploits the signal that adversarial representations are less stable than clean representations under high noise, avoiding the "false stability" of the low-noise region. It improves the clean-robust trade-off on 13 datasets, and on eight fine-grained datasets the average clean plus adversarial accuracy of the defense-like method rises from 65.7% to 71.4%.