TAME: MoE-Based Test-Time Adversarial Prompt Tuning for Vision-Language Models
TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models
AI summary
TAME is a test-time adversarial defense for vision-language models such as CLIP, replacing the single adaptive prompt of the prior TAPT with an input-conditioned MoE framework, without downstream task retraining. On 11 datasets including ImageNet, TAME improves the zero-shot adversarial robustness of original CLIP under AutoAttack by at least 49.1%, largely preserves clean-sample generalization, and improves over existing adversarial prompt tuning methods by at least 30.2% on average.