Research on Adversarial Robustness of Multimodal Large Language Models

The researchers systematically studied the visual adversarial robustness of multimodal large language models (MLLMs), proposed a diagnostic CLIP alignment protocol that can predict, before full MLLM training, which robust visual encoders can transfer effectively, and found that large-scale multimodal adversarial pretraining, rather than unimodal scale alone, is key to robustness transfer. Integrating such encoders into MLLMs via end-to-end multimodal training improved image captioning by an average of 28 CIDEr points and VQA accuracy by 11.7% under strong adversarial attacks compared with constrained plug-and-play baselines. The paper also found that directly adversarially training a standard non-robust MLLM degrades both clean and adversarial performance, indicating that robust visual representations are a strict prerequisite, while end-to-end adversarial training starting from a robust backbone can further add 1.9 CIDEr points and 4.3% VQA accuracy. Lightweight visual random transformations at test time can serve as a black-box defense, raising the adversarial performance of non-robust models from near zero to a level comparable to robust models; robust models also significantly reduce toxic generation under white-box visual jailbreak attacks. Code and pretrained weights will be publicly released.