Multi-Paradigm Collaborative Adversarial Attack for Multimodal Large Language Models

Independent optimization on single-paradigm surrogates limits transferable adversarial examples against MLLMs.

• Scenario: The authors believe MLLMs are used for joint visual and textual understanding, but like unimodal models, visual perturbations can alter their perception and reasoning, producing erroneous or misleading text. The authors claim this necessitates stronger attack frameworks to evaluate safety.
• Existing limitations: The authors note that transferable attacks commonly use white-box surrogates from a single learning paradigm (e.g., CLIP-style cross-modal alignment) and independently optimize in their respective feature spaces with simple fusion. The authors argue that a single paradigm covers only part of the semantics, and perturbations easily overfit that paradigm's biases; lacking synergy makes gradient directions redundant, and optimization stalls at local optima.
• Observation: The authors believe that cross-modal alignment, multimodal understanding, and visual self-supervision capture complementary semantics, and synergistic optimization can mitigate representation bias.
• Proposed content: They propose MPCAttack and its MPCO, which aggregate image and text semantics for contrastive matching, supporting targeted and non-targeted attacks.
• Threat model:
  • Goal: Generate x_adv = x_s + δ so that the black-box MLLM's response to x_adv is as close as possible to the response to x_t or the annotated answer. Evaluation also includes untargeted settings, requiring deviation from the source image's semantics.
  • Knowledge: The surrogate is a white-box surrogate, and the victim model is a black-box target / victim black-box MLLM.
  • Capability: The operation specified in the paper is adding perturbations to the visual input, with ||δ||_∞ ≤ ε. The paper does not state whether the attacker can modify text prompts or model weights.
  • Victim system: Open-source and closed-source MLLMs. Success is determined by response semantic similarity, or yes/no misclassification on MME.

Section 2 of the paper divides related work into adversarial attacks, MLLMs, and large-scale learning paradigms.

Transferable Multimodal Adversarial Attacks

• AttackVLM (Zhao et al., 2023): Uses a pretrained vision-language model to align image-text and image-image features, generating targeted attacks against MLLMs. The paper only describes this here and does not list separate criticisms.
• AnyAttack (Zhang et al., 2025) and X-Transfer (Huang et al., 2025): The former uses self-supervised contrastive learning on large-scale unlabeled data to generate targeted samples without labels; the latter dynamically selects a small number of surrogates from a large search space to generate more generalizable transferable samples.
• CoA, M-Attack, and FOA-Attack: CoA (Xie et al., 2025) uses multimodal semantic optimization to iteratively update adversarial samples; M-Attack (Li et al., 2025) uses image augmentation and CLIP ensemble surrogates, targeting closed-source MLLMs such as GPT-4o; FOA-Attack (Jia et al., 2025) uses Feature Optimal Alignment with dynamic model weight ensembling. The introduction criticizes [25, 29, 48] for relying on a single paradigm and treating different surrogate features as independent objectives; Section 4.2 states that CoA has limited transferability due to its lightweight single-paradigm model.

Multimodal Large Language Models

• Open-source models: The paper lists LLaVA, Qwen-VL, and InternVL, stating that they are jointly trained on large-scale image-text data and can perform image captioning, visual question answering, and cross-modal reasoning.
• Closed-source models: The paper lists GPT-5, Claude, and Gemini, stating that they have strengthened complex reasoning and real-world scenario adaptation. The authors state that strong comprehension ability does not eliminate challenges in adversarial robustness.

Large-Scale Learning Paradigms

• Cross-modal alignment: Using CLIP and SigLIP as examples, learning image-text correspondences in a shared space.
• Multimodal understanding: Using InternVL and Qwen-VL as examples, performing cross-modal reasoning in a unified space rather than only feature alignment.
• Visual self-supervision: Using DINOv2 as an example, learning visual representations from large-scale unlabeled images. The authors state that the three perspectives are complementary and serve as the foundation of MPCAttack.

Baselines and Benchmarks

• Baseline methods: AnyAttack, CoA, X-Transfer, M-Attack, and FOA-Attack, all generating adversarial samples according to their respective original settings.
• Benchmarks/datasets: The NIPS 2017 adversarial competition subset of ImageNet, Flickr30K, and MME.

The authors position the difference between this paper and the above works as: simultaneously using aggregated features from three paradigms and using collaborative optimization rather than a single paradigm or independent fusion to generate transferable perturbations.

MPCAttack aggregates visual and textual features from three learning paradigms on a white-box surrogate, then uses MPCO for contrastive matching to optimize transferable perturbations. Figure 2 divides the pipeline into adversarial example generation and attacking black-box MLLMs.

Problem Setting

Given a source image x_s and a target image x_t, minimize a feature distance or similarity loss on the surrogate f, subject to an imperceptibility constraint:

min_δ L(f(x_s+δ), f(x_t)), s.t. ||δ||_∞ ≤ ε

where x_adv = x_s + δ, and ε is the maximum perturbation strength. The goal in Section 3.1 is to make the black-box MLLM's response to x_adv as close as possible to the response to x_t or the annotated answer. Targeted and untargeted share this optimization; the difference mainly lies in the success criteria described later.

Multi-Paradigm Features

• Image branch: Simultaneously use a cross-modal alignment encoder f_cI (e.g., CLIP), a multimodal understanding encoder f_m (e.g., InternVL3), and a visual self-supervised encoder f_v (e.g., DINOv2) to extract image features from x_s, x_t, and x_adv.
• Text branch: Feed x_t and x_s into the text generator f_mg of the multimodal understanding model to obtain target/source descriptions, then encode them with the text encoder f_cT of the cross-modal alignment model.
• Weighted fusion: The cross-modal alignment image and text features are combined as z^c = λ z^c_I + (1−λ) z^c_T. The authors state that λ balances visual and textual contributions. In the main experiments, λ = 0.6.
• Concatenation: Features from the three paradigms are ℓ2-normalized and then concatenated into z_s, z_t, and z_adv. The authors state this is to maintain scale consistency.

MPCO

• Matching direction: On the concatenated features, pull z_adv closer to z_t and push z_adv away from z_s. Similarity is measured with cosine similarity.
• Two coefficients: Temperature τ controls the sharpness of the similarity distribution, and balance factor ω adjusts the trade-off between attracting positive pairs and repelling negative pairs. In the main experiments, τ = 0.2 and ω = 2.
• Authors' rationale: Optimizing only in a single paradigm space tends to overfit to that paradigm's representation bias; multi-paradigm contrastive matching expands the global receptive field and focuses optimization on more discriminative semantic regions.
• Iteration: Algorithm 1 in the supplementary material applies random crops to the current adversarial image, accumulates normalized gradients with momentum, then updates in the sign direction and clips to the ℓ∞ budget. The paper lists the momentum factor µ but does not provide its value.

Inference and Success Criteria

• Responses: x_adv, x_s, and x_t are each input to the same victim black-box MLLM. When annotations are available, they can replace the model-generated source/target responses.
• Evaluation: The three responses are given to an evaluation model (the paper writes e.g., GPT-4o-mini), and GPTScore is used to compute semantic similarity. Targeted compares the adversarial response with the target response; untargeted compares the adversarial response with the source response.
• Threshold: Section 4.1 sets the threshold at 0.5. Targeted requires the similarity between the adversarial image and the target image to exceed 0.5; untargeted requires the similarity with the source image to be below 0.5. Figure 7 in the supplementary material provides the full evaluation prompt.
• MME: The supplementary material uses yes/no questions and only uses questions with answer "yes". Targeted pairs the adversarial sample with the target image's questions, and success is indicated by a response showing misclassification; untargeted pairs with the source image's questions.

Surrogates and Budget in Main Experiments

• Surrogates: The main text uses CLIP, InternVL3-1B, and DINOv2. Table 5 in the supplementary material specifies the main CLIP models as clip-vit-base-patch16, clip-vit-base-patch32, and CLIP-ViT-G-14-laion2B-s12B-b42K, and states that the primary ensemble uses these three sizes.
• Budget: ϵ = 16/255, ℓ∞; step size 1/255; 300 optimization steps. Experiments are conducted on a single NVIDIA RTX 3090.
• Victim-side prompting: For closed-source models, the text prompt is set to describe the image in one sentence of no more than 20 words. Open-source models are evaluated with VLMEvalKit.

Experimental Setup

• Victim models: Open-source Qwen2.5-VL-7B, InternVL3-8B, LLaVa-1.5-7B, GLM-4.1V-9B-Thinking; closed-source GPT-4o, GPT-5, Claude-3.5, Gemini-2.0. The MME table writes Qwen as Qwen2.5-VL-7B-Instruct. Figure 4 and Figure 8 also show responses from Gemini-2.0-flash, Gemini-2.5-flash, Claude-3.5-Sonnet, and Claude-4.5-Sonnet; the latter two are not included in the result tables.
• Data: ImageNet subset of 1,000 images, taken from the NIPS 2017 adversarial competition subset. Flickr30K has about 31,783 images with five human descriptions each; 1,000 images and captions are selected as source–target pairs; the supplementary material states that source images are taken in order and target images in reverse order, and evaluation directly uses the annotated captions. MME has 1,187 images, each with yes/no questions; evaluation only uses the yes questions.
• Baselines: AnyAttack, CoA, X-Transfer, M-Attack, FOA-Attack. Tables 1–4 for targeted do not report X-Transfer. Tables 6–7 for MME only report M-Attack, FOA-Attack, and MPCAttack.
• Metrics and evaluation: ASR and AvgSim. For targeted, higher AvgSim is better; for untargeted, lower is better. Similarity is computed with GPTScore, threshold 0.5; the evaluation model is written as e.g., GPT-4o-mini. The paper does not report the number of repetitions or agreement with human judgment.
• Attack settings: ϵ = 16/255, step size 1/255, 300 iterations, λ = 0.6, τ = 0.2, ω = 2. Main surrogates are CLIP, InternVL3-1B, and DINOv2. The victim InternVL3-8B and the surrogate InternVL3-1B are not the same checkpoint.
• Open-source evaluation tool: VLMEvalKit. Closed-source models use the one-sentence description prompt given in Section 4.1.

Main Results

The table below shows ASR (%) on ImageNet. The first four rows are from Table 1, the last four from Table 2.

Swipe horizontally to read the table

Victim model	targeted MPCAttack	targeted FOA-Attack	untargeted MPCAttack	untargeted FOA-Attack
Qwen2.5-VL-7B	32.5	20.0	84.9	61.8
InternVL3-8B	88.7	72.3	99.1	93.0
LLaVa-1.5-7B	73.9	63.8	99.3	96.3
GLM-4.1V-9B-Thinking	58.2	38.3	85.1	68.1
GPT-4o	78.4	63.9	98.7	93.0
GPT-5	88.0	69.3	99.2	90.7
Claude-3.5	8.2	7.3	66.7	61.8
Gemini-2.0	78.9	50.4	97.6	85.0

• Averages: The authors state that on Table 1 and Table 2, for both targeted and untargeted, MPCAttack's ASR and AvgSim are higher than other methods in the tables. Open-source Average: targeted 63.33% (FOA-Attack 48.60%), untargeted 92.10% (FOA-Attack 79.80%). Closed-source Average: targeted 63.38% (FOA-Attack 47.73%), untargeted 90.55% (FOA-Attack 82.63%). These are the averages of the four models in each table, not a further combination of open-source and closed-source.
• Baseline exception: On Qwen2.5-VL-7B in Table 1 untargeted, X-Transfer's ASR is 66.2%, higher than FOA-Attack's 61.8%, but still lower than MPCAttack's 84.9%. The authors state that CoA transfers poorly; its Average ASR in Table 1 targeted is 0.18%.
• Claude-3.5: Also in Table 2 targeted, MPCAttack achieves 8.2% on Claude-3.5 and 88.0% on GPT-5. The paper does not separately explain this gap.

Flickr30K and MME

• Flickr30K: The authors state that the open-source targeted average ASR is 29.54%, with FOA-Attack at 15.41%; untargeted average is 63.99% (Table 3). Closed-source targeted/untargeted averages are 19.09% / 54.89% (Table 4). The authors believe the overall ASR is relatively low, possibly because each image has five captions from different perspectives, and the model's description may not cover all semantics.
• Not highest in every cell: On Qwen2.5-VL-7B in Table 3 untargeted, MPCAttack is 32.14%, lower than X-Transfer's 42.84%, AnyAttack's 35.20%, and CoA's 32.55%. On Claude-3.5 in Table 4 untargeted, MPCAttack is 65.22% and X-Transfer is 64.66%.
• MME: Table 6 open-source average ASR: targeted is MPCAttack 39.34%, FOA-Attack 33.44%; untargeted is 48.06% and 38.77%; the authors state improvements over FOA-Attack of 5.90% and 9.29%. Table 7 closed-source average ASR: targeted 36.37% vs. 31.02%, untargeted 48.90% vs. 42.68%; the authors state improvements of 5.35% and 6.22%. The MME tables do not report AvgSim.

Defenses, Noise, and Figures

• Table 8: Average results over four open-source black-box models. Under DiffPure, targeted ASR is MPCAttack 38.50%, FOA-Attack 17.43%, M-Attack 15.05%; under Noisy, it is 50.85%, 30.85%, 27.75%. For untargeted under DiffPure/Noisy, MPCAttack is 78.13% / 83.03%. The paper does not specify the dataset. The authors state that MPCAttack degrades less than the other two methods under these settings.
• Figures: The authors state that in Figure 3, MPCAttack's perturbations better capture target image features; the main text does not provide verifiable pixel or norm values. Figure 4 shows that closed-source models' descriptions of adversarial images are drawn toward the target semantics. In Figure 1(d), the authors state MPCAttack outperforms other methods in the figure; the main text does not list per-model readings.
• Supplementary visualization: Figure 8 shows commercial MLLMs' descriptions of adversarial images. The paper does not provide ASR for these examples.

Other Ablations and Analyses

• Removing components (Figure 5, dataset not specified in the paper): The authors state that removing any paradigm or MPCO reduces ASR and AvgSim; removing cross-modal alignment causes the largest drop; removing MPCO is particularly weak on GLM-4.1V-9B-Thinking. The main text does not provide bar heights.
• Changing backbones (Figure 5): The authors state that replacing CLIP with SigLIP2 reduces transferability; replacing InternVL3-1B with InternVL3-2B improves it while increasing computational cost; the difference between DINOv2 and DINOv3 is small. The main text does not provide ASR.
• λ (Figure 6, average over open-source MLLMs, dataset not specified in the paper): The authors state that too small λ over-relies on text, too large emphasizes images; intermediate values give the best trade-off; λ = 1 (image only) is not optimal. The main text does not provide ASR for each λ.
• Old methods with Multi-Paradigm (Table 9, dataset not specified in the paper; M-Attack, FOA-Attack, and MPCAttack rows are the same as Table 1): For targeted, M-Attack+Multi-Paradigm ASR across four models is 24.2%, 79.7%, 69.6%, 33.0%; FOA-Attack+Multi-Paradigm is 32.7%, 84.3%, 72.9%, 42.4%. On Qwen, the latter's 32.7% is higher than MPCAttack's 32.5%.
• Table 9 counterexample: On GLM-4.1V-9B-Thinking untargeted, M-Attack+Multi-Paradigm's ASR is 59.2%, lower than M-Attack's 61.2%; AvgSim is 0.3730, higher than M-Attack's 0.3696 (higher is worse for untargeted). This contradicts the main text's claim that "ASR and semantic similarity improve on all victim models." For targeted on the same model, ASR goes from 31.0% to 33.0%, and AvgSim from 0.3118 to 0.3088.
• τ and ω (Table 10, dataset not specified in the paper): The authors state that too small τ makes optimization unstable, too large weakens contrastive guidance, τ = 0.2–0.4 is better; too small ω is detrimental to targeted alignment, too large reduces discriminability, ω = 2–3 is better. For targeted, τ = 0.05, ω = 2, the four-model ASR is 6.6%, 38.1%, 37.1%, 21.3%; for τ = 0.2, ω = 0.5, it is 11.9%, 36.9%, 31.7%, 22.7%. The row for τ = 0.2, ω = 2 in this table differs from MPCAttack in Table 1, and the paper does not explain why.

Limitations and future directions noted by the authors

The paper has no Limitations section, no limitation list in the Discussion, no Future Work, and the conclusion does not frame any result as an open problem. Section 5 only states that experiments show current MLLMs still have vulnerabilities exploitable by transferable attacks, and that MPCAttack plays an important role in designing more effective and more transferable adversarial examples. The limitation mentioned in Section 12 of the supplementary material refers to the fact that adding Multi-Paradigm directly to M-Attack and FOA-Attack still leaves the optimization mutually independent, not that MPCAttack itself is listed as a limitation. Within the above scope, the paper does not specifically discuss limitations.

Experimental coverage

• The black-box victim models in Section 4.1 are 4 open-source and 4 closed-source MLLMs. Figure 4 and Figure 8 additionally show responses from Gemini-2.5-flash and Claude-4.5-Sonnet, but the result tables do not report their ASR.
• Quantitative data are from an ImageNet subset of 1,000 images, Flickr30K with 1,000 images, and MME with 1,187 images; MME evaluation uses only yes questions. Flickr30K evaluation uses annotated captions, while ImageNet and closed-source models mostly use model-generated responses and then compute GPTScore.
• The main comparison methods are AnyAttack, CoA, X-Transfer, M-Attack, and FOA-Attack. Tables 1–4 for targeted attacks do not report X-Transfer; Tables 6–7 for MME do not report AnyAttack, CoA, or X-Transfer.
• Defense-related results are DiffPure and Noisy in Table 8, with metrics being the average ASR and AvgSim of four open-source black-box models. The paper does not specify the dataset for that table, nor does it report closed-source models there.
• The main surrogates are CLIP in three sizes, InternVL3-1B, and DINOv2; the ablation additionally uses SigLIP2, InternVL3-2B, and DINOv3. The perturbation is ℓ∞ with ϵ = 16/255, step size 1/255, and 300 iterations. The paper does not report the number of repetitions, the number of queries to victim models, or the value of µ in Algorithm 1.

MPCAttack is a transferable visual adversarial attack against MLLMs, where the same optimization serves both targeted and untargeted evaluations.

• Problem: Existing transferable attacks mostly use white-box surrogates from a single learning paradigm and independently optimize each feature space. The authors believe this narrows the perturbation search space and causes perturbations to overfit a single paradigm's representation bias.
• Method: CLIP, InternVL3-1B, and DINOv2 extract image features; descriptions generated by the understanding model are then incorporated into the cross-modal branch via a text encoder. After ℓ2 normalization and concatenation, contrastive matching pulls adversarial features toward the target and pushes them away from the source image. The budget is ℓ∞, ϵ = 16/255, with 300 iterations, then transferred to black-box MLLMs.
• ImageNet: Average ASR across four open-source models is 63.33% targeted and 92.10% untargeted (Table 1); across four closed-source models, 63.38% and 90.55% (Table 2). The authors state that MPCAttack outperforms the baselines in these tables. Claude-3.5's targeted ASR is 8.2%.
• Other data: Flickr30K open-source targeted average ASR is 29.54% (Table 3). The authors believe the overall ASR on this dataset is low, possibly due to five captions per image; among them, Qwen2.5-VL-7B's untargeted ASR is lower than X-Transfer. On MME, the authors claim that relative to FOA-Attack, the open-source average ASR improves by 5.90% targeted and 9.29% untargeted.
• Authors' conclusion: Multi-paradigm synergistic optimization can produce semantically more consistent and more transferable perturbations; current open-source and closed-source MLLMs still exhibit vulnerabilities exploitable by such samples. The paper does not list future work separately.