FusionAudit: Path-Conditioned Robustness Auditing for Natively Multimodal Models

In native multimodal VLMs, the visual channel is both perceptual input and a carrier of rendered text; single-attack, pixel-centric evaluations collapse different attack surfaces into one scalar.

• Scenario: The authors write that early systems feed a frozen visual encoder through a projection bottleneck into the language model; native architectures perform multi-layer visual injection and early fusion, where pixels, scene text, and document fields compete with user prompts in a unified representation. The authors argue that robustness evaluation has not kept up, still split into pixel adversarial and typographic overlay.
• Gap: On the pixel path, attacks anchored on the visual encoder cannot see deep decoder gradients; end-to-end PGD spreads gradients across all output tokens, reducing the position where the model changes its answer. On the rendered-text path, the authors state that native models attend heavily to overlaid text but refuse to treat it as the task answer, so direct typographic overlay does not fully test it.
• Observation and approach: The authors state there is a see-but-not-deceived signature, and that in deep fusion encoder objectives and answer-level objectives are misaligned. They propose the L1–L3 nativeness taxonomy, the audit framework PaRMA and its instance FusionAudit, and use diagnostics to design MANTIS and Nat-Evid. The primary scalar is pathway-complete residual risk, which the authors call an empirical lower bound. Layer trends are written as correlations, not causal.
• Threat model:
  • Goal: Estimate the failure probability of the union of pixel and rendered-text interventions on the clean-correct set, and check whether failures carry behavioral signatures that can guide stronger attacks.
  • Knowledge: The text prompt is fixed; only the image is modified. Pixel attacks are white-box, infinity-norm-bounded PGD; the authors write that MANTIS requires white-box gradients. Rendered-text insertion uses task-matching incorrect answers.
  • Victim system: 18 open-source VLMs, organized by alignment era, shallow fusion, deep fusion, or early fusion. The authors state that no audited open-source model reaches the envisioned L4.
  • Judgment: ASR and risk are computed only on the subset answered correctly before intervention; samples in that subset answered incorrectly count as failures. The paper does not define success as jailbreak compliance.

Paper Section 2 divides related work into native VLMs, adversarial robustness, evaluation and diagnosis.

Native vision-language models

• Modular pipelines: Early VLMs often use frozen or lightly adapted vision encoders, connected to language model via projection, query modules, or connectors. Paper writes BLIP-2, InstructBLIP, MiniGPT-4, LLaVA show this alignment-era pipeline can do VQA, description, and visual instruction tuning; later work compares vision backbones, connectors, and training recipes.
• Native architectures: Paper writes recent models shift to multi-layer visual injection or early fusion shared backbone, visual and text interact earlier or deeper. Authors consider visual channel no longer just front-end conditioning signal; evaluation cannot reuse modular pipeline assumptions.

VLM adversarial robustness

• Pixel attacks: Paper cites Carlini and Wagner (2017) and Madry et al.'s PGD (2018): small-norm pixel perturbations can expose vision model vulnerabilities. Later work connects these methods to VLM encoder representations, cross-modal alignment, or final answers, and writes adversarial examples can transfer across models.
• Typography and split evaluation: Paper writes rendering text into images can change VLM interpretation of visual content. Authors point out existing evaluations usually treat pixel perturbation and text overlay as separate threats, without systematically measuring joint risk across visual pathways on native VLMs.

Evaluation and diagnostic audit

• Static suites: Paper cites Athalye et al.: single attack may underestimate risk, leading to fixed attack sets like AutoAttack. Multimodal evaluations start aggregating multiple perturbations, but authors write most still rely on pre-given static suites.
• Post-hoc diagnosis: Representation similarity, attention analysis, and activation patching are used to look at internal representations. Authors point out these diagnoses usually explain after failure, not to guide subsequent stronger attacks.

Baselines and data

• Baseline methods: Pixel pathway is Vis-PGD and End-PGD; rendered text pathway is Typo-Evid, authors write following Cheng et al. Evaluation set consists of ChartQA, DocVQA, MME, total 1500 questions.
• Positioning: Authors write PaRMA as putting pathway diagnosis into audit, reporting residual risk after diagnosis strengthening, not taking maximum over pre-specified attack set.

PaRMA does not turn auditing into a fixed attack checklist: baseline attacks provide failures and diagnostics, diagnostics then drive enhanced attacks, and finally the strengthened union failure rate is used as a falsifiable residual risk. The instance FusionAudit covers pixel perturbations and rendered text insertion (Figure 1).

Taxonomy and audit scope

• L1: frozen or lightly adapted vision encoders are connected to the language model via a projection, with modalities structurally separated.
• L2: large-scale joint multimodal pretraining; visual tokens enter the backbone through a connector but do not re-enter at deeper layers, and the backbone processes a fixed visual prefix.
• L3: visual information participates in the entire language computation, including DeepStack with multi-layer re-injection, and Early Fusion which shares transformer blocks from the first layer. The paper writes Qwen2.5-VL, Qwen3-VL, Qwen3.5 as one family, a comparison chain from shallow to deep, without annotating subtypes one by one in the list.
• Scope: the model maps image xv and text xt to an answer. To separate from text-only jailbreaks, xt is fixed and only xv is intervened on. The authors write that fusion level covaries with model family, scale, and training data; the taxonomy is an organizing axis, not a causal ladder. The envisioned L4 (single backbone, no separate vision tower) is not reached by any audited open-source model.

Pathway-conditioned risk

• πpix: the perturbation's infinity norm does not exceed ε, and the intervention image is the clipped xv plus the perturbation.
• πtxt: candidate strings have length at most L, positions are taken from an allowed set; the intervention renders the string into the image at the positions.
• Risk: on the clean-correct set, the subset flipped by attack a is denoted Sa. The pathway-complete risk is the empirical frequency of the union of flipped sets across pathways and attacks:

\widehat{R}_{\mathrm{FA}}(f_\theta;D,\mathcal{A})=|\bigcup_k\bigcup_{a\in\mathcal{A}_k}S_a|/|C_\theta(D)|

The meaning is: among samples answered correctly before intervention, the proportion that are answered incorrectly by at least one visual intervention in the suite. The authors state that it strengthens with diagnostics, so it is a lower bound, not a static union. The pixel-ball radius and PGD steps are given in the experimental settings.

Baselines and diagnostics

• Vis-PGD: minimizes the cosine similarity between the intervention image and the clean image in the vision encoder features within the ε ball.
• End-PGD: within the same ball, maximizes cross-entropy against the teacher-forced full correct answer.
• Typo-Evid: overlays a task-matching wrong answer string on a fixed region. The paper writes that it follows Cheng et al., and does not expand the overlay content here.
• Two readings: RUG is the proportion of times the string can be read under explicit reading, minus the proportion of times the model answers incorrectly under the task prompt and the answer contains the string:

RUG(f_θ)=P_read(f_θ)−P_use(f_θ)

OAR uses the attention from text tokens after the image to image patches, averaged over heads and later decoding layers, then computes the attention share of the overlay region relative to its area ratio. The authors write that under the task prompt OAR is greater than 1 and RUG is positive, consistent with see-but-not-deceived, and consider that Typo-Evid has refusal caused by visual salience.

Diagnostic-driven enhancement

• MANTIS: the correct continuation and a deterministic wrong continuation from the reference answer are fixed before optimization. The margin is only pushed on the first token where they diverge: raising the log-probability of the wrong token and lowering the log-probability of the correct token, with the constraint still that the infinity norm does not exceed ε. The reason the authors give is that End-PGD spreads the gradient over the whole sequence.
• Nat-Evid: uses style-conditioned image editing to draw the same wrong answer as content consistent with the host, such as chart labels or scene text, rather than an externally overlaid string. The wrong answer is fixed before generation; the editor is asked to only add text, not modify or delete original content. The authors use it to test whether host-consistent rendering can narrow "read but not use".
• Delivery: adding these two yields R+. R+ minus R0 is the risk additionally exposed by diagnostics. The authors state that R+ is the main deliverable and is falsifiable.

Success determination

• The risk denominator is the clean-correct set size, and the numerator is the union flipped by the attack suite. The determination function J marks a sample as entering the flipped set when the output is judged as a failure relative to the reference answer.
• The rendered-text diagnostic additionally uses the reading event and the "answered incorrectly and contains the wrong string" event, which are not the same event as task ASR.
• The paper does not write the string-matching rule of J, nor name the judge in Figure 1.

Experimental Setup

• Models: 18 open-source VLMs. L1 includes SmolVLM2-2.2B, BLIP2-OPT-2.7B, InstructBLIP-Vicuna-7B, LLaVA-1.5-7B. L2 includes Qwen-VL-Chat, Qwen2-VL-2B, Qwen2.5-VL-3B, Qwen2.5-VL-7B, Gemma3-4B, Gemma3-12B, DeepSeek-VL-7B. L3 includes Qwen3-VL-2B, Qwen3-VL-4B, Qwen3-VL-8B, Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Gemma4-E4B.
• Data: A fixed set of 1500 questions, balanced across ChartQA, DocVQA, and MME. Of MME's 500 questions, 460 (92%) come from natural image categories (400 artwork, 60 celebrity), and 40 are code reasoning.
• Protocol: Each attack uses the same 1500 questions. ASR and risk are conditioned on each model's own clean-correct set. Pixel attacks use ε=8/255 and 40-step PGD. When GPU memory is insufficient, perturbations are computed on a downscaled image, only the noise field is upsampled, and it is added back at original resolution before inference. The paper does not state which models trigger downscaling.
• Suite: Baselines are Vis-PGD, End-PGD, Typo-Evid; enhancements are MANTIS, Nat-Evid. δ is the proportion of clean-correct samples flipped by an enhanced attack and not flipped by any corresponding baseline. Layer-level means are unweighted macro averages. The table header has a Clean column, but the table note does not define it in a sentence.
• Diagnostic subsets: Layer recovery uses the first 40 End-PGD flips per model sorted by identifier, retaining 24–40 reproducible ones. The stability ablation uses 100 questions per dataset, 300 total, across all 18 models.
• Not specified: Which image generator is used for Nat-Evid in Table 1; the implementation of J; the number of repetitions for the main 1500-question audit. Figure 1 has a judge box, but the paper does not name it.

Main Results

The table below lists only 8 models. Numbers are percentages on the clean-correct set, according to Table 1. R+ is the pathway-complete risk after enhancement. SmolVLM2-2.2B with the highest R+ is included.

Swipe horizontally to read the table

Model	End-PGD	MANTIS	Typo-Evid	Nat-Evid	R+
SmolVLM2-2.2B (L1)	80.4%	99.5%	17.2%	30.9%	100.0%
BLIP2-OPT-2.7B (L1)	43.1%	63.0%	69.4%	4.4%	91.2%
Qwen2.5-VL-3B (L2)	66.6%	59.5%	19.5%	36.6%	85.7%
Gemma3-12B (L2)	35.7%	69.5%	14.1%	34.0%	81.5%
Qwen2.5-VL-7B (L2)	45.4%	59.0%	15.6%	33.9%	79.5%
Qwen3-VL-4B (L3)	37.4%	61.1%	12.6%	29.9%	78.9%
Gemma4-E4B (L3)	46.7%	72.4%	8.9%	27.1%	88.7%
Qwen3.5-9B (L3)	38.3%	39.2%	16.1%	25.1%	59.5%

Omitted: InstructBLIP-Vicuna-7B, LLaVA-1.5-7B, Qwen-VL-Chat, Qwen2-VL-2B, Gemma3-4B, DeepSeek-VL-7B, Qwen3-VL-2B, Qwen3-VL-8B, Qwen3.5-2B, Qwen3.5-4B. Among these, InstructBLIP-Vicuna-7B has R+ of 99.5%, and DeepSeek-VL-7B has 99.2%.

• Layer gap: The authors report from baseline union to enhanced union, L1/L2/L3 macro-average increases of +10.6%, +16.9%, +25.1%; fixed-model-pair bootstrap 95% CIs are [9.2, 12.0], [16.1, 17.8], [24.2, 26.0] percentage points, none containing 0. The proportion flipped only by MANTIS is 15.2%, 17.6%, 24.6%. Nat-Evid exclusive flips are 10.4%, 23.7%, 20.6%, which the authors call large but non-monotonic. Absolute R+ is 97.0%, 88.4%, 75.6%; the authors write that absolute risk is lower on newer models, but the diagnostic gap remains.
• Ranking: The authors argue that End-PGD as a default proxy is insufficient for native architectures. Spearman ρ with R+ ranking is 0.837, Kendall τb is 0.634; 8 of 18 models move at least three positions. The authors state that Gemma3-12B ranks 2nd under End-PGD and 7th under R+.
• Family and exceptions: The text also gives R+ for Qwen3-VL-2B as 82.7%, Qwen3-VL-8B as 73.1%, Qwen3.5-2B as 78.0%, Qwen3.5-4B as 68.1%. The authors state that the direction within the family is consistent with the layer trend in the table and believe it exposes vulnerabilities related to the native transition, not just differences between old and new model families. In the table, Qwen2.5-VL-3B's MANTIS is lower than End-PGD; BLIP2-OPT-2.7B's Nat-Evid is much lower than Typo-Evid.

Pixel Pathway: Stability and Layer Recovery

• Stability: On 300 questions and three seeds, the authors state that MANTIS's min-max variance is small relative to absolute ASR, with a median standard deviation of 1.3%. Sweeping steps from 10 to 50, the authors state that ASR largely saturates at 30–40 steps; the layer-mean gain from adding up to 50 steps is +2.8%, +1.0%, +2.2% (L1–L3). With ε at 2, 4, 8/255, the authors state that the relative order of L1, L2, L3 is maintained. The text does not give per-model bar heights for Figure 2.
• Target mismatch: On the L3 macro average in Table 1, Vis-PGD is 16.5%, End-PGD is 39.8%, and MANTIS is 56.8%. The authors state this supports that "end-to-end loss spreads gradients, and concentrating gradients on answer-discriminative tokens can flip more failures."
• Layer recovery: 16 models (Qwen-VL-Chat and DeepSeek-VL-7B are excluded because they do not support activation hooks). Replacing the decoder prefix hidden states of End-PGD flipped samples with values from the clean image, greedy decoding, and counting recovery if the correct answer is recovered at any depth. Macro averages are L1 92.2% [87.6, 96.1], L2 62.6% [55.5, 69.6], L3 25.6% [20.7, 30.7]. The Qwen family goes from 70.4% (Qwen2-VL-2B and Qwen2.5-VL-3B/7B) to 21.3% (Qwen3-VL and Qwen3.5). The authors state that in alignment-era models, recovering early-layer prefixes usually undoes the attack; in deep fusion, most flips persist at every depth.

Rendered Text: Seen but Not Adopted

• Two text attacks: The authors state that Typo-Evid is strongest at L1, driven by one alignment-era model; not named. In Table 1, that layer's Typo-Evid is BLIP2-OPT-2.7B's 69.4%, with the other three rows at 17.2%, 10.8%, 8.5%. Nat-Evid is higher than Typo-Evid by 13.6% at L2 and 15.0% at L3 (layer means 34.5% vs. 20.9%, 28.2% vs. 13.2%).
• RUG: Table 2's unweighted layer means are L1 Pread 38.9, Puse 22.4, RUG +16.5; L2 68.6, 15.9, +52.7; L3 81.0, 11.6, +69.5. The authors interpret RUG as accessibility relative to adoption and state that together with OAR it supports see-but-not-deceived, leading to Nat-Evid.
• OAR: Relative to matched clean images, average OAR increases are L2 +138%, L3 +135%, L1 +13%. From L2 to L3, Typo-Evid ASR goes from 20.9% to 13.2%. The right side of Figure 3 is an MME sample from Qwen3.5-2B: OAR goes from 2.49 to 4.85, and the caption states that attention is concentrated on the injected banner (blue box), while the model still outputs the visual answer. The banner text is not reproduced.

Other Ablations and Analyses

• Generator (Figure 4): The authors state that the Gemini variant is strongest and driven by MME; the average ASR for GPT-Image-2 and the two Qwen variants is written as about 19%. The legend is Gemini, Qwen-Pro, GPT-Image, Qwen-Edit, while the text writes GPT-Image-2; the names are not fully aligned.
• Evidence preservation: On Qwen3-VL-8B with 100 MME images where the original image is answered correctly and Nat-Evid is answered incorrectly, covering the inserted clue with black blocks restores correct answers in 98/100. On charts and documents, in samples where non-generative OCR can read the answer from the original image, the edited image exactly recovers the ground-truth string in 92.3%; in the remaining manual review, evidence is destroyed in 0.85% of images.
• MME natural images only: Any layer-mean ASR change is no more than 3.2 percentage points; L3 MANTIS exclusive-flip is 37.0%, while on all 500 MME images it is 36.4%. This is an MME subset, not Table 1's 24.6%.
• Exceptions: L1 macro-average Nat-Evid 15.3% is lower than Typo-Evid 26.5%; on DeepSeek-VL-7B Nat-Evid 29.9% vs. Typo-Evid 31.1%; on Qwen3.5-4B 24.5% vs. 27.2%; on Qwen3.5-9B the in-table gain of MANTIS relative to End-PGD is +0.9. The authors do not provide additional mechanistic explanations for these individual cases.

Limitations and future directions noted by authors

• Classification is not causal ladder (Conclusion and Limitations): L1–L3 are observational framework for audit, not strict causal ladder. Stricter L4 is not in empirical audit; Chameleon points to that design, but Chameleon paper reports image tokenizer limitations on text-dense images, which are core to ChartQA, DocVQA, and rendered text settings. Qwen family chain mitigates some confounding, adjacent levels still differ in training data and scale.
• Only wrong answer injection tested (same section): Rendered text pathway only does task-related wrong answer injection. Instruction-following visual prompt injection (e.g., jailbreak) is another security risk requiring compliance-based metrics, not correctness-based ASR.
• Pathway extensible (same section): Currently only two visual pathways instantiated. Authors write framework designed extensible; when VLM extends to audio, video, embodied tool use, new pathways and diagnoses can be added.

Experimental coverage

• Tested models are 18 open-source VLMs listed in main text: L1 four, L2 seven, L3 seven; no audited model reaches authors' defined L4.
• Main audit uses same 1500 questions (ChartQA, DocVQA, MME balanced), ASR conditioned on each model's clean-correct set; pixel attack is white-box PGD, ε=8/255, 40 steps. Authors write MANTIS requires white-box gradients.
• Pixel side also includes 300 questions, three seeds stability, and step and ε scans; layer recovery covers 16 models, Qwen-VL-Chat and DeepSeek-VL-7B excluded due to unsupported activation hooks.
• Rendered text side includes RUG and OAR for 18 models, and Nat-Evid comparison across four image generation variants; evidence preservation check on Qwen3-VL-8B with 100 MME images, plus OCR 92.3% and human review 0.85%.
• Paper does not report repetitions of main 1500-question audit, nor specify which generator in Figure 4 corresponds to Nat-Evid in Table 1; paper does not state which models compute perturbations on scaled images due to memory.

FusionAudit is a vision-pathway audit for native VLMs: first use diagnostics to find blind spots related to fusion style, then estimate residual risk after stronger attacks.

• Problem: When the pipeline of a frozen encoder plus projection becomes deep fusion, pixels and text in images compete with the prompt. The authors argue that single-scalar methods like End-PGD alone cannot give a stable ranking of native models.
• Approach: L1–L3 serve only as an organizational axis. PaRMA fixes the text and modifies only the image. On the pixel side, encoder cosine and full-sequence cross-entropy serve as baselines, then the marginal attack MANTIS targets the first-divergence token. On the text side, direct typographic overlay serves as a baseline, and Nat-Evid, which edits text consistent with the host, tests "read but not adopted." The primary metric is the union failure rate of the two pathways, computed only on questions answered correctly before intervention.
• Main results: On ChartQA, DocVQA, and MME totaling 1500 questions and 18 open-source models, the L3 macro-average pathway-complete risk ranges from 50.5% to 75.6% (the authors write +25.1%, 95% CI [24.2, 26.0] percentage points). The proportion flipped only by MANTIS at L3 is 24.6%; the RUG at L3 is +69.5%. The Spearman ρ between End-PGD and R+ rankings is 0.837, and 8 of 18 models move at least three positions. Absolute R+ goes from 97.0% at L1 to 75.6% at L3, and the direction within the Qwen family is consistent.
• Exceptions: Stronger attacks are not higher on every model. Qwen2.5-VL-3B has MANTIS at 59.5%, lower than End-PGD at 66.6%; BLIP2-OPT-2.7B has Nat-Evid at 4.4% and Typo-Evid at 69.4%.
• Authors' conclusions: The authors argue that robustness audits should change with architecture, and FusionAudit provides a pathway-complete template. Classification is not a causal ladder; rendered text is not yet an instruction-injection evaluation. The authors write that the same attack could also mislead deployed VLMs, and that what is released is the audit procedure, not precomputed adversarial images.