XSPA: Constructing Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks Against VLMs

With fixed geometry and very small support, is the perturbation already sufficient to disrupt cross-task semantics in the VLM shared space.

• Scenario: The authors argue that CLIP, BLIP-2, etc., rely on shared visual–textual representations to support zero-shot classification, image captioning, and VQA. If a perturbation affects visual encoding, it may propagate through this space, causing a single input to influence multiple decisions simultaneously; the authors claim this is particularly concerning in interactive and decision-support scenarios.
• Existing limitations: The authors categorize existing visual attacks into dense full-image perturbations under Lp constraints, and local or physical interventions such as patches, shadows, and optical manipulations. They argue that these settings typically still retain significant freedom in support, shape, or optimization space, and therefore are insufficient to answer whether VLMs remain vulnerable when degrees of freedom are simultaneously constrained.
• Core hypothesis: The authors do not treat perturbation shape as a free variable but impose a strict geometric prior to probe at how small a geometric degree of freedom multimodal alignment becomes unstable.
• This paper proposes: They propose XSPA (X-shaped Sparse Pixel Attack), which restricts perturbations to two intersecting diagonal lines and jointly optimizes within this support a classification objective, cross-task semantic guidance, and magnitude and along-line smoothness regularization. In the default setting, the authors claim only about 1.04% of pixels are modified.
• Threat model:
  • Goal: Induce transferable misclassification and drive semantic drift in captioning and VQA while remaining visually subtle. The classification loss includes a non-targeted margin and targeted cross-entropy.
  • Knowledge: The paper is written as white-box surrogate / black-box downstream. Gradient optimization is performed on a CLIP-style surrogate; the appendix states that downstream captioning and VQA evaluations do not adapt the perturbation to those models.
  • Capability: Only pixels within a pre-fixed X-shaped binary mask can be modified, subject to an infinity-norm bound. The optimizer cannot choose the support otherwise.
  • Victim systems: Zero-shot classification targets four CLIP-style vision encoders; captioning and VQA target six downstream VLMs. Success is primarily measured by accuracy drop, T-ASR, U-ASR, and GPT-4 consistency or correctness on open-ended outputs.

The paper organizes related work along three lines and lists baselines separately in experiments.

Sparse and Structured Visual Attacks

• Early adversarial examples: Goodfellow, Shlens, and Szegedy (2015), Madry et al. (2018), Moosavi-Dezfooli, Fawzi, and Frossard (2016), Xie et al. (2019), Andriushchenko et al. (2020) discuss the vulnerability of deep networks to perturbations under various threat models. The paper treats these as background and does not compare each one against XSPA.
• Sparse and physical forms: Su, Vargas, and Sakurai (2019), Modas, Moosavi-Dezfooli, and Frossard (2019), Zhong and Liu (2025) modify only a few pixels or restricted regions. Brown et al. (2018), Ran et al. (2023), Zhong et al. (2022), Huang and Ling (2022), Hsiao et al. (2024) use patches, shadows, or natural lighting. The authors argue these methods still primarily optimize support size, patch area, or physical realizability, rather than testing a fixed, highly constrained geometric prior.

Attacks and Robustness of Vision-Language Models

• Attacks in shared spaces: CLIP by Radford et al. (2021) makes robustness across cross-modal embeddings a core issue. Zhang, Yi, and Sang (2022), Guo et al. (2023), Zhao et al. (2023), Cui et al. (2024) discuss how perturbations affect retrieval, classification, and generation through shared representations, and the vulnerability of large VLMs in classification and open-ended settings.
• Transferability and cross-task design: Zhang et al. (2025), Xie et al. (2025), Liu, Ouyang, and Cui (2025), Zhao et al. (2025), Liu et al. (2025), Li et al. (2026a,b) improve attack transferability or unify cross-task design. The paper describes these without detailing differences from XSPA for each.
• Robustness methods: Wang et al. (2024), Schlarmann et al. (2024), Tong et al. (2025), Sheng et al. (2025), Dong et al. (2025), Mirza et al. (2026) explore certification, adaptation, and test-time conditioning.

Automatic Evaluation for Open-Ended Tasks

• Lexical metrics: Zhao et al. (2023) and Cui et al. (2024) note that exact match and lexical overlap often fail to capture semantically meaningful changes.
• LLM judges: Zheng et al. (2023) and OpenAI (2024) discuss LLM-as-a-Judge. The authors adopt GPT-4 to judge semantic correctness and factual consistency, enabling comparison of classification, description, and VQA under the same protocol.

Baselines and Datasets

• Baseline methods: NaturalLightAttack, ShadowAttack, and ITA correspond to natural light, shadow, and illumination transformation attacks. The authors state their threat models do not fully align with fixed sparse geometry, so comparisons should be seen as robustness references rather than strictly matched rankings.
• Benchmarks/datasets: The main evaluation follows the ITA protocol, using 300 images from COCO-80 shared across three tasks; the zero-shot label space is COCO-80. Table 4 additionally reports COCO-1000, ImageNet-1K, and ImageNet-R.

The authors claim that, to their knowledge, this paper presents the first X-shaped sparse pixel attack that constrains perturbations to a fixed geometry with very small support, and within that region simultaneously uses momentum optimization, input diversity, cross-task semantic guidance, and source semantic suppression. The authors treat XSPA both as an attack and as a controlled diagnostic to characterize the vulnerability boundary of the shared vision–text space under strongly constrained structural perturbations.

XSPA first fixes an X-shaped sparse support, optimizes the perturbation only within that support, then transfers to downstream VLMs.

Problem setting

• Image x belongs to 3×H×W in [0,1]. The perturbation δ can only fall on a fixed binary mask M, and the adversarial image is x_{\mathrm{adv}}=\mathrm{clip}(x+M\odot\delta,0,1), i.e., unchanged outside the mask, with the result clipped to the legal pixel range.
• The setting is written as white-box surrogate / black-box downstream: optimize on a CLIP-style surrogate, then transfer to zero-shot classification, image captioning, and VQA. Unlike full-image Lp-bounded attacks, the modifiable pixels are given in advance.
• The appendix algorithm uses white-box gradients for the surrogate encoder; downstream models are not attacked by modifying the perturbation for them during evaluation.

X-shaped sparse mask

• Two diagonal line segments intersect at cctr=(ρcol W, ρrow H). Given an angle set A={θ1, θ2}, relative length r, and line width b, the support is the union of each segment taken at the proportion of the shorter side, then expanded to width b.
• The sparsity ratio ηM is the number of support pixels divided by HW. At the default 384×384, the support is 1,532 pixels, and the authors write ηM≈1.04%. The appendix states the implementation samples two center lines, rasterizes them, then does a neighborhood expansion of width b, clipping out-of-bounds coordinates to the image boundary.
• The authors' rationale: fixing the support before optimization separates geometric effects from freely choosing pixels, avoiding the optimizer distributing the perturbation to visually more convenient regions; transferability can only come from the values on this X structure.

Cross-task joint objective

• The objective on the fixed support is L=λ_clip^(t)L_clip+λ_tar^(t)L_tar+λ_src^(t)L_src+L_smooth. The first three terms are classification attack, cross-task semantic guidance, and source semantic suppression, and the last term is a structural regularizer.
• Lclip uses the source class logit minus the maximum logit of other classes in the untargeted case, and cross-entropy in the targeted case. The target text pool and source text pool come from target labels, description-side semantic drift, and VQA-side answer shift. Ltar is the negative average of the inner product of image features with target texts, and Lsrc is the average of the inner product with source texts.
• The authors state the two types of texts are encoded by the same frozen text encoder, hence lie in the same embedding space, and define a semantic direction independent of the specific task. The smoothness term is further split into a magnitude term and an along-line term: the former suppresses isolated spikes, and the latter makes adjacent pixel values close on each ordered center line.
• For the five weight types clip, tar, src, mag, line, one set of coefficients is used when t is less than γN, then another set is used afterward. The authors state the early stage emphasizes classification guidance to stabilize the direction, and the later stage strengthens semantic misalignment to promote cross-task transfer. The paper does not give the values of these coefficients or γ.

Mask-constrained optimization

• The feasible set requires the perturbation to be 0 outside the mask, and the infinity norm not to exceed ε. At each step, random scale-fill input diversity is applied to the current adversarial image, the joint gradient is computed, then momentum is accumulated in the MI-FGSM manner; the denominator is the mean of the absolute gradient values plus 10 to the power of negative 12.
• The update takes the sign step of the momentum, then projects back to the mask and infinity norm bound. The authors state input diversity reduces overfitting to the surrogate, momentum stabilizes the direction, and projection guarantees the change stays within the X-shaped support. The appendix gives full pseudocode and states momentum and diversity correspond to MI-FGSM and DI-FGSM, respectively.
• The number of iterations in the main experiments is given in the experiments section as 200. The method section does not list the step size, ε, or the specific surrogate model name for the main experiments.

How success is determined

• Zero-shot classification uses Top-1 accuracy and its absolute change relative to clean samples. T-ASR is the proportion pushed to a predefined target, and U-ASR is the untargeted misclassification proportion; higher means a stronger attack.
• Captioning and VQA do not rely solely on exact matching. GPT-4 evaluates consistency and correctness under a fixed LLM-as-a-Judge prompt and scale; GPT-4o evaluates naturalness on a four-point scale, covering visual naturalness, physical consistency, and adversarial plausibility.

Experimental Setup

• Data: Following the ITA protocol, 300 images were selected from COCO-80 as a unified evaluation set for the three tasks. The appendix states this is 300 COCO validation images covering the COCO-80 label space; classification metrics use all 300 image IDs.
• Models under test: Zero-shot classification uses OpenCLIP ViT-B/16, Meta-CLIP ViT-L/14, EVA-CLIP ViT-G/14, OpenAI CLIP ViT-L/14. Downstream models are LLaVA-1.5 (7B), LLaVA-1.6 (7B), OpenFlamingo (3B), BLIP-2 (FlanT5XL ViT-L, 3.4B), BLIP-2 (FlanT5XL, 4.1B), InstructBLIP (FlanT5XL, 4.1B). Table 2 and Table 3 list the image encoders of the first two models as OpenAI CLIP ViT-L/14 and the last four as EVA-CLIP ViT-G/14.
• Surrogate and implementation: The main experiments are optimized on a CLIP-style surrogate and then transferred, with 200 iterations, using a single NVIDIA RTX 4090 24GB; other schedules are said to follow the method section. The paper does not specify the exact surrogate name, step size, or ε for the main experiments. The ablation freezing protocol specifies the surrogate as OpenCLIP ViT-B/16.
• Baselines: NaturalLightAttack, ShadowAttack, ITA. The authors state that the threat models do not fully match, so the comparison is a robustness reference, not a strictly same-condition ranking.
• Metrics and evaluation: Classification reports Top-1 and absolute change relative to clean samples. Naturalness is scored by GPT-4o on a four-point scale. Description consistency and VQA correctness are scored by GPT-4 under fixed prompts and scales; table notes state that lower scores indicate the attack better matches its objective.
• Repeats and appendix samples: The paper does not report the number of repeats for Tables 1–3. The appendix matching budget uses three optimization seeds. The strictly targeted VQA plan is 380 questions on 224 images; the common eligible images for joint analysis are 214.

Main Results

Zero-shot Top-1 from Table 1 and GPT-4o naturalness for XSPA are as follows. Clean samples have GPT-4o scores of 3.65 for all. Parentheses show absolute change relative to clean samples.

Swipe horizontally to read the table

Encoder	Clean	ITA	XSPA	XSPA GPT-4o
OpenCLIP ViT-B/16	97	46 (↓51)	30 (↓67)	2.71 (↓0.94)
Meta-CLIP ViT-L/14	98	64 (↓34)	60.67 (↓37.33)	2.89 (↓0.76)
EVA-CLIP ViT-G/14	98	84 (↓14)	63.71 (↓34.29)	2.78 (↓0.87)
OpenAI CLIP ViT-L/14	93	51 (↓42)	40.67 (↓52.33)	2.76 (↓0.89)

• The authors state that XSPA's accuracy drops are larger than the three baselines in the table across all four encoders, and it has the highest GPT-4o naturalness among attack methods. They report further drops relative to ITA of 16.00, 3.33, 20.29, and 10.33 percentage points. The abstract states OpenCLIP drops 67.00 points, and Table 1 parentheses show ↓67.
• In the same table, NaturalLightAttack still has 93 accuracy on OpenAI CLIP ViT-L/14, with no drop. The paper does not specify which encoder is the main experiment surrogate, so one cannot treat any row of Table 1 as a white-box result.
• Figure 2 shows class label changes for clean and XSPA images. The authors use Grad-CAM to compare attention; Figure 3 caption states that adversarial responses are more dispersed or leave the main object. The figure shows example labels such as elephant, bear, stop sign, bird, airplane, phone, boat.

Description, VQA, and Qualitative Outputs

GPT-4 scores for XSPA in Tables 2 and 3 are as follows. Parentheses show absolute drops relative to clean samples.

Swipe horizontally to read the table

Model	Image Encoder	Description Consistency	VQA Correctness
LLaVA-1.5	OpenAI CLIP ViT-L/14	46.40 (↓32.20)	47.45 (↓20.55)
LLaVA-1.6	OpenAI CLIP ViT-L/14	44.20 (↓27.90)	48.87 (↓15.13)
OpenFlamingo	EVA-CLIP ViT-G/14	11.60 (↓58.60)	14.29 (↓30.71)
BLIP-2 (FlanT5XL ViT-L)	EVA-CLIP ViT-G/14	21.90 (↓53.20)	18.75 (↓44.25)
BLIP-2 (FlanT5XL)	EVA-CLIP ViT-G/14	59.00 (↓15.96)	19.78 (↓34.22)
InstructBLIP (FlanT5XL)	EVA-CLIP ViT-G/14	40.90 (↓35.60)	44.67 (↓23.33)

• The authors state that XSPA has the lowest description consistency across the six models; OpenFlamingo drops 58.60 points from clean, the largest drop in Table 2. They state that generated descriptions often no longer preserve object categories, local attributes, or scene relationships.
• For VQA, the authors state that XSPA has the lowest correctness on four of the six models and remains competitive on the other two; on three of the four EVA-CLIP image encoder models, XSPA is best. Exceptions are LLaVA-1.6 where ITA is 43.00 (↓21.00) and XSPA is 48.87 (↓15.13); InstructBLIP where ITA is 44.00 (↓24.00) and XSPA is 44.67 (↓23.33). The authors state the largest VQA drop is BLIP-2 (FlanT5XL ViT-L) at 44.25 points.
• Figure 4 provides qualitative examples for the two tasks. The authors state that clean descriptions largely preserve the main object, scene, and attributes, while adversarial descriptions drift in object identity, material, or fine-grained attributes; adversarial VQA shows more errors in category, color, counting, and local attributes.

Matching Budget, Joint Success, and Evaluation Calibration

• Same-budget support selection: The appendix compares XSPA with saliency Top-k, with the surrogate being OpenCLIP ViT-B/16, i.e., the target is the surrogate itself. Both use 1,532 pixels, L∞=0.25, 100 steps, step size 0.03, 300 images, and three seeds. Table 6: Top-k targeted ASR is 94.44±0.38%, XSPA is 83.44±0.77%; untargeted are 94.78±0.19% and 86.78±0.19%, respectively. The authors report the targeted difference (Top-k minus XSPA) as 11.00 percentage points, with a 95% interval from 20,000 image-cluster bootstraps of [7.78, 14.44]; untargeted difference is 8.00 points, interval [5.00, 11.11]. They report that exact paired McNemar tests per seed are significant after Holm correction, with the largest corrected p-values being 8.70×10−6 for main ASR and 7.17×10−4 for untargeted ASR. The authors state that X-shaped support is effective under sparse constraints but is not the strongest support selector under the same pixel budget.
• Three-task targeted joint success: Seed 3407, 214 common eligible images. Description outputs come from BLIP-2 FlanT5-XL, VQA from LLaVA-1.6 Mistral-7B. Table 8: XSPA's CLIP targeted success is 177/214 (82.71%), description targeted success is 0/214 for both, VQA any is 9/214 (4.21%), and joint success with VQA is 0/214. The authors state that this result does not support that three-task targeted failures are correlated, but rather separates the stronger surrogate classification effect from the weaker output-space transfer.
• Evaluation and perception: Table 9 compares GPT-4 and GPT-4o-mini on a LLaVA-1.5 subset, with 30 clean outputs plus 30 XSPA outputs per task. Description ICC(A,1) is 0.301 [0.120, 0.443], Pearson r is 0.580 [0.255, 0.801]; VQA exact agreement is 0.933 [0.850, 1.000], Cohen’s κ is 0.815 [0.526, 1.000]. The authors state this is inter-model calibration, not human agreement; GPT-4's description scores average 3.283 higher than GPT-4o-mini on a 0–30 scale. They therefore treat reviewer-model-based description differences as secondary evidence. In Table 7, XSPA's SSIM is 0.97285±0.00003, LPIPS-Alex is 0.06516±0.00008; Top-k's PSNR is 34.5981±0.0093 dB, MAE is 0.001680±0.000003. The authors state that perceptual evidence is mixed and do not treat attack success as evidence of imperceptibility.

Other Ablations and Analyses

• Iterations, step size, ε, and area: The authors state that increasing iterations, perturbation bound, or area strengthens the attack but increases GPU time, lowers PSNR, or modifies more pixels; larger iteration counts have diminishing returns (Figure 5a–d). After plain-text conversion, the correspondence between endpoints and coordinates in the figures is incomplete, so we do not cite point-by-point.
• X geometry: The authors state that center, angle, and length are moderately sensitive, with center and angle scans being flat; increasing line width or decreasing pixel spacing raises U-ASR along with support density (Figure 5e–j).
• Shape: Under the same 1,532 pixels and final step size 0.03, seven shapes are compared; the authors state that XSPA has the highest U-ASR on Full 300 and Held-out 240 (Figure 5k). The paper does not give specific U-ASR values for each shape.
• Loss: The authors state that no single loss alternative dominates simultaneously in targeted ASR, U-ASR, and feature separation (Figure 5l).
• Cross-dataset U-ASR means (Table 4): COCO-300 is 51.24%, COCO-1000 is 55.92%, ImageNet-1K is 95.01%, ImageNet-R is 62.82%. The authors state that ImageNet-1K is highest, COCO and ImageNet-R are lower, and argue that dataset composition and encoder pretraining affect transferability, not just image count. The surrogate in this section is OpenCLIP ViT-B/16, so its column is the surrogate itself.
• Ablation default configuration (Table 5): 384×384, 1,532 pixels, 100 steps, step size 0.03, L∞=0.25. The main experiment uses 200 iterations, which differs from this frozen protocol.

Limitations and Future Directions Noted by the Authors

• Conclusion: The authors state that the results motivate future consideration of structure-aware defenses and robustness evaluation (Conclusion).
• Support Selection: The authors state that the X-shaped support is effective under strict sparsity constraints, but is not the strongest support selector under the same pixel budget; the geometric contribution should be understood as a controlled structured support stress test, not an attack strength advantage over adaptive saliency selection (Appendix Matched-budget).
• Cross-task Targeting: The authors state that existing results do not support correlated failures across three tasks toward predefined targets, but rather separate stronger surrogate classification effects from weaker output-space transfer (Appendix Same-image Cross-task Joint Success).
• Imperceptibility: The authors state that evidence supports sparse surrogate attacks and a perceptual trade-off, but not universally better, uniformly imperceptible, or stable three-task targeted transfer (Appendix Statistical and Artifact Audit).
• Human Evaluation: The authors state that human perception studies and human–GPT studies are not yet complete and are not reported as findings; there are currently no independent human ratings, so existing calibration does not constitute human validation (Appendix Judge Calibration and Audit).
• Caption Review: The authors state that review-model-based caption differences serve only as secondary evidence and are not used alone to support cross-task transfer claims (Appendix Judge Calibration and Claim Boundary).

Experimental Coverage

• Main results cover four CLIP-style encoders and six downstream VLMs, with data from 300 images in COCO; classification uses all 300 image IDs (Experiments; Appendix Evaluation).
• Main experiment baselines are NaturalLightAttack, ShadowAttack, ITA, with 200 iterations, on a single NVIDIA RTX 4090 24GB. The paper does not report the surrogate model name, step size, ε, or number of repetitions for the main experiments (Implementation Details).
• The ablation surrogate is stated as OpenCLIP ViT-B/16; the frozen protocol is 384×384, 1,532 pixels, 100 steps, step size 0.03, L∞=0.25. Table 4 also includes COCO-1000, ImageNet-1K, ImageNet-R, each with 1,000 images.
• Open-task main scores are given by GPT-4, and naturalness by GPT-4o on a four-point scale. The appendix cross-checks GPT-4 with GPT-4o-mini on a LLaVA-1.5 subset, with 30 clean outputs and 30 XSPA outputs per task. The paper does not report agreement between the judge and human ratings.
• The matched-budget comparison is saliency Top-k, with three optimization seeds. The joint targeted analysis uses seed 3407, the captioning model is written as BLIP-2 FlanT5-XL, the VQA model as LLaVA-1.6 Mistral-7B, with 214 jointly qualified images; the strictly targeted VQA plan is 380 questions on 224 images.

XSPA is an attack that restricts adversarial perturbations to a fixed X-shaped sparse line and transfers from a CLIP-style surrogate to downstream VLMs.

• Problem: The authors investigate whether classification, captioning, and VQA still exhibit cascading semantic failures in the shared visual–textual space when support is very small and geometry is fixed. The setting is white-box surrogate / black-box downstream, and downstream evaluations do not adapt the perturbation.
• Method: Two diagonal line segments through a specified center form the mask; at default 384×384, this is 1,532 pixels, ηM≈1.04%. The objective includes classification margin or cross-entropy, target text attraction, source text suppression, and magnitude and along-line smoothness. Updates use input diversity and momentum, and project back to the mask and infinity-norm feasible set. Main experiments run 200 iterations. The paper does not specify the surrogate name, step size, or ε for this run.
• Classification: On 300 COCO images, in Table 1, XSPA reduces OpenCLIP ViT-B/16 Top-1 from 97% to 30% (↓67), and OpenAI CLIP ViT-L/14 from 93% to 40.67% (↓52.33). Across four encoders, the three baselines in the table show smaller accuracy drops; among attack methods, XSPA has the highest GPT-4o naturalness.
• Open-ended tasks: In Table 2, OpenFlamingo's GPT-4 description consistency drops from 70.20% to 11.60% (↓58.60), and XSPA is lowest across all six models. In Table 3, the largest VQA drop is BLIP-2 (FlanT5XL ViT-L) at 44.25 points; on LLaVA-1.6 and InstructBLIP, ITA correctness is lower.
• Appendix boundaries: With the same 1,532 pixels, on the surrogate OpenCLIP ViT-B/16, saliency Top-k targeted ASR is 94.44±0.38%, while XSPA is 83.44±0.77%. On 214 jointly qualified images with seed 3407, targeted joint success across three tasks is 0/214. The authors claim this does not support that three-task targeted failures are correlated, nor that attack success is uniformly imperceptible; description review differences are treated as secondary evidence. Human evaluation was not completed.
• Authors' conclusion: The authors claim that with fixed geometric priors and very low budget, VLMs still exhibit cross-task semantic interference, and they argue this motivates future structure-aware defenses and robustness evaluations. They also position XSPA as a controlled diagnostic, not the strongest support selector under the same budget.