SmoothShift: Disrupting Visual Adversarial Attacks against LVLMs at Test Time
SmoothShift: Disrupting Visual Adversarial Attacks on LVLMs at Test Time
The authors propose a training-free test-time defense, SmoothShift (local smoothing plus spatial transformation), achieving the highest defense scores under APGD attack on LLaVA-1.5-7B, Qwen3.5-4B and InternVL3.5-4B.
L∞ image-space attack, modifying only the visual input, with the text query and model parameters fixed. In the main evaluation, the attacker has full white-box access to the undefended LVLM and has ground-truth labels; adversarial examples are generated on the base model, and the defense is applied only at inference. A defense-aware mixed-gradient BPDA with EOT is also tested.
- LVLMs are sensitive to visually imperceptible adversarial image perturbations; the authors argue that existing training-based defenses are costly and that test-time defenses insufficiently exploit the attack mechanism, and it remains unclear how small perturbations cause large errors.
- The authors first analyze the optimization and propagation of the attack, and propose the training-free test-time defense SmoothShift: sequentially applying local smoothing (bilateral/guided/local similarity averaging) and spatial transformation (cyclic shift/rotation/scaling) to the input image, disrupting fine-grained perturbations and their alignment with the patch grid.
- Across three LVLMs, four datasets, eight attacks, and multiple perturbation budgets, SmoothShift's defense scores are highest in every column of Table I–IV, with a small drop in clean performance; under mixed-gradient BPDA+EOT, accuracy decreases but remains higher than the undefended baseline (Figure 7).
- The paper does not include a Limitations section specifically discussing limitations. The experiments cover three LVLMs, two robust CLIPs, four LVLM datasets, and eight classification datasets; adversarial evaluation samples 500 images per dataset, BPDA evaluation is limited to TextVQA, and the paper does not report the runtime cost of the defense.
- Evaluated models
- LLaVA-1.5-7BQwen3.5-4BInternVL3.5-4BTeCoAFARE
- Benchmarks
- COCOFlickr30kTextVQAVQAv2Caltech101Oxford-IIIT PetsUCF101Flowers102DTDEuroSATStanford CarsFGVC Aircraft
- Metrics
- CIDErVQA accuracyzero-shot classification accuracy
The researchers propose SmoothShift, a training-free test-time defense method that weakens visual adversarial attacks against LVLMs while largely preserving clean-sample performance. The method combines local smoothing, which weakens fine-grained perturbation patterns, with spatial transformations, which disrupt the alignment between perturbations and the visual encoder's patch grid, requiring no model updates, auxiliary networks, or extra inference. The paper's analysis finds that representation differences caused by subtle input perturbations are amplified layer by layer through the visual encoder and persist into the language model, and that disrupting the coupling between input patterns within patches and the model's computation can significantly weaken attacks. Experiments cover three LVLM families on four datasets and a robustly fine-tuned CLIP model on eight classification datasets, involving eight attack methods and multiple perturbation budgets, and show improved robustness; robustness gains also hold under defense-aware attacks using hybrid-gradient BPDA plus EOT.
Paper analysis
What problem does this paper attempt to solve?
Addressing the problem of defending against visual adversarial perturbations in LVLMs, the authors propose the training-free test-time defense SmoothShift.
Core question: how to defend against L∞ adversarial perturbations on LVLM visual inputs without retraining the model.
- Problem setting: The authors state that small, visually imperceptible perturbations can substantially alter visual representations, damage cross-modal alignment, and induce incorrect answers, affecting LVLM trustworthiness (Introduction).
- Shortcomings of existing methods: The authors consider training-based defenses costly to train and potentially harmful to generalization; test-time methods are flexible to deploy, but how optimized visual perturbations induce and sustain adversarial effects remains insufficiently understood.
- Core observations: The authors report that attacks strengthen with more iterations and finer step sizes (Figure 2(a)); the difference between clean and adversarial representations is amplified layer by layer in the vision encoder and persists into the language model (Figure 1(b)(c)); small cyclic shifts that are not integer multiples of the patch width weaken the attack, whereas shifts of 1–2 patch widths provide weaker mitigation (Figure 2(d)).
- Paper hypothesis: Finely optimized perturbations exploit the precise coupling between the perturbed input and the model computation, and this coupling is sensitive to the pixel arrangement within patches.
- Paper approach: Proposes SmoothShift—an input-level test-time defense combining local smoothing and spatial transformation, requiring no model updates, auxiliary networks, or additional LVLM forward inference.
- Threat model (Section V-A):
- Goal: L∞ image-space attack, modifying only the visual input.
- Fixed items: text query and model parameters unchanged.
- Attacker knowledge: in the main evaluation, full white-box access to the undefended LVLM, and ground-truth labels.
- Implementation: adversarial examples are generated on the base model, and the defense is applied only at inference; a defense-aware mixed-gradient BPDA with EOT is also tested.
What related research exists?
Related work is divided into three groups: LVLM visual attacks, training-based defenses, and test-time defenses; this paper compares with 13 baselines on four datasets.
LVLM visual adversarial attacks
- Schlarmann & Hein [5], Cui et al. [6]—early work confirming that imperceptible image perturbations can damage classification, captioning, and VQA of multimodal models.
- Representation-oriented attacks [7–9] (including VEAttack [9])—improve cross-model and cross-prompt transfer by attacking visual representations.
- Cross-prompt/query optimization [10, 11] (e.g., QAVA [11]) and cross-modal semantic chain attacks [12]—improve prompt generalization and transferability. The authors state that this line of research mainly focuses on stronger attack objectives and has insufficient understanding of the iterative optimization mechanism.
Training-based defenses
- TeCoA [13]—text-guided contrastive adversarial training.
- FARE [14] and [24]—adversarial fine-tuning that preserves pretrained representations; [15] performs adversarial vision-language pretraining and instruction tuning.
- AdPO [17]—enhances LVLM robustness with preference optimization. The authors consider such methods costly to train and potentially harmful to model generalization.
Test-time defenses
- Input transformation: quantization, JPEG compression, TVM [25], random transformation BaRT [26]; ET3 [27] optimizes the input at inference using an energy function derived from model logits.
- Purification: diffusion-model purification [20, 28] and the LVLM-specific F3 [19].
- Other interventions: prompt adaptation [21, 29], adversarial detection and adaptation [22], spectral correction CSR [30], part-aware supervision DPS [18], test-time counterattack TTC [31].
Baselines and benchmarks: training-based baselines are R-LLaVA-G [6] and AdPO [17]; test-time baselines are Quantize, TVM, Freq, BaRT, DPS, F3-v3, TTC, CSR, DBD, AOM, ET3 (Section V-A-e). Evaluation benchmarks are COCO, Flickr30k (CIDEr) and TextVQA, VQAv2 (VQA accuracy), plus eight classification datasets for robust CLIP.
Positioning of this paper: The authors state that SmoothShift is an input-level test-time defense with a different design principle—it does not estimate a clean counterpart, perform detection, or apply downstream correction, but directly disrupts the properties that fine-grained adversarial optimization relies on.
How does the paper address this problem?
SmoothShift sequentially applies randomly sampled local smoothing and spatial transformation to the input image, disrupting the perturbation structure and its alignment with the patch grid.
Overall idea: first use empirical observations to identify the two properties the attack relies on (fine-grained optimization, intra-patch pixel alignment), then use the smoothing-plus-spatial-transformation combination of SmoothShift to destroy these two properties before inference.
Problem formulation
- Let the LVLM be F, with input image u and query q generating an answer; the attack maximizes the attack objective ℓ within the constraint |u−x|∞ ≤ ϵ (L, the answer loss, for untargeted; −L for targeted), i.e., a defense-oblivious attack (Eqs. (1)(2)).
- The defense applies a transformation Φθ to the input before inference, with clean and adversarial inputs going through the same pipeline (Eq. (3)).
Empirical observations (motivation)
- Optimization sensitivity: Figure 2(a) shows that the attack becomes stronger (task score lower) as the number of APGD iterations K increases and the minimum step size ηmin decreases; the authors therefore argue that the attack relies on fine-grained optimization, motivating local smoothing.
- Spatial alignment sensitivity: Figure 2(b)(d) shows that a 3-pixel cyclic translation can restore the correct answer, whereas a 14-pixel translation (equal to the patch width of the LLaVA-1.5 vision encoder) gives weaker mitigation; Figure 2(d) shows local minima of adversarial accuracy at 1 and 2 patch widths on three models, while clean accuracy remains relatively stable. The authors therefore hypothesize that the perturbation exploits the coupling between intra-patch input patterns and the model's computation.
SmoothShift pipeline
- The composition is Φθ(u) = Tβ(Sα(u)), i.e., smoothing S (parameter α) first, then spatial transformation T (parameter β), with θ = (S, α, T, β) (Eq. (4)).
- Local smoothing, one of three: bilateral filtering (Eq. (5), dual Gaussian weighting over space and color), guided filtering (Eq. (6), local linear model, εg controls smoothing strength), local similarity averaging (Eq. (7), averages only neighborhood values whose difference from the center value does not exceed threshold τ).
- Spatial transformation, one of three: cyclic translation (Eq. (8), integer-pixel translation with wraparound, no interpolation), rotation (Eq. (9), rotation about the center, bilinear interpolation), scaling (Eq. (10), center scaling, ρ<1 缩小、ρ>1 magnification).
Inference-time configuration
- For each image, one smoothing operator and one spatial transformation and their parameters are uniformly sampled from the candidate operator pool and applied in order; the clean and adversarial versions of the same image use the same configuration across all attack rounds (Section V-A-c).
- The candidate parameter pools and baseline configurations are in the Supplementary material; the main text does not list specific values.
Success criterion
- Defense effectiveness is measured by task metrics: CIDEr for captioning, standard VQA accuracy for VQA, and zero-shot accuracy for classification; the same metric reports clean, adversarial, and defended performance simultaneously, higher being better.
What experiments did the paper conduct?
Across three LVLMs plus two robust CLIPs, eight attacks, and multiple budgets, SmoothShift has the highest defense score in every table column.
Experimental Setup
- Models evaluated: LLaVA-1.5-7B, Qwen3.5-4B, InternVL3.5-4B; plus two robustly fine-tuned CLIP ViT-B/32 (TeCoA, FARE, trained with perturbation budget 4/255).
- Datasets: captioning COCO, Flickr30k (CIDEr); VQA uses TextVQA, VQAv2 (VQA accuracy); classification uses Caltech101, Oxford-IIIT Pets, UCF101, Flowers102, DTD, EuroSAT, Stanford Cars, FGVC Aircraft.
- Attacks: main attack APGD (ϵ = 2/255, 4/255; Table IV adds 16/255, 32/255); additionally FGSM, PGD, C&W, EAD, SPSA, VEAttack, Verbose, eight defense-oblivious attacks in total; defense-aware evaluation uses mixed-gradient BPDA with EOT.
- Baselines: training-based R-LLaVA-G, AdPO; test-time Quantize, TVM, Freq, BaRT, DPS, F3-v3, TTC, CSR, DBD, AOM, ET3, 13 comparison methods in total.
- Sample sizes and rounds: clean uses the full evaluation set; adversarial evaluation randomly samples 500 images per dataset, with at most 5 rounds of APGD per image and 100 updates per round, taking the lowest per-image score with early stopping; CLIP classification uses the full test set, PGD-100 (4/255).
- Evaluation method: all are automatic metrics; the paper does not use LLM evaluation or human evaluation.
Main Results
According to Table I (LLaVA-1.5-7B, APGD):
Swipe horizontally to read the table
Method COCO Clean COCO 4/255 TextVQA Clean TextVQA 4/255 VQAv2 4/255 Average 4/255 No defense 115.13 14.95 48.61 10.92 30.34 16.15 R-LLaVA-G 106.89 76.54 25.26 14.94 50.02 45.10 TVM 114.48 81.70 46.32 33.60 61.74 55.84 Freq 115.35 91.84 35.69 28.96 64.92 60.68 AOM 111.98 88.42 46.89 34.14 67.14 61.47 SmoothShift 114.25 101.94 45.78 42.42 69.78 70.08 - The authors state that SmoothShift has the highest defense score on every dataset-budget combination at ϵ = 2/255 and 4/255; at 4/255 it exceeds the strongest comparison in each column by 10.10 CIDEr (COCO) and 8.28 percentage points (TextVQA).
- Clean cost: COCO CIDEr drops by 0.88, Flickr30k by 2.31, TextVQA accuracy drops by 2.83 percentage points, VQAv2 by 0.69.
- Cross-model (Table II, III, 4/255): on Qwen3.5-4B, TextVQA 56.16%, VQAv2 73.56%, 5.82 and 1.98 percentage points higher than the strongest comparison; on InternVL3.5-4B, 67.20% and 71.72%, 14.32 and 5.18 percentage points higher; clean drops are 4.22/1.37 points for Qwen and 1.58/1.00 points for InternVL.
Cross-Attack and Robust CLIP
- Eight attacks: Figure 4 reports average CIDEr on COCO and Flickr30k for LLaVA-1.5-7B; the authors state that SmoothShift is highest among the 13 methods under all eight attacks FGSM, PGD, APGD, C&W, EAD-EN, Verbose, VEAttack, SPSA (the figure shows only faceted bar charts with truncated y-axes; specific values are in the Supplementary material).
- Robust CLIP: Figure 3 reports adversarial accuracy of TeCoA and FARE under PGD-100 (4/255); the authors state that SmoothShift is highest on all eight datasets for both models; in the figure each dataset axis is scaled independently and only the axis maximum is labeled (e.g., Caltech101 65.64, Pets 41.48, Cars 12.52, UCF101 17.02, DTD 17.20); per-method values are in the Supplementary material.
- Larger budgets: In Table IV, SmoothShift is highest on all four datasets at 16/255 and 32/255, e.g., COCO 16/255 is 83.47 (no defense 2.66), 32/255 is 61.88 (no defense 1.70); the authors acknowledge that performance decreases as the budget increases.
Ablation and Defense-Aware Evaluation
- Component ablation (Table V, LLaVA-1.5-7B): spatial transformation only (w/o smoothing) is stronger than smoothing only (w/o spatial); at 4/255 the combination raises COCO CIDEr from 93.95 to 101.94 and TextVQA from 40.38% to 42.42%, with a small drop on clean.
- Cross-configuration transfer (Figure 6): the attack is strongest when the attack shift amount matches the inference shift amount; changing the inference configuration generally weakens the attack; the authors state that cross-shift transfer is limited, which may benefit practical deployment.
- BPDA+EOT (Figure 7, TextVQA, 500 samples): exact gradients are retained for differentiable operations, and approximate gradients are used for the hard selection in local similarity averaging; with EOT-20, accuracy drops to 15.22% (2/255) and 12.34% (4/255), and the claim that these remain above the undefended baselines of 14.88% and 10.92% holds only for 4/255—at 2/255, 15.22% is above 14.88%, and at 4/255, 12.34% is above 10.92%; the authors state "higher than the undefended baseline at both budgets." Retaining exact gradients can partially bypass the defense, and more EOT samples generally further reduce accuracy.
Other Ablations and Analysis
- Smoothing strength (Figure 5(a)-(c)): stronger smoothing generally improves adversarial performance; clean COCO is relatively stable, clean TextVQA drops more; when the local similarity threshold τ is larger, COCO gains saturate and TextVQA gains diminish.
- Rotation and scaling (Figure 5(e)(f)): both improve robustness relative to the identity setting; stronger transformations do not consistently yield larger gains; under aggressive settings, scaling reduces clean TextVQA performance.
- Cyclic shift (Figure 5(d)): small shifts improve adversarial performance while clean is largely maintained; gains decrease at 14 and 28 pixels (1 and 2 patch widths).
- Negative result: Table IV shows that at 32/255 all methods score low (SmoothShift COCO 61.88, TextVQA 24.18%); the authors only state that performance decreases with the budget and give no further explanation.
What are some points that can be further explored?
The paper does not include a limitations section; the experiments cover three LVLMs, two robust CLIPs, eight attacks, and BPDA adaptive evaluation.
Limitations and future directions noted by the authors
- The paper does not specifically discuss limitations: the main text has no Limitations or Future Work subsection, and the Conclusion (Section VI) only summarizes findings and methods without listing shortcomings or future plans.
- Author statements related to the boundaries of the evidence: in BPDA evaluation, "retaining exact gradients can partially bypass SmoothShift" (Section V-D-b); cross-configuration transfer is "limited," which the authors say may be beneficial for practical defense (Section V-D-a); under large budgets, "performance decreases as the budget increases" (Section V-B-e). These are descriptions of results rather than limitations declared by the authors.
Scope of experimental coverage
- The models tested are LLaVA-1.5-7B, Qwen3.5-4B, and InternVL3.5-4B, three LVLMs in total, plus two robustly fine-tuned CLIP ViT-B/32 models, TeCoA and FARE (Section V-A).
- LVLM evaluation covers four datasets: COCO, Flickr30k, TextVQA, and VQAv2; CLIP evaluation covers eight classification datasets; adversarial evaluation uses 500 images per dataset, up to 5 rounds of APGD, and 100 updates per round (Section V-A-c).
- Attacks cover eight defense-oblivious attacks and the defense-aware mixed-gradient BPDA with EOT; BPDA evaluation is conducted only on TextVQA, with EOT sample counts of 1, 5, 10, 15, and 20 (Figure 7).
- Defense-oblivious evaluation uses the same data, prompts, generation settings, adversarial inputs, and metrics (Section V-A-e); the paper does not report the number of repeated runs, variance, or statistical tests.
- The paper does not report measured runtime or additional computational overhead for SmoothShift (it only states that no additional LVLM forward inference is required); the candidate parameter pool values for smoothing and transformation are in the Supplementary material and are not listed in the main text.
Summarize the main content of the paper
SmoothShift uses smoothing plus spatial transformation to disrupt adversarial perturbation structure at test time, achieving the highest defense scores on three LVLMs with small clean loss.
An LVLM visual adversarial defense paper, centered on a training-free test-time input transformation method and the mechanistic observations behind it.
- Problem: The authors state that small visual perturbations can substantially damage LVLM captioning and VQA performance, while training-based defenses are costly and test-time defenses insufficiently exploit the attack mechanism.
- Observations: Attacks strengthen with more iterations and finer step sizes (Figure 2(a)); the clean–adversarial representation difference is amplified layer by layer and persists into the language model (Figure 1(b)(c)); cyclic shifts that are not integer multiples of the patch width weaken the attack, while mitigation weakens at multiples of the patch width (Figure 2(d)). The authors accordingly argue that the attack depends on fine-grained perturbation patterns and their coupling with computation within the patch grid.
- Method: SmoothShift randomly samples and sequentially applies a local smoothing operator (bilateral/guided/local similarity averaging) and a spatial transformation (cyclic shift/rotation/scaling) to each image, requiring no model updates, auxiliary networks, or additional LVLM inference.
- Main results: On LLaVA-1.5-7B under APGD 4/255, COCO CIDEr 101.94, TextVQA 42.42% (no defense 14.95, 10.92%, Table I); on Qwen3.5-4B and InternVL3.5-4B, VQAv2 73.56% and 71.72% respectively (Table II, III); still highest under the large budgets 16/255 and 32/255 (Table IV); the authors state it is highest on eight attacks and two robust CLIPs (Figure 3, 4).
- Adaptive attack: mixed-gradient BPDA+EOT-20 pushes TextVQA accuracy down to 15.22% (2/255) and 12.34% (4/255), still above the undefended 14.88% and 10.92% (Figure 7).
- Authors' conclusion: Adversarial effectiveness depends on fine-grained optimization and pixel alignment within patches; disrupting these two can improve robustness while largely maintaining clean performance.
Related papers
- Semantic-aware Adversarial Fine-tuning for CLIPDefensesarXiv