Unveiling Physical-World Semantic Vulnerabilities: Universal Adversarial Patches for Infrared Vision-Language Models

Can one reusable physical patch cause stable open-ended semantic deviations in an infrared vision-language model (IR-VLM).

• Use scenario: The authors state that IR-VLM is used for low-visibility semantic perception; after vision and text are aligned, failures expand from wrong labels to generated captions and answers.
• Existing shortcomings: Existing adversarial patches mainly target RGB or closed-set infrared detectors, and do not directly address open-ended IR-VLM. The authors state that it is not yet clear whether one deployable object can simultaneously rewrite the visual evidence shared by classification, captioning, and VQA.
• Core hypothesis: Physical attacks usually cannot make a separate patch for every frame, prompt, or downstream task. Infrared texture is weaker and statistically dominated by noise, and visible-light high-frequency patterns may disappear after thermal capture or be difficult to fabricate. The authors believe an effective attack should be shared, low-frequency, confined to the target region, and act on the visual representation before task decoding.
• Approach of this paper: Proposes the Universal Curved-Grid Patch (UCGP), making the patch a deployable low-frequency curved grid, and optimizing one shared patch with a representation-layer objective. The authors state that to their knowledge, this is the first systematic study of universal physical adversarial patches for IR-VLM.
• Threat model:
  • Goal: Learn a set of shared parameters that, after being pasted onto the target region of each image, are reused across samples, causing classification, captioning, and VQA to deviate from clean semantics, rather than only flipping closed-set labels.
  • Knowledge: The three types of responses can be obtained through the forward interface, as well as the encoder normalized features z=f(x); no gradients, parameters, or training data.
  • Authors' naming: representation-visible audit setting, not output-only black-box; weaker than white-box, stronger than commercial API-style output-only access. Strict output-only API is left for future work.
  • Control and target: The attacker controls the patch on the target region. During optimization, EOT samples imaging variations, and TPS models local non-rigid deformation. The victim system is an IR-VLM on infrared images. Digital evaluation uses rendered samples, and physical evaluation uses real-captured infrared video.

The introduction and Section 2 divide related work into four groups and place this paper at the intersection of infrared physical security and multimodal semantic robustness.

Infrared physical attacks

• Deployable perturbations: EOT by Athalye et al. (2018), physical-world attacks by Eykholt et al. (2018), and patches by Brown et al. (2017) and Chen et al. (2018), discussing that perturbations may remain effective after capture, viewpoint, and scale changes.
• Infrared detectors: early work hid thermal signatures with adversarial clothing or insulating materials (Zhu et al., 2022; Wei et al., 2023); later work used blocks, curves, and grids to improve deployability and multi-view performance (Hu et al., 2023; Hu and Shi, 2024; AdvGrid, 2025).
• Closest neighbors named in the paper: Wei et al. (2023) used shape-optimized patches to attack visible–infrared detector pairs simultaneously. The paper says these works provide deployment priors, but their main goal is still closed-set detection or recognition.

IR-VLMs and benchmarks

• Model adaptation: Infrared-LLaVA (Jiang et al., 2024) and IRGPT (Cao et al., 2025) study infrared adaptation and image-text data construction.
• Understanding benchmarks: RGB-Th-Bench (Moshtaghi et al., 2025) and IF-Bench (Zhang et al., 2025) evaluate paired understanding and open-ended infrared reasoning.
• The paper's distinction: these studies ask whether models can understand infrared content; this paper asks whether semantics remain faithful under physical perturbations. The authors say infrared images encode shape, thermal distribution, and low-texture cues, and local thermal patches may affect global category semantics.

Multimodal and CLIP/VLM attacks

• Visible-light side: visual jailbreaks by Qi et al. (2024), VLAttack by Yin et al. (2023), transfer attacks by Xie et al. (2025), as well as physical-world LVLM patches, large-scale targeted attacks, and highly transferable attacks on CLIP representations (Liu et al., 2026; Zhang et al., 2025; Huang et al., 2025).
• The paper's distinction: the authors say most of these optimize pixel or generator perturbations in visible-light digital settings, or rely on task-specific image-text objectives, and do not require fabricating stable infrared objects.

Gradient-free physical optimization

• Search: differential evolution (Storn and Price, 1997), covariance matrix adaptation (Hansen and Ostermeier, 2001), and score-based black-box attacks with limited queries (Ilyas et al., 2018; Andriushchenko and Croce, 2020).
• Transformation: EOT sampling captures conditions, TPS models non-rigid deformation (Athalye et al., 2018; Bookstein, 1989). The paper says that if the search space is unconstrained, optimizers may find high-frequency patterns that are digitally effective but hard to fabricate or attach.

Baselines and data: experimental baselines are HCB, AdvGrid, and QR Code, adapted to a unified region, loss, and evaluation protocol. The main experiments use Infrared-COCO; transfer uses LSOTB-TIR, LLVIP, M3FD, and FLIR ADAS v1.3.

The authors position UCGP as a reusable thermal patch: it attacks the clean category manifold in the visual representation and treats semantic vulnerability as a system property, evaluated across classification, description, VQA, transfer, and physical deployment.

UCGP does not change labels or prompts, but instead disrupts the clean class manifold in the visual representation space, and then observes downstream deviations from classification, captioning, and VQA. The same parameters are reused across images, and the main comparison also does not tune a separate patch for each downstream task.

Problem Setting

• Rendering and pasting: Given an infrared image set and target regions, the shared parameters are applied to each image via an operator. x′_i=R(x_i,θ,b_i) denotes the same patch rendered and pasted onto the target region of the i-th image.
• Interface: Optimization uses the normalized features of the image encoder; the text side is used only for downstream evaluation. The paper states that the continuous objective is formed by the visual representation.
• Two types of augmentation: EOT samples the capture conditions and mild attachment variations of the final image; TPS warps the rendered patch before pasting, to simulate local non-rigid changes such as clothing folds. Both are physical augmentations at evaluation time, not separate attack objectives.

Curved-Grid Parameterization

• Low-dimensional structure: On a G×G grid, the parameters include cell gating and edge deformation. Gating determines whether a cell boundary participates; adjacent activated cells share a boundary and it is drawn only once. Selected edges use quadratic Bézier curves, deformation moves control points along the normal, and the maximum displacement is δ times the cell edge length.
• Reasons given by the authors: Pixel space has high degrees of freedom, and gradient-free search would exploit isolated high-frequency pixels, which are unstable after fabrication, attachment, downsampling, and infrared capture. The curved grid constrains the search to low-frequency connected line segments.
• Topology and default values: In Fig. 3, when δ<0.50 the grid is continuous and smooth; when δ≥0.50 paths intersect or self-intersect, breaking contour continuity. The main experiments use G=5, δ=0.40, edge length of 1/4 of the target height, line width of 0.20 of the cell edge length, and default black.

Representation-Layer Objective

The clean reference set takes samples that the model stably recognizes as the target class under clean input; the main experiment class is person. After feature normalization, the mean is computed, then PCA/SVD is used to take the top k principal directions. Falling within this subspace is regarded as normal intra-class variation, and orthogonal directions are regarded as leaving that class structure. The specific value of k is not given in the paper.

L=L_subspace+λ_topoL_topo+λ_budgetL_budget adds subspace deviation, topology disruption, and local appearance regularization on the same shared parameters. In the main experiments λtopo=0.12, λbudget=0.03.

• Subspace term: The orthogonal deviation of adversarial features relative to the principal subspace is batch-averaged; when minimizing, a negative sign is added, thereby increasing the deviation.
• Topology term: Graph-KL is used on the row-normalized Gaussian adjacency of clean and adversarial features. The authors state that minimizing this loss is equivalent to increasing the KL of the neighborhood distributions, emphasizing structural misalignment, not just mean shift. The scale is estimated by the median of pairwise squared distances of clean features.
• Budget term: Thermal statistics, boundary gradients, and area. The thermal term penalizes the difference in mean and variance between the patch and the surrounding background ring; the boundary term suppresses excessively high gradients on the activated boundary; the area term constrains effective coverage. The authors state that these terms do not pursue infrared invisibility.

Optimization Procedure

• Why MetaDE is used: Soft rendering, TPS, pasting, EOT, and forward inference make the objective non-differentiable with respect to the patch parameters, and each evaluation is expensive and noisy. The authors use MetaDE for gradient-free search.
• Steps: First compute clean statistics and initialize the population; each generation samples an evaluation subset, mutation and crossover produce trial patches, for each sample in the subset one EOT transform and one TPS warp are sampled, Monte Carlo fitness is used for comparison and better individuals are accepted. When stalled, part of the population is refreshed. Algorithm 1 gives the pseudocode.
• Budget: Population 50, generations 100, and a fixed query budget is written; the specific number of queries and the number of Monte Carlo samples K are not given in the paper. The hardware is eight RTX 4090s; unless otherwise stated, each optimization or evaluation process uses one GPU. Throughout, only one patch parameter is kept, and no further optimization is done per image.

Success Criteria

• Classification: In the 80-class MS-COCO label space, a person sample is counted as successful if its top-1 is not person. ASR is this proportion. Conditioned on successful samples, ∆sprom and Madv are also reported: the former is the increase of the winning competing class relative to its clean logit, and the latter is the final margin of that competing class relative to the original class. The logits come from the pre-softmax scores of scaled cosine similarity.
• Generation: LLM-as-a-judge is followed. Captioning normalizes 0–30 scores into a 0–100 Consistency Score; VQA averages answer correctness into a 0–100 Correctness Score. The downward arrows in the table are the absolute score drops from clean to adversarial. The judge model is GPT-5; the prompts are in the supplementary material, and the main text does not give the prompts.
• Naturalness: GPT-5's 5-point Nat. is only for perceptual inspection and does not enter optimization. The authors state that it should not be interpreted as infrared invisibility.

Experimental Setup

• Main data: Except for transfer, experiments are on Infrared-COCO. This set has about 17k infrared images synthesized from MS-COCO, and 12k question-answer pairs. 300 images are sampled from valid images with person class probability above 0.50, shared by classification, captioning, and VQA. For cross-class, a clean reference is rebuilt for each target class, and the same pipeline is rerun, without class-specific design changes.
• Transfer sets: LSOTB-TIR, LLVIP, M3FD, FLIR ADAS v1.3. The authors state that they correspond respectively to scale and motion in tracking, low-light visible–infrared, multispectral detection in complex backgrounds, and thermal driving.
• Models and training: Classification uses OpenAI CLIP ViT-L/14, OpenCLIP ViT-B/16, Meta-CLIP ViT-L/14, EVA-CLIP ViT-G/14. Generation interfaces are LLaVA-1.5 (7B), LLaVA-1.6 (13B), OpenFlamingo (3B), BLIP-2 FlanT5XL ViT-L (3.4B), BLIP-2 FlanT5XL (4.1B), InstructBLIP FlanT5XL (4.1B); the image encoder is OpenAI CLIP ViT-L/14 or EVA-CLIP ViT-G/14. Weights start from natural image pretraining, the backbone is frozen, and LoRA is used; classification uses InfoNCE, generation uses next-token, and training stops when validation loss no longer decreases. The paper states that unless otherwise stated, subsequent classification experiments use OpenAI CLIP ViT-L/14 as both surrogate and evaluation model.
• Baselines and metrics: HCB, AdvGrid, QR Code are adapted to the same region, loss, images, and downstream models. Classification looks at ASR; captioning looks at Consistency Score; VQA looks at Correctness Score; transfer additionally looks at ∆sprom, Madv; Table I additionally looks at Nat. Both generation and Nat. are judged by GPT-5. The paper does not report the number of repetitions, nor the agreement between the judge and humans.
• Physical collection: InfiRay XL19V2, 384×288, sensitivity <18 mK. The cold patch is kept at 24°C for 10 hours. The patch is attached to underwear and covered by outer clothing, and remains clear in infrared. Indoor 6.2–11.0 m, outdoor 7.8–12.6 m. Each scene video is about 80–90 seconds, uniformly sampled at 3 fps.

Main Results

Table I is classification ASR and Nat. The Nat. of clean input is all 3.26. The paper does not state whether these four rows each use that backbone as the surrogate when optimizing.

Swipe horizontally to read the table

Model	UCGP ASR	Highest baseline in the same table	UCGP Nat.
OpenAI CLIP ViT-L/14	94.33%	QR Code 22.67%	2.33 (↓0.93)
OpenCLIP ViT-B/16	61.67%	HCB 34.00%	2.30 (↓0.96)
Meta-CLIP ViT-L/14	34.33%	HCB 20.67%	2.25 (↓1.01)
EVA-CLIP ViT-G/14	48.00%	AdvGrid 27.67%	2.37 (↓0.89)

• The authors state that on the four backbones UCGP's ASR is highest or clearly leading, and the local appearance score remains comparable. In the table, Meta-CLIP ViT-L/14 has the lowest UCGP ASR, and the smallest gap to the strongest baseline; in that row UCGP's Nat. is 2.25, and AdvGrid's is 2.26.
• The authors state that backbone differences are uneven: OpenAI CLIP ViT-L/14 is the easiest to attack, Meta-CLIP ViT-L/14 is the hardest, and the other two are in between. The authors state that in Fig. 5 the same patch changes the predicted semantics of OpenAI CLIP and EVA-CLIP.
• The authors state that the three baselines can perturb local evidence, but the global person representation often remains; UCGP is optimized to leave the clean class manifold, and does not need to imitate some non-person object. Nat. is only for perceptual inspection.

Captioning and VQA

The authors state that UCGP has the largest drop in all six captioning and VQA settings. The table below lists only UCGP's absolute drop (points) relative to clean output; baseline scores are in Table II, III. The judge is GPT-5.

Swipe horizontally to read the table

Model	Image encoder	Captioning drop	VQA drop
LLaVA-1.5 (7B)	OpenAI CLIP ViT-L/14	17.24	16.87
LLaVA-1.6 (13B)	OpenAI CLIP ViT-L/14	15.24	16.15
OpenFlamingo (3B)	OpenAI CLIP ViT-L/14	19.40	19.53
BLIP-2 FlanT5XL ViT-L (3.4B)	OpenAI CLIP ViT-L/14	17.67	15.54
BLIP-2 FlanT5XL (4.1B)	EVA-CLIP ViT-G/14	18.40	18.49
InstructBLIP FlanT5XL (4.1B)	EVA-CLIP ViT-G/14	19.52	22.55

• The authors state that captioning drops are 15.24 to 19.52 points, and VQA drops are 15.54 to 22.55 points, and state that the patch affects the visual evidence shared by different language interfaces.
• In Fig. 6 the adversarial outputs remain fluent, but often change the subject category, action, or scene relations. The authors therefore state that this is semantic degradation, not just wording changes. The specific scores of the scoring examples in the figure are not fully preserved in plain text, and are not copied one by one.
• The main comparison does not tune a separate patch for each downstream task. The authors state that this is more conservative than task-specific attacks.

Physical, Transfer, and Cross-Class

• Physical: The authors emphasize that the patch is not intended to hide the wearer in infrared. Fig. 7 includes 0°, +30°, −30°, as well as indoor 6.2–11.0 m and outdoor 7.8–12.6 m. Fig. 8's scene-average ASR (AASR) is 70.76% indoor and 73.96% outdoor. The authors believe the slightly higher outdoor value may be partly because the thermal background is more complex; the farthest outdoor still has a drop, because long distance reduces the patch's effective resolution. This paragraph does not separately write a model name; the default classification model of Section 4 is OpenAI CLIP ViT-L/14.
• Cross-dataset: Table IV's AVG includes the diagonal where source and target are the same. The authors report that OpenAI CLIP's average ASR over five sources is 95.37%–97.24%; OpenCLIP is 21.75%–29.69%; Meta-CLIP is 22.11%–57.17%; EVA-CLIP is 21.35%–64.58%. The authors believe the differences are consistent with backbone pretraining and infrared domain representation geometry, rather than uniform transfer across encoders. Cells not explained one by one in the main text are not copied separately.
• Cross-model and cross-class: In Table V, the same-backbone diagonal is generally the strongest: OpenAI CLIP source 94.33%–100.00%, EVA-CLIP source 26.67%–80.45%. OpenAI surrogate's average victim ASR is 31.24%–50.93%, Meta-CLIP source is 12.74%–28.01%, and the averages all include the diagonal. The authors state that off-diagonal is weaker but still observable. Table VI has N=300 per class: person 94.33%, dog 98.67%, car 61.00%, bicycle 56.00%. The authors believe the latter two classes are lower possibly because the objects are more rigid, longer, or more extended, and a fixed-size local patch covers less discriminative evidence. Table VI does not separately write a classifier name.

Defense Evaluation

Table VIII's Drop is the absolute ASR drop relative to no defense. The table does not write a model name; the no-defense ASR is 94.33%. AT only fine-tunes LoRA, with the backbone frozen; adversarial samples come from random transformed patch renderings of the same EOT/TPS family.

Swipe horizontally to read the table

Setting	Defender information	ASR	Drop
No defense	None	94.33%	—
DW-1	Position-blind image inpainting	45.33%	49.00
DW-2	With patch position prior	24.67%	69.66
AT-1	Against the original patch	15.33%	79.00
AT-2	Adaptive re-optimization against the hardened model	78.00%	16.33

• The authors state that existing defenses can mitigate the attack, but none is a complete defense. Inpainting with a position prior is stronger than blind inpainting, but still incomplete.
• AT-1 is the lowest. The authors state that this is because the hardened model has directly seen the original universal patch distribution. AT-2 rises to 78.00%. The authors state that robustness to a fixed patch does not eliminate the underlying semantic vulnerability, and defense evaluation should include both known patches and adaptive re-optimization.

Other Ablations and Analysis

• Table VII changes only the named components. On OpenAI CLIP ViT-L/14: full model 94.33%, pixel optimization 30.33%, straight edges 7.00%, random topology 5.33%, replacing MetaDE with standard DE 76.00%. The lowest value for random topology is 2.00% on Meta-CLIP ViT-L/14; the corresponding cells for the other three backbones are in Table VII.
• Main text on Fig. 9: on OpenAI CLIP ViT-L/14, ASR is 68.33% after removing the topology loss, and 35.00% after removing the subspace loss. The authors state that the order is the same for the other backbones. The remaining bars in the figure cannot be aligned in plain text, and are not copied.
• Fig. 12: The authors state that the four backbones' ASR roughly converges after 300 optimized images; the figure labels Stable regime (Δmax ≤ 2.73 pp). The main experiments therefore use 300 images.
• Fig. 3: δ<0.50 grid is continuous, δ≥0.50 shows intersection or self-intersection. Fig. 10 scans population, line width, grayscale, and λtopo; Fig. 11 scans patch area ratio and grid number, and the gold box marks the main experiment default configuration. The main text does not give the ASR of these curves point by point, and does not copy figure numbers that may be misaligned.
• The authors state that straight edges and random topology reduce ASR to near failure, and the connected curve is the main structural factor; replacing with standard DE has a smaller drop than the structural ablation.
• Exceptions: In Table I, Meta-CLIP has the lowest ASR; in Table VI, bicycle and car are lower than person and dog; in Table V, several heterogeneous pairings are weaker, and the authors do not explain each pair individually; in Table VIII, AT-2 rises back to 78.00%.

Limitations and future directions noted by the authors

• Audit interface (Section 6): The authors list the representation-visible audit interface as an evidence boundary, i.e., optimization relies on readable visual representations.
• Physical factors (Section 6): The authors state that EOT/TPS does not fully cover material-dependent thermal response, ambient temperature variation, sensor nonlinearity, occlusion, or camera post-processing.
• Future evaluation (Section 6): The authors state that future evaluation should include jointly optimized patch placement, longer videos, multiple thermal cameras, different materials, a larger class set with confidence intervals, and adaptive defenses combining image inpainting with representation-layer training.
• Defense needs (Section 7): The authors state that stronger defenses are needed against cross-modal semantic manipulation in practice.

Experimental coverage

• Classification evaluates four CLIP backbones; description and VQA evaluate six generation configurations, with image encoders being OpenAI CLIP ViT-L/14 or EVA-CLIP ViT-G/14; generation scores and Nat. are reviewed by GPT-5 (Section 4, Tables I–III).
• The main task uses person from Infrared-COCO, with class probability above 0.50, 300 images; the classification label space is 80-class MS-COCO. The transfer set is LSOTB-TIR, LLVIP, M3FD, FLIR ADAS v1.3 (Section 4, Table IV).
• Cross-class is person, dog, bicycle, car, with N=300 per class (Table VI). Defenses are digital watermarks DW-1, DW-2 and adversarial training AT-1, AT-2 (Section 5, Table VIII).
• Physical deployment uses InfiRay XL19V2, indoors 6.2–11.0 m, outdoors 7.8–12.6 m, viewing angles include 0° and ±30°, each scene video is about 80–90 seconds, sampled at 3 fps (Section 4, Figs. 7–8).
• The paper does not report the specific number of fixed query budgets, Monte Carlo sample count K, number of repetitions, or agreement between GPT-5 review and humans. Section 3 leaves a strict output-only API for future work; the main experiments are conducted under the representation-visible interface.

UCGP is a universal physical adversarial patch for infrared vision-language models. The question is: can a patchable and cross-sample reusable object cause stable semantic bias in the infrared visual evidence shared by classification, description, and VQA.

The patch is constrained to a low-frequency curved mesh. Optimization does not change labels or prompts, but increases orthogonal deviation from the relative clean class subspace, widens the Graph-KL of the neighborhood graph, and regularizes thermal statistics, boundary, and area. MetaDE searches the same parameter set under EOT and TPS.

• Classification: On Infrared-COCO, person, 300 images, UCGP ASR for four CLIP backbones is 94.33%, 61.67%, 34.33%, and 48.00% (Table I), all higher than baselines in the same table.
• Generation: Under GPT-5 review, description consistency drops by 15.24–19.52 points, VQA accuracy drops by 15.54–22.55 points (Tables II, III).
• Physical and transfer: Indoor AASR 70.76%, outdoor 73.96% (Fig. 8). OpenAI CLIP cross-dataset average ASR is 95.37%–97.24% (Table IV, including diagonal).
• Defenses: In Table VIII, AT-1 ASR is 15.33%, and AT-2 after re-optimization against the hardened model is 78.00%.

The authors state that a shared patch can cause semantic degradation beyond closed-set label flipping, and also brings out boundaries under defenses and real-world conditions; and state that current infrared multimodal systems have robustness blind spots, requiring stronger cross-modal semantic defenses. The authors state that to their knowledge, this is the first systematic study of this setting.