TRAG: Text-Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
Textual Response-Aware Guidance for Zero-Shot Adversarial Robustness of Vision-Language Models
The authors propose TRAG, which uses the local responses of a pretrained text encoder to constrain visual representation drift during adversarial fine-tuning; the authors state that on CLIP ViT-B/32 it achieves 59.60%/35.09% average clean/robust accuracy across 16 datasets.
The defender performs adversarial fine-tuning on the VLM image encoder, and the attacker applies ℓ∞ perturbations in pixel space (budget ϵ, default 2/255); evaluation uses two white-box attacks from AutoAttack, APGD-CE and targeted APGD-DLR, each with 20 steps, plus the text attack BERT-Attack and the multimodal Sep-Attack and Co-Attack.
- VLM adversarial fine-tuning improves robustness at the cost of zero-shot generalization. The authors argue that existing pretrained-model guidance methods each have shortcomings: feature matching has no semantic emphasis, and prediction-distribution matching is limited by the candidate texts.
- TRAG uses known text as semantic anchors, computes the Jacobian of the local response with respect to a frozen text encoder and performs SVD, constructs anchor-level constraint matrices under a fixed budget, and then aggregates them weighted by pretrained image–anchor relevance to obtain an image-adaptive quadratic penalty on drift, which is used together with the task loss for adversarial fine-tuning.
- On 16-dataset zero-shot classification, Flickr30k/COCO retrieval, text and multimodal attacks, and LLaVA-1.5-7B transfer, the authors state that TRAG's clean and robust metrics are mostly highest; in the relation-preservation analysis, its adversarial KL is lowest and Retention highest.
- The paper does not specifically discuss limitations. Experimental coverage: image encoder fine-tuning (text encoder frozen), ℓ∞ budgets 1/255–4/255, three VLM architectures and LLaVA-1.5-7B transfer; the paper does not report statistical significance tests or adaptive attack evaluation.
- Evaluated models
- CLIP ViT-B/32CLIP ViT-B/16CLIP ViT-L/14RN101CoCa ViT-L/14SigLIP ViT-B/16LLaVA-1.5-7B
- Benchmarks
- ImageNetCaltech-101Caltech-256CIFAR-10CIFAR-100DTDEuroSATFGVC AircraftFlowers102Food-101ImageNet-RImageNet-SOxford-IIIT PetStanford CarsSTL-10SUN397Flickr30kCOCOTextVQAVQAv2CC3M
- Metrics
- clean accuracyrobust accuracyTR@1/TR@10IR@1/IR@10CIDErVQA accuracyRetention
The researchers propose a text-response-aware guidance method, TRAG, which uses the response of a pretrained text encoder to local changes in input embedding vectors to constrain visual representation drift, improving the zero-shot adversarial robustness of vision-language models. The method centers on semantic anchors, constructs anchor-level constraints based on the direction and magnitude of the response, and then aggregates them according to pretrained image-anchor relevance to form image-adaptive constraints. Experiments across architectures and tasks show that its clean and robust performance reaches SOTA, and that it helps preserve pretrained image-text relationships.
Paper analysis
What problem does this paper attempt to solve?
How to preserve the pretrained image-text alignment that supports zero-shot generalization while adversarially fine-tuning VLMs to improve robustness.
Core problem: adversarial fine-tuning of VLMs improves robustness but breaks pretrained image-text alignment, causing zero-shot generalization to degrade; the paper studies how to achieve a better balance between the two.
- Setting and importance: VLMs such as CLIP obtain zero-shot capability from large-scale image-text pretraining, supporting classification and retrieval and serving as the visual backbone of LVLMs; however, the authors point out that they remain vulnerable to adversarial examples, where small perturbations can cause incorrect predictions.
- Shortcomings of existing methods: The authors' analysis argues that two types of pretrained model guidance have complementary limitations—feature matching (FM) constrains all directions equally, with broad coverage but no semantic emphasis; predictive distribution matching (DM) has semantic emphasis but is limited by candidate texts and predicted probabilities (Figure 1, Sec 3.2).
- Core observation: Visual representation drift affects image-text relations differently along different directions; zero-shot generalization requires preserving diverse image-text relations on unseen tasks, so directional constraints that are both broad in coverage and semantically focused are needed.
- Proposed approach: The paper proposes TRAG (Textual Response-Aware Guidance), which uses the pretrained text encoder's response to local changes in the input embedding to construct anchor-level directional constraints, then aggregates them into image-adaptive constraints according to pretrained image–anchor relevance.
- Threat model:
- Attacker's goal: cause the model to make incorrect predictions by applying small perturbations to the input image.
- Knowledge and capability: evaluation uses white-box attacks (AutoAttack's APGD-CE and targeted APGD-DLR), with pixel-space ℓ∞ budget ϵ (default 2/255); there are also the text-side BERT-Attack and multimodal Sep-Attack, Co-Attack.
- Victim system: adversarially fine-tuned VLMs (CLIP, CoCa, SigLIP) and LLaVA-1.5-7B using them as visual encoders.
- Defender's setting: only the image encoder is fine-tuned, the text encoder is frozen, and the pretrained image encoder serves as a frozen reference.
What related research exists?
Related work falls into three categories: VLM adversarial robustness methods, pretrained-model-guided methods, and six adversarial fine-tuning methods as baselines.
VLM adversarial robustness
- TeCoA [30]: fine-tunes the image encoder with a text-guided contrastive objective to achieve zero-shot adversarial robustness, and is the basis for subsequent methods.
- Subsequent directions: increasing the spacing of class text representations [22], adjusting classification margins [27], aligning regions along adversarial paths [7], aligning image-text subspaces [6]; adversarial prompt tuning (text [21], visual [56]) and test-time defenses (prompt tuning [39, 44], input counterattack [47]).
Pretrained model guidance (most relevant to this paper)
- Feature level: FARE [37] matches adversarial visual features to the clean features of the pretrained model; QT-AFT [45] adds supervision from high-quality image annotations on top of feature matching.
- Prediction level: PMG-AFT [43] combines the adversarial task objective with predictive distribution matching to the pretrained model; AGFT [5] replaces hard labels with temperature-scaled soft alignment distributions from the pretrained model.
- Others: AdvFLYP [48] performs contrastive adversarial fine-tuning on web image-text pairs and combines two kinds of matching; TGA-ZSR [51] preserves the original model's text-guided spatial attention. The paper's analysis points out that FM-type and DM-type methods have complementary limitations in directional constraints (Sec 3.2), which is the starting point of TRAG.
Baselines and benchmarks
- Baseline methods: TeCoA, PMG-AFT, FARE, TGA-ZSR, QT-AFT, AGFT, with AdvFLYP additionally for retrieval experiments (Sec 5.1, 5.2).
- Benchmarks: ImageNet plus 15 zero-shot classification datasets, Flickr30k and COCO retrieval, TextVQA and VQAv2.
The authors position TRAG as a new form different from the above guidance methods: it constructs image-adaptive drift constraints from the local responses of the pretrained text encoder, rather than matching features or predictive distributions.
How does the paper address this problem?
TRAG derives anchor-level directional constraints from the local responses of a frozen text encoder, aggregates them by image–anchor relevance, and then performs adversarial fine-tuning together with the task loss.
Overall idea: treat the response of a pretrained text encoder to local changes in the input embedding as a directional reference for latent semantic change, construct a quadratic penalty on visual representation drift, and guide adversarial fine-tuning. Method name TRAG (Textual Response-Aware Guidance).
Problem setting and analysis framework
- Only the image encoder parameters θ are fine-tuned; the pretrained image encoder f0 is frozen as a reference, and the text encoder g is shared and fixed; drift is defined as ∆r(x, xadv) = rθ(xadv) − r0(x) (Eq. 5).
- Drift changes the image–text similarity via s(t)⊤∆r, and affects it differently along different directions, so guidance is viewed as a directional constraint on drift.
- Diagnosis: FM's constraint matrix is the identity matrix (all directions equal); DM near r0 is approximately 1/2·∆r⊤H0∆r, where H0 is weighted by predicted probability over candidate text-pair differences (Eqs. 7, 8). Figure 2 measures directional coverage by the eigenvalue participation ratio and task relevance by the normalized Frobenius inner product with the downstream class-text covariance; in the figure FM has the highest coverage and DM low coverage, and TRAG's similarity is higher than FM, DM, and their best fixed mixture.
Anchor-level constraint construction (Sec 4.1)
- Anchor set A = {aj} (not necessarily equal to the candidate texts); for anchor aj, take the token embedding ej, denote by Jj the Jacobian of the frozen text encoder's normalized output with respect to ej, and use the local response ∆sj = Jj∆ej as a proxy for latent semantic change.
- Perform SVD on Jj: the left singular vectors give orthogonal response directions, and the singular values give response magnitudes (Eq. 9).
- Under a fixed total weight tr(Q) = d, minimize the average worst-case risk; Proposition 1 gives the anchor-level optimal penalty shape Wj = Uj Σj Uj⊤ / √nj, i.e., weighted by response direction and scaled by singular values; the input proxy is taken as zero-mean isotropic, Cov(∆ej) = I/nj.
Image-adaptive aggregation and fine-tuning (Sec 4.2)
- Use the pretrained image–anchor similarity on clean images through softmax to obtain weights πj(x) (Eq. 14), aggregate the anchor constraints with weighting and perform global trace normalization to obtain Mx, with tr(Mx) = d fixed (Eq. 15).
- The TRAG loss is LTRAG = Ei[Δ ri⊤ Mxi Δ ri], combined with the task loss (LTeCoA for classification, InfoNCE for image–text pair training), optimizing a min–max objective: the inner layer uses PGD to generate adversarial images, and the outer layer updates the image encoder (Eqs. 16, 17).
- Key parameters: λ = 15; anchors are 1,000 ImageNet class names; s(aj) and Wj are precomputed once before fine-tuning, and text-encoder differentiation and SVD are not inside the training loop.
Success criteria and evaluation method
- Zero-shot classification reports clean and robust accuracy; retrieval reports TR@1/TR@10 and IR@1/IR@10; robustness is evaluated with two white-box attacks from AutoAttack, each with 20 steps.
- Relation preservation uses three metrics: feature distance, relation error (KL divergence DKL(p0∥pθ) with respect to the pretrained model), and Retention (the proportion of correct predictions of the reference model retained under attack).
What experiments did the paper conduct?
On 16 classification datasets, two retrieval sets, text/multimodal attacks, and LVLM transfer, TRAG has the highest value on most metrics; it is also best on relation-preservation metrics.
Experimental setup
- Evaluated models: CLIP ViT-B/32 (main experiments), ViT-B/16, ViT-L/14, RN101; CLIP ViT-B/16, CoCa ViT-L/14, SigLIP ViT-B/16 for image-text pair fine-tuning; LLaVA-1.5-7B for LVLM transfer (only its CLIP ViT-L/14 visual encoder is replaced).
- Datasets: ImageNet fine-tuning + 15 zero-shot classification datasets (Caltech-101/256, CIFAR-10/100, DTD, EuroSAT, FGVC Aircraft, Flowers102, Food-101, ImageNet-R/S, Oxford-IIIT Pet, Stanford Cars, STL-10, SUN397); after CC3M fine-tuning, retrieval is tested on Flickr30k and COCO; the LVLM is tested on COCO/Flickr30k captioning and TextVQA/VQAv2.
- Baselines: TeCoA, PMG-AFT, FARE, TGA-ZSR, QT-AFT, AGFT; AdvFLYP is added for retrieval experiments.
- Training configuration: default ℓ∞ budget 2/255, 5 epochs, AdamW, learning rate 2×10⁻⁵, global batch 256, inner 5-step PGD (step size 1/255), λ = 15, anchors are 1,000 ImageNet class names, two RTX 3090s.
- Attack evaluation: AutoAttack's APGD-CE and targeted APGD-DLR, 20 steps each; on the text side BERT-Attack, multimodal Sep-Attack, Co-Attack; for the LVLM, APGD and Ensemble attacks (2/255).
- Metrics: clean/robust accuracy, retrieval R@1/R@10, CIDEr, VQA accuracy; relation preservation uses feature distance, relation error (KL), Retention (500 images from each of the 15 zero-shot datasets). The paper does not report number of repetitions or statistical significance tests.
Main results
Table 1: CLIP ViT-B/32 fine-tuned on ImageNet, training and test budgets both 2/255, 16-dataset averages (%):
Method Clean Robust CLIP (not fine-tuned) 65.60 0.35 TeCoA 49.20 26.33 PMG-AFT 54.08 31.99 FARE 57.09 29.34 TGA-ZSR 52.57 29.32 QT-AFT 56.07 29.86 AGFT 56.35 33.60 TRAG 59.60 35.09 - The authors state that TRAG's average clean and robust accuracy both exceed all fine-tuning baselines, with absolute improvements of 2.5% and 1.5% over the strongest baselines (clean FARE, robust AGFT).
- The authors state that TRAG ranks first in clean accuracy on 11 of 16 datasets and first in robust accuracy on 12; exceptions include EuroSAT, where TRAG's clean 62.78% is below TGA-ZSR's 70.53%, and FGVC, where 13.47% is below FARE's 14.94% (Table 1).
Different budgets and backbones
- Training budget (Table 2, 16-dataset averages): at 1/255, TRAG is 63.66 clean/44.39 robust; at 3/255, 56.35/28.12; at 4/255, 54.55/22.18, with the highest robust accuracy at each budget; the authors state the clean lead increases from 1.8% at 1/255 to 5.6% at 4/255.
- Backbones (Table 3, 2/255): TRAG is 62.95/39.08 on ViT-B/16, 70.78/49.54 on ViT-L/14, and 55.39/29.54 on RN101, all highest; the authors state that relative to the best baseline for each backbone, clean improves by 1.7%–3.5% and robust by 0.7%–2.8%. TGA-ZSR is not reported on RN101.
Image-text pair fine-tuning and retrieval
- Zero-shot retrieval (Table 4, CC3M fine-tuning, PGD 2/255): TRAG has the highest clean and robust recall across three VLMs, two datasets, and two retrieval directions; e.g., SigLIP on COCO has robust TR@1 37.94% and IR@1 26.90%, which the authors state is 3.0% and 2.6% higher than the strongest baselines (AGFT, QT-AFT).
- Text and multimodal attacks (Table 5, CLIP ViT-B/16, Flickr30k, TR@m/IR@m are the means of R@1 and R@10): TRAG ranks first on all six metrics under BERT-Attack, Sep-Attack, and Co-Attack, which the authors state is up to 2.8% higher than the strongest baseline.
- Retrieval checkpoints transferred to classification (Table 6): the same checkpoints are evaluated directly on 16 classification datasets, and TRAG has the highest averages on CLIP (61.79/35.05), CoCa (71.40/48.19), and SigLIP (70.73/43.77).
LVLM transfer
- Setup: for LLaVA-1.5-7B, only the visual encoder is replaced with the CLIP ViT-L/14 fine-tuned in Sec 5.1, with no further LVLM fine-tuning (Table 7).
- Results: TRAG leads in seven of eight task–attack combinations; under Ensemble attack, COCO captioning CIDEr is 73.47 and Flickr30k 40.86, which the authors state is 2.2 and 1.4 higher than the strongest baseline. Exceptions: on Flickr30k clean CIDEr, TRAG's 71.38 is below QT-AFT's 74.27; on TextVQA under APGD attack, TRAG's 19.72 is below FARE's 19.82.
- Figure 4 presents a comparison of LLaVA's caption outputs on clean and adversarial images after fine-tuning with different methods.
Other ablations and analyses
- Constraint ablation (Table 8, CLIP ViT-B/32, 2/255): TRAG is 59.60 clean/35.09 robust, higher than FM (58.69/34.44), DM (54.81/34.91), FM+DM (58.72/34.44), Uniform (58.92/34.66), and Shuffled (58.83/34.76); the authors argue that the lower Uniform and Shuffled results support image-adaptive weighting and correct anchor–response correspondence.
- Relation preservation (Table 9): TRAG's feature distance (C 0.242/A 0.271) is larger than FM and FM+DM, but its adversarial KL is lowest (0.995) and Retention highest (46.26%); the authors argue this indicates TRAG preserves image-text relations while allowing visual adaptation.
- λ trade-off (Figure 5a): as λ increases, clean accuracy rises and robust accuracy falls, and both are above the fine-tuning baselines in Table 1 within the tested range; λ = 15 is used.
- Anchor set ablation (Figure 5b): anchors are CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, and ImageNet-21K class names; only 10 anchors already exceed the baselines, robust accuracy is highest with ImageNet-1K, and slightly decreases with ImageNet-21K.
- Training cost (Table 10): precomputing constraints for 1,000 anchors takes 6.25 minutes; total training time is 4.11 hours (TeCoA 4.08), peak GPU memory 7.32 GiB.
What are some points that can be further explored?
The paper does not specifically discuss limitations; the experiments cover three VLM architectures, two fine-tuning paradigms, and LVLM transfer, with budgets limited to 1/255–4/255.
Limitations and future directions noted by the authors
- The paper does not specifically discuss limitations: the main text has no Limitations section, and the Conclusion (Sec 6) only summarizes contributions without listing shortcomings or future work.
Scope of experimental coverage
- The evaluated models are CLIP (ViT-B/32, ViT-B/16, ViT-L/14, RN101), CoCa ViT-L/14, and SigLIP ViT-B/16, 6 VLM configurations in total, plus visual encoder replacement transfer on LLaVA-1.5-7B (Sec 5.1–5.3).
- Fine-tuning paradigms cover two: ImageNet class-supervised fine-tuning (LTeCoA objective) and CC3M image-text pair fine-tuning (InfoNCE objective); only the image encoder is updated, and the text encoder is always frozen (Sec 4.2, 5.2).
- Attack evaluation covers: image white-box attacks (APGD-CE, targeted APGD-DLR, 20 steps each), the text attack BERT-Attack, multimodal Sep-Attack and Co-Attack, and APGD and Ensemble attacks on the LVLM; perturbation budgets are ℓ∞ 1/255–4/255 (Sec 5.1, 5.2, 5.3).
- Anchor set experiments cover CIFAR-10/100, Tiny-ImageNet, ImageNet-1K, and ImageNet-21K class names; the main experiment is fixed at 1,000 ImageNet class names (Figure 5b).
- The paper does not report: number of repeated experiments and variance, statistical significance tests, adaptive attack evaluation against TRAG, or causal validation between relation-preservation metrics and downstream performance (Table 9 is a correlation observation).
Summarize the main content of the paper
TRAG uses the local responses of a pretrained text encoder to construct image-adaptive drift constraints, achieving the highest clean and robust performance on multiple VLMs and tasks.
A defense paper on VLM adversarial fine-tuning, studying how to preserve pretrained image-text alignment and maintain zero-shot generalization while improving adversarial robustness.
- Problem: The authors' analysis argues that existing pretraining guidance has complementary limitations—feature matching constrains all directions equally with no semantic emphasis; predictive distribution matching has semantic emphasis but is limited by candidate texts (Sec 3.2).
- Method: TRAG uses known texts as semantic anchors, computes the local response Jacobian with respect to the frozen text encoder and performs SVD, derives anchor-level constraint matrices under a fixed penalty budget, then aggregates them into image-adaptive constraints weighted by pretrained image–anchor relevance, and performs min–max adversarial fine-tuning together with the task loss; the constraints are precomputed once before training.
- Main results: CLIP ViT-B/32 averages 59.60% clean and 35.09% robust across 16 datasets, both exceeding all fine-tuning baselines (Table 1); this advantage holds at 1/255–4/255 budgets and on ViT-B/16, ViT-L/14, and RN101 backbones (Table 2, 3).
- Generalization: After CC3M fine-tuning, all metrics are highest on Flickr30k/COCO retrieval and under text and multimodal attacks (Table 4, 5); after replacing the LLaVA-1.5-7B visual encoder, it leads in seven of eight task–attack combinations (Table 7).
- Mechanistic evidence: TRAG has the lowest adversarial relation error (0.995) and the highest Retention (46.26%); the authors argue this indicates it preserves pretrained image-text relations while allowing visual adaptation (Table 9).
- Authors' conclusion: Local textual responses can be converted into image-adaptive directional guidance for preserving diverse pretrained image-text relations during adversarial fine-tuning; training cost is close to TeCoA (4.11 hours vs. 4.08 hours, Table 10).
Related papers
- Semantic-aware Adversarial Fine-tuning for CLIPDefensesarXiv
Show 3 more papers
- Simple Transparent Adversarial ExamplesAttacksarXiv