Semantic-aware adversarial fine-tuning for CLIP

Adversarial examples generated using the cosine similarity of a single handcrafted template may fail when transferred to more semantically rich image-text similarities, and the resulting fine-tuned CLIP image encoder is not robust enough.

• Scenario: CLIP encodes images and text into a shared space for zero-shot classification and is also used as an image encoder in large vision-language models like Flamingo and LLaVA. The authors argue that its vulnerability to adversarial examples is critical for safe deployment.
• Limitation: Existing adversarial fine-tuning uses the cosine similarity between the image and a single template (e.g., "A photo of a {label}"), i.e., the CLIP score, to generate adversarial examples. The authors claim this score is insufficient to measure image-text alignment. In Figure 1, PGD-minimized CLIP score examples on six animal classes from ImageNet-1K show 100% of CLIP scores below the diagonal; the text states that switching to CuPL and WCA yields only 64.8% and 28.6% respectively (the caption says 24.8% for WCA). Points above the diagonal are explained by the authors as the model being more confident after the attack.
• Observation: The authors believe that perturbations optimized for a single phrasing overfit to the template phrase and fail to transfer to attributes or richer context descriptions, making the samples used in adversarial training less effective and less general.
• Proposal: Semantic-aware Adversarial Fine-Tuning (SAFT) uses a foundation model to generate and filter class descriptions, constructs adversarial examples with a semantic ensemble attack, and then fine-tunes the CLIP image encoder.
• Threat model:
  • Goal: For a task T in an unseen task distribution, maximize the loss within a perturbation ball to degrade zero-shot classification performance.
  • Knowledge: The attacker fully knows the true labels y_T of the task and maximizes L(f^θ(x+δ), y_T). The paper does not specify white-box or black-box.
  • Capability: Perturbations belong to Δ={δ:∥δ∥p≤ϵ}. Experiments use ℓ∞; training uses PGD-2 with ε=1/255.
  • Victim system: The defender does not know the task identity T or distribution D_T, only trains the model. The evaluation is of the CLIP image encoder's cross-task zero-shot robustness, unlike standard adversarial robustness on a known task T_0.

The paper's Related Work is discussed in four groups; only practices that the authors explicitly criticize or compare against are described with differences.

Adversarial Robustness and Adversarial Training

• Adversarial examples and adversarial training: Szegedy et al. (2014), Goodfellow et al. (2015), Carlini & Wagner (2017) study adversarial examples. The paper states that adversarial training (Madry et al., 2018, etc.) is one of the most effective strategies against adversarial examples and remains effective against adaptive attacks (Athalye et al., 2018).
• Trade-off: Tsipras et al. (2019) discuss that robustness and clean accuracy may conflict. The paper states that better balancing this trade-off remains a focus of adversarial training.
• Relation to CLIP: The paper states that existing adversarial training mostly targets supervised image classifiers, is computationally expensive, and is difficult to scale to CLIP.

Vision-Language Models

• CLIP: Radford et al. (2021) use image-text contrastive learning for zero-shot transfer.
• Downstream large models: The paper lists Flamingo (Alayrac et al., 2022), LLaVA (Liu et al., 2023), etc., and states that most large vision-language models use CLIP as the pretrained image encoder due to its text supervision (Tong et al., 2024).
• Vulnerability: Mao et al. (2023), Schlarmann et al. (2024) discuss the vulnerability of these models to adversarial examples. The paper thus considers improving CLIP robustness an important problem.

CLIP Adversarial Fine-Tuning

• TeCoA: Mao et al. (2023). The paper claims it is the first method to improve zero-shot robustness of vision-language models, using fixed zero-shot templates to generate and train adversarial examples.
• Subsequent methods: FARE (Schlarmann et al., 2024) adds an unsupervised objective to enlarge embedding distances; PMG-AFT (Wang et al., 2024a) uses an auxiliary branch to align adversarial and clean representations to mitigate adversarial overfitting; TGA-ZSR (Yu et al., 2024) uses text-guided attention. Zhou et al. (2024) consider multimodal inputs. Pretraining from scratch (Gan et al., 2020; Wang et al., 2024b) is called computationally intensive by the paper.
• Authors' positioning: The paper claims this is the first attempt to show that richer text embeddings can generate adversarial examples that generalize better to slight text variations.

Text Prompts

• Prompt sensitivity: Radford et al. (2021) and Zhou et al. (2022) note that CLIP's downstream performance depends on prompt design.
• Semantically rich descriptions: Menon & Vondrick (2023) and CuPL (Pratt et al., 2023) use large language models to generate class descriptions; WCA (Li et al., 2024) cross-aligns local image patches with CuPL descriptions; Cai et al. (2025) generate descriptive and discriminative attributes; Sun et al. (2026) use IoU and embedding cosine similarity to remove redundant crops and similar descriptions.
• Unexplored: The paper states that these descriptions are effective for image-text alignment, but whether they improve the zero-shot accuracy–robustness trade-off remains underexplored to the authors' knowledge.

Baseline methods are clean-data fine-tuned CLIP, TeCoA, FARE, PMG-AFT, TGA-ZSR. Benchmarks are CIFAR-10/100, STL-10, Tiny-ImageNet, ImageNet-1K, Caltech-101/256, Food101, Flowers102, FGVC-Aircraft, StanfordCars, OxfordPets, PCAM, SUN397, EuroSAT, DTD. The main experiment uses Tiny-ImageNet as the source and the rest as zero-shot targets.

The authors position the paper between "single-template adversarial examples may fail when the similarity is changed" and "semantically rich prompts have not yet been used for this trade-off," constructing adversarial examples with a refined description ensemble and then fine-tuning.

SAFT uses a foundation model to generate and filter text descriptions for each class, then makes adversarial examples deviate from the average embedding of these descriptions, and finally fine-tunes only the CLIP image encoder.

Problem Setting

• Zero-shot adversarial robustness follows Mao et al. (2023): tasks T come from an unseen task distribution, each task has its own label space and data distribution D_T. The defender minimizes the expected worst-case risk over all tasks, rather than training only on a known task T_0.
• CLIP inference embeds labels as text using templates, and takes the argmax over cosine similarity between the image embedding and each class text embedding. Previous template-based fine-tuning made the attack maximize the cosine dissimilarity between the image and f_text(p(y)), with the inner loop approximated by PGD.
• Limitations noted by the authors: adversarial examples easily overfit to phrases like "photo of a", and may not transfer when phrased differently; fixed templates also fail to cover class attributes and context.

Hallucination-Aware Description Generation

• Generation: for class y, the foundation model f_FM generates M descriptions under instruction and hyperparameters ϕ, T_y=f_FM(y;ϕ), intended to cover attributes, context, and synonymous phrasings.
• Filtering: the class name and the m-th description are both encoded by the CLIP text encoder, and the relevance is s^(m)=(f_text(y)· f_text(t_y^(m)))/(|f_text(y)||f_text(t_y^(m))|), i.e., their cosine similarity. Keep the top-K by descending score. The authors give an example: a hallucinated description might be 0.2, a valid description might be 0.8, and the former is removed; this is an example, not a measured summary.
• Two implementations: SAFT-L uses CuPL, with the prompt "What does a {label} look like?", and the experiments use its public descriptions. SAFT-M uses Cai et al. (2025), giving representative images of each class and appearance-class prompts to a multimodal large model to generate descriptive and discriminative attributes.
• Not using WCA: the paper says it requires cropping images into multiple patches, which is computationally expensive, and its text generation aligns with CuPL, so it is not used as an implementation.

Semantic Ensemble Attack

• Objective: instead of aligning to a single template, maximize the average dissimilarity between the perturbed image embedding and the set of description embeddings. The perturbation remains within the ℓp ball.
• Loss: the paper writes the average cosine dissimilarity L_SAFT=−1/M∑_m=1^M(f_img^θ(x+δ)· f_text(t_y^(m)))/(|f_img^θ(x+δ)||f_text(t_y^(m))|). The average is over M descriptions; Algorithm 1 filters first then attacks, and experiments keep K=5.
• Solving: the inner loop in experiments is written as ℓ∞ PGD-2, ε=1/255. The paper gives the update form for template-based PGD, but does not give a numerical value for the step size α.

Outer Fine-Tuning and Evaluation

• Learning objective: Section 4.3 minimizes L_SAFT(f_img^θ(x+δ*), description set), updating only the image encoder parameters θ. The authors say this makes perturbed images fall into regions more invariant to both visual perturbations and language variations. Figure 2 and Algorithm 1 give the pipeline of generation, filtering, encoding, sample construction, and per-epoch updates.
• Experimental loss: Section 5.1 additionally writes the default loss function as TGA-ZSR, and uses top-5 refined descriptions. The paper does not specify how this loss combines with L_SAFT in Eq. (5).
• Success evaluation: zero-shot classification still takes the argmax over cosine similarity with template text. Robust accuracy is classification accuracy after attack; clean accuracy is accuracy on clean images. The paper does not use an LLM judge.
• Generation settings in experiments: SAFT-M uses GPT-4o-mini, 5 training images per class, temperature 0.99, max tokens 100, 5 responses per prompt. Fine-tuning on Tiny-ImageNet for 10 epochs; on ImageNet-1K for 1 epoch. Optimizer is SGD, learning rate 1e-4, momentum 0.9, weight decay 0, batch 128, backbone default ViT-B/32.

Experimental Setup

• Tested backbones: main experiments use ViT-B/32; extension uses CLIP-L/14. Comparison methods are clean fine-tuned CLIP, TeCoA, FARE, PMG-AFT, TGA-ZSR.
• Data: 16 classification sets. Main experiments use Tiny-ImageNet as source, the other 15 are zero-shot targets; Table 7 uses ImageNet-1K as source instead. Retrieval uses COCO, Flickr8k, Flickr30k. Out-of-domain uses TU-Berlin sketch, ImageNet-R.
• Training: SGD, learning rate 1e-4, momentum 0.9, weight decay 0, batch 128. Inner loop is ℓ∞ PGD-2, ε=1/255. 10 epochs on Tiny-ImageNet, 1 epoch on ImageNet-1K. K=5. Outer loop defaults to the TGA-ZSR loss.
• Descriptions: SAFT-L uses CuPL public descriptions. SAFT-M uses GPT-4o-mini, temperature 0.99, max tokens 100, 5 images per class, 5 responses per prompt.
• Evaluation: mainly PGD-100, AutoAttack, C&W, ε=1/255; also PGD-100, ε=4/255. Metrics are clean accuracy, robust accuracy, and 15-set zero-shot average; retrieval uses IRR, TRR. Evaluation is classification accuracy, no LLM judge.
• Repeats: SAFT in Table 1, Table 8, and Table 10 are averages of three runs; Table 1 gives standard deviations for SAFT. The paper does not report the number of repeats for baselines.

Main Results

According to Table 1. Source is Tiny-ImageNet, attack is PGD-100 (ε=1/255), units are %. Zero-shot average excludes the source set. SAFT is mean±std of three runs.

Swipe horizontally to read the table

Method	Source Robust	Source Clean	Zero-shot Avg Robust	Zero-shot Avg Clean
CLIP	13.55	79.04	6.62	52.72
TeCoA	31.29	54.59	26.00	43.08
FARE	23.88	77.54	17.88	58.68
PMG-AFT	32.46	47.36	27.54	37.42
TGA-ZSR	47.87	76.84	31.50	54.58
SAFT-L	53.27±1.56	73.47±0.38	35.35±1.04	55.27±0.52
SAFT-M	50.79±1.43	74.88±0.41	33.78±1.15	56.01±0.61

• The authors say SAFT-L and SAFT-M have zero-shot average robust accuracy 3.85% and 2.28% higher than TGA-ZSR, and 5.40% and 2.92% higher on the source set. The abstract writes at least 3.85%, while the main text writes 2.28% for SAFT-M.
• Clean zero-shot average is highest for FARE; in the table SAFT-M is higher than SAFT-L. The authors list them as second and third respectively, and give 55.27% and 56.01% in that order, which is inconsistent with the table ranking. The authors say their zero-shot robustness is 17.46% and 15.90% higher than FARE.
• Not every cell has SAFT highest: on PCAM robust accuracy, PMG-AFT is 48.92%, SAFT-L is 47.09% (Table 1).

Unseen Attacks

• Table 2 evaluates AutoAttack and C&W on the same source set, ε=1/255, comparing TeCoA, PMG-AFT, TGA-ZSR, and SAFT-L.
• Zero-shot average robust accuracy: on AutoAttack, SAFT-L is 30.06%, TGA-ZSR is 28.89%; on C&W, SAFT-L is 37.45%, TGA-ZSR is 29.94%. The authors say the average gains are 1.17% and 7.51%.
• Exceptions: on AutoAttack OxfordPets, SAFT-L is 36.11%, TGA-ZSR is 39.49%; on C&W PCAM, PMG-AFT is 48.33%, SAFT-L is 42.93%.

Different Backbone and Different Source Data

• Table 4 uses only CLIP-L/14, comparing FARE and SAFT-L, PGD-100, ε=1/255. Zero-shot average clean accuracy is 63.12% vs 61.89%, robust is 53.52% vs 43.51%. The authors say clean is about 2% higher, robust about 10% higher, and say on CLIP-B/32 SAFT-L's clean accuracy is slightly lower than FARE.
• Table 7 uses ImageNet-1K as source, training only 1 epoch. Zero-shot average clean accuracy is highest for SAFT-M at 63.69%; SAFT-L is 63.14%. Robust average is highest for PMG-AFT at 32.78%, higher than SAFT-M's 31.67% and SAFT-L's 30.82%. The authors say their clean accuracy is 8.70% and 8.15% higher than PMG-AFT, with comparable robustness.

Template Transfer and Retrieval

• Table 10 and Figure 3 use 13 templates, 1 of which is the training-time seen "This is a photo of a {label}", 12 unseen; PGD-100; report three-run averages on CIFAR-10, CIFAR-100, STL-10. The authors say SAFT-L's average zero-shot robustness across all templates is higher than TGA-ZSR; the seen template gain is 6.41%, the unseen template "a tattoo of the {label}" is 6.96%. In the table, some unseen templates show clean accuracy drops, plastic is -0.54, toy is -0.72.
• Table 5 uses the fine-tuned image encoder for retrieval, AutoAttack, ε=1/255. On COCO, SAFT-L's IRR/TRR is 35.36%/43.32%, TGA-ZSR is 34.27%/41.64%; on Flickr8k, 58.86%/68.60% vs 57.40%/66.80%; on Flickr30k, 63.86%/72.10% vs 61.14%/69.40%. The paper does not report clean retrieval metrics.

Other Ablations and Analyses

• Semantic filtering (Table 3, PGD-100, ε=1/255): zero-shot average robust 33.64% vs 35.35%, the authors say an improvement of 1.71%; STL-10 robust without filtering is 80.20%, with filtering is 78.97%.
• K (Table 8, average over four datasets, Section 5.3 writes for PGD-100): K=5 robust 53.21% is highest in the table; K=3 is 43.87%, lower than K=1's 50.26%.
• Description model (Table 9, table note does not specify attack): Tiny-ImageNet robust Qwen3-4B-instruct 53.45%, Llama-3.2-3B-instruct 50.14%, Llama-3.2-1B-instruct 49.08%, Qwen3-4B 47.10%. The authors say instruction-tuned and larger models generate better descriptions, but the LLM is not a necessary component of the method itself.
• Larger perturbation (Table 11, PGD-100, ε=4/255): SAFT-L zero-shot average robust 8.65%, TGA-ZSR 7.13%; on the source set SAFT-L 15.92%, lower than TGA-ZSR's 16.76%. The authors say it is still at least 1.52% above the baseline.
• Computation (Table 6, 1 NVIDIA A100, batch 128): SAFT-L (K=5) 38839Mb, 1098s/1.41s; TGA-ZSR 21253Mb, 786s/1.01s. The authors say 0.4 seconds more per batch. The paper writes that SAFT-M and SAFT-L have the same memory and training time.
• Out-of-domain (Table 12, attack and ε not specified): Sketch robust SAFT-L 29.75%, TGA-ZSR 27.05%; ImageNet-R robust 29.70%, 26.54%. Clean accuracy is 43.85% vs 36.80%, 52.85% vs 46.74% respectively.

Limitations and future directions noted by the authors

• Description quality (Limitations): SAFT relies on a foundation model to generate descriptions for each class, and the effectiveness depends on relevance, faithfulness, and diversity. The authors write that if hallucinated or semantically vague prompts occur, the resulting descriptions may be suboptimal. The mitigation proposed in the paper is to filter low-scoring descriptions by relevance.
• Additional computation (Limitations): The description ensemble introduces extra cost. The authors state the process is lightweight and computationally feasible relative to existing adversarial fine-tuning.
• Future directions (Conclusion): The authors hope this framework lays the groundwork for future methods that consider richer text supervision. The paper does not have a separate Future Work section.

Experimental coverage

• Classification experiments use the 16 datasets listed in the paper. The main results use Tiny-ImageNet as source and fine-tune for 10 epochs; Table 7 uses ImageNet-1K as source and fine-tunes for 1 epoch, with Section 5.1 stating this is due to limited computational resources.
• Backbones are ViT-B/32 and CLIP-L/14. Table 4 only reports FARE and SAFT-L.
• Attacks include ℓ∞ PGD-100, AutoAttack, C&W (ε=1/255), and PGD-100 (ε=4/255). The inner training loop uses PGD-2. The paper does not report the step size α.
• There are also retrieval experiments on COCO, Flickr8k, and Flickr30k (Table 5), and TU-Berlin sketch and ImageNet-R (Table 12). Table 12 does not specify the attack name or ε.
• SAFT-L descriptions come from CuPL public results; SAFT-M uses GPT-4o-mini. Table 9 additionally uses Qwen3-4B, Qwen3-4B-instruct, Llama-3.2-1B-instruct, and Llama-3.2-3B-instruct. SAFT in Table 1 is from three runs. The paper does not report baseline repetitions or human agreement. The implementation does not include WCA, which the paper says is due to its cropping overhead and because the text side aligns with CuPL.

SAFT is an adversarial fine-tuning method for the CLIP image encoder, aiming to improve the accuracy–robustness trade-off on zero-shot classification.

• Problem: adversarial examples constructed with a single-template CLIP score may not still reduce alignment when transferred to more semantically rich similarities such as CuPL or WCA. In Figure 1, the proportion of CLIP score below the diagonal is 100%; the main text writes CuPL and WCA as 64.8% and 28.6%, while the figure caption writes WCA as 24.8%.
• Approach: a foundation model generates class descriptions, keeps the top-K by cosine similarity with the class name embedding, then maximizes the average dissimilarity between the image and the description ensemble to construct samples, and fine-tunes only the image encoder. The experimental default outer loss is TGA-ZSR, K=5. The attacker knows the task labels, and the perturbation is within the ℓp ball; the defender does not know the identity or distribution of unseen tasks.
• Main results: with Tiny-ImageNet as source, ViT-B/32, PGD-100 (ε=1/255), SAFT-L's 15-set zero-shot average robust accuracy is 35.35% and clean accuracy is 55.27%; SAFT-M is 33.78% and 56.01% (Table 1). The authors say the robustness gains over TGA-ZSR are 3.85% and 2.28%. The highest clean zero-shot average is FARE.
• Other results: under C&W, SAFT-L's zero-shot average robust accuracy is 37.45% (Table 2). On CLIP-L/14, SAFT-L is 53.52%, FARE is 43.51% (Table 4). With ImageNet-1K as source, the highest clean average is SAFT-M's 63.69%, and the highest robust average is PMG-AFT's 32.78% (Table 7).
• Authors' conclusions: the authors say SAFT improves the zero-shot trade-off on 16 datasets relative to existing adversarial fine-tuning, and generalizes to unseen text templates; they hope this promotes the use of richer text supervision in fine-tuning. The authors also state that the effectiveness depends on the quality of foundation model descriptions, and the description ensemble increases computation.