When Can Contrastive Learning Preserve Adversarial Robustness from Pretraining to Fine-tuning?

1. 01What problem does this paper attempt to solve?

   How to preserve the adversarial robustness obtained from self-supervised contrastive pretraining for downstream tasks when only standard linear fine-tuning is used.

   Core problem: under the "self-supervised pretraining + supervised fine-tuning" paradigm, the adversarial robustness obtained from contrastive learning (CL) pretraining is difficult to transfer to downstream tasks, which the authors call the "cross-task robustness transferability" problem.

   • Problem setting: CNN image classification faces adversarial example threats in high-risk scenarios such as face recognition, medical imaging, and autonomous driving (Introduction). Adversarial training (AT) is a strong defense but relies on labeled data.
   • Shortcomings of existing methods: The authors state that the downstream robustness achieved by existing self-supervised robust pretraining (ACL [27], RoCL [28]) largely depends on adversarial full fine-tuning or adversarial linear fine-tuning; RoCL has a large performance gap between standard linear fine-tuning and adversarial linear fine-tuning (Sec. 2).
   • Core observation: Even if the pretraining and fine-tuning data are from the same distribution, robustness is still "forgotten" because the pretraining task (instance discrimination) does not match the downstream task (supervised classification); the authors consider the design of contrastive views and the pseudo-supervision signal to be key.
   • Method proposed in the paper: Proposes AdvCL, which constructs contrastive views from adversarial examples of the original image and high-frequency components, and uses ClusterFit clustering to generate pseudo-labels that provide supervisory stimulation, with the goal of enhancing cross-task robustness transferability without sacrificing standard accuracy and fine-tuning efficiency.
   • Threat model:
     • Defender's goal: pretrain robust representations on unlabeled data so that downstream classifiers can resist adversarial examples with only lightweight fine-tuning (Sec. 3).
     • Attacker's capability: at test time, perturbations are generated with 20-step PGD and Auto-Attack under the l∞ norm with ε=8/255; inside pretraining, 5-step PGD is used (Sec. 5).
     • Victim system: an image classifier with ResNet-18 as the encoder, obtained via linear or full fine-tuning.
     • Knowledge setting: the paper does not explicitly discuss the attacker's knowledge of the model (white-box/black-box).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: self-supervised representation learning, adversarial training and its variants, and self-supervised adversarial training; this paper directly compares with ACL, RoCL, and AP-DPE.

   Self-supervised/contrastive representation learning

   • Early methods relied on hand-designed pretext tasks, such as rotation prediction [38], jigsaw puzzles [39, 40], geometric prediction [41], and Selfie [42].
   • Contrastive learning and its variants (SimCLR [30], MoCo [34, 44], BYOL [31], SimSiam [46], etc.) learn representations by maximizing the consistency of different views of the same sample; this paper uses SimCLR's NT-Xent loss as the base framework.
   • ClusterFit [54]: uses K-means to cluster pre-trained representations to generate pseudo-labels to improve generalization; this paper uses it to produce supervisory stimuli.

   Adversarial training (AT) and its variants

   • Madry et al. [11] proposed min-max adversarial training, which is this paper's supervised baseline; there are many subsequent variants [19–21, 47–50].
   • [22–24] extend AT to the semi-supervised setting, using additional unlabeled data to improve robustness; the authors are inspired by this and further ask whether labels can be dispensed with entirely.

   Self-supervised adversarial training

   • AP-DPE [26]: uses self-supervised pretext tasks for adversarial pre-training; the authors point out that its proposed adversarial full fine-tuning requires updating all pre-trained weights, weakening the advantages of self-supervised pre-training.
   • ACL [27] and RoCL [28]: combine AT with contrastive learning; the authors point out that they lack a systematic study of "when and how to preserve downstream robustness without sacrificing the efficiency of lightweight fine-tuning," and that [27]'s pseudo-label generation draws on labeled data, whereas this paper uses clustering information instead.
   • [29] studies self-supervised adversarial robustness in the low-label, high-data regime; the paper only describes it.

   Baselines and benchmarks: the comparison baselines are supervised AT [11], AP-DPE [26], RoCL [28], and ACL [27]; the datasets are CIFAR-10, CIFAR-100, STL-10 (main text) and SVHN, TinyImageNet (Appendix E).

   The authors position this paper as: without relying on labels or adversarial fine-tuning, solving through view design and pseudo-supervisory stimuli the previously unexplored problem of "robustness transfer from pre-training to linear fine-tuning" (end of Sec. 2).

   Collapse this answer

3. 03How does the paper address this problem?

   AdvCL = four-view adversarial contrastive loss (adversarial view of the original image + high-frequency view) + ClusterFit pseudo-label adversarial training regularization term.

   Overall approach: during contrastive pre-training, inject two types of signals simultaneously—use adversarial views and high-frequency views to construct a robustness-aware contrastive objective, and use ClusterFit clustering pseudo-labels to provide class-discriminative supervisory stimuli, so that the learned representations themselves can support downstream robust classification.

   Problem formulation

   • Pre-training: min_θ E_{x∈ X} max_{|δ|∞ ≤ ϵ} ℓ_pre(x+δ, x; θ), i.e., min-max optimization of the contrastive loss on unlabeled data (Equation (2)).
   • Fine-tuning: freeze the encoder fθ and train only the linear head φθc to minimize the supervised cross-entropy (Equation (3)); the paper does not use standard full fine-tuning, because [26] points out that it cannot preserve robustness.

   Robustness-aware view selection

   • Adversarial view: unlike [27–29], which add perturbations to augmented views τi(x), AdvCL adds perturbations directly to the original image x, with the perturbation generated by maximizing the three-view contrastive loss: δ^∗ = argmax_|δ|≤ϵ ℓ_CL(τ₁(x), τ₂(x), x+δ) (Equation (7)). The authors' rationale is that perturbations generated on aggressive augmentations (small crops) are not suitable for defending against attacks acting on the full image.
   • High-frequency view: use the Fourier transform to decompose the image into a high-frequency component xh and a low-frequency component xl (Equation (8)), with a hard split at a spectral center distance threshold r=8; the authors cite [52, 53] to state that high-frequency components are related to both generalization ability and adversarial perturbations.
   • The final contrastive objective is the adversarial contrastive loss over four views (τ1(x), τ2(x), x+δ, xh) (Equation (9)).

   Pseudo-supervisory stimulus generation

   • Use a pre-trained network fpre (in the experiments, a network trained on ImageNet with SimCLR or supervised training) to extract representations of unlabeled data, cluster into K classes with K-means, and use cluster indices as pseudo-labels.
   • The total objective (Equation (10)) is the adversarial contrastive loss plus an adversarial cross-entropy regularization term on the pseudo-labels, with weight λ=0.2; pseudo-labels can be generated offline and add no training overhead (Figure 2 caption).
   • When the number of clusters K is unknown, ensemble over K ∈ {2, 10, 50, 100, 500}, with one linear head per K, averaging the losses; the authors state that the ensemble version generalizes better.

   Training and implementation details

   • Pre-training: SGD, batch size 512, initial learning rate 0.5, 1000 epochs, cosine decay, contrastive loss temperature t=0.5 (Appendix B).
   • Pre-training perturbations: 5-step l∞ PGD, ε=8/255; the augmentation set T includes random cropping (scale 0.2–1), horizontal flipping, color jitter, and random grayscale.
   • Because Equation (10) contains two types of adversarial perturbations (δ maximizing the contrastive loss and δce maximizing the cross-entropy), the authors introduce three BNs (TriBN), corresponding respectively to the two types of adversarial data and the normal augmented data (Appendix B).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Comparing AA/RA/SA on datasets such as CIFAR-10/100/STL-10, AdvCL exceeds all baselines on most metrics under SLF, and view and pseudo-label ablations were performed.

   Experimental setup

   • Models under test: the encoder is uniformly ResNet-18; the comparison targets are models produced by different pretraining methods (supervised AT, AP-DPE, RoCL, ACL, AdvCL).
   • Datasets: CIFAR-10, CIFAR-100, STL-10 (main text); SVHN, TinyImageNet (Appendix E); cross-dataset transfer includes CIFAR-10→STL-10, CIFAR-100→STL-10, etc.
   • Metrics: AA (accuracy under Auto-Attack), RA (accuracy under PGD), SA (clean accuracy).
   • Attack settings: at test time, Auto-Attack and 20-step l∞ PGD are used, ε=8/255; during pretraining, 5-step PGD is used, ε=8/255.
   • Fine-tuning schemes: standard linear fine-tuning (SLF, 25 epochs), adversarial linear fine-tuning (ALF), adversarial full fine-tuning (AFF, using the trades loss); the metrics of the best model for each method under each setting are reported (Appendix B).
   • Number of repetitions: not reported in the paper.

   Main results

   According to Table 1 (ResNet-18, SLF setting):

   Swipe horizontally to read the table

   Pretraining method	CIFAR-10 AA	CIFAR-10 RA	CIFAR-10 SA	CIFAR-100 AA	CIFAR-100 RA	CIFAR-100 SA
   Supervised AT	42.22	44.4	79.77	19.53	23.41	50.53
   AP-DPE	16.07	18.22	78.30	4.17	6.23	47.91
   RoCL	28.38	39.54	79.90	8.66	18.79	49.53
   ACL	39.13	42.87	77.88	16.33	20.97	47.51
   AdvCL	42.57	50.45	80.85	19.78	27.67	48.34

   • The authors state that AdvCL achieves improvements over "almost all" baselines; according to the table, the exception under SLF is SA on CIFAR-100 (48.34, lower than Supervised's 50.53 and RoCL's 49.53).
   • Under the AFF setting (lower half of Table 1): AdvCL has the highest AA 49.77 and SA 83.62 on CIFAR-10, while RA 52.77 is slightly lower than ACL's 52.82; on CIFAR-100, AA 24.72 and SA 56.77 are the highest, while RA 28.73 is lower than ACL's 29.38.
   • The abstract and contributions section state: under Auto-Attack (which the authors call the strongest threat model), AdvCL improves over existing self-supervised methods by 3.44% and 3.45% on CIFAR-10 and CIFAR-100, respectively.

   Cross-dataset transfer

   • Table 2 (SLF): for CIFAR-10→STL-10, AdvCL's AA 25.74, RA 35.80, and SA 63.73 are all the highest; for CIFAR-100→STL-10, RA 30.35 is the highest, but AA 20.86 is 0.89 percentage points lower than ACL, and SA 50.71 is 4.8 percentage points higher than ACL (main text).
   • Appendix E (SLF): on SVHN, AdvCL's RA 45.15 and SA 92.85 are the highest; on TinyImageNet, RA 19.39 is the highest, and SA 42.50 is 0.2 percentage points lower than Supervised; on SVHN→STL-10 and TinyImageNet→STL-10, AdvCL's RA and SA are both the highest (Table A3, A4).

   Attack strength and obfuscated gradient check

   • Figure 3 (SLF, CIFAR-10): as PGD steps (0–100) and ε (0–32/255) increase, RA of all methods decreases; the figure annotates AdvCL with RA 50.1 at 100 steps and about 5.72 at ε=32, with curves above the baselines under each setting. The authors state that this rules out spurious robustness caused by obfuscated gradients.
   • Appendix Figures A2/A3/A4 perform the same sweep on CIFAR-10 (SLF and ALF), CIFAR-10→STL-10, and CIFAR-100, and the authors state that AdvCL outperforms the baselines under all attack settings; the caption of Figure A4 is worded as "most cases, especially when the attack is stronger."

   Qualitative analysis

   • t-SNE (Figure 4, CIFAR-10): the authors state that the class boundaries of representations learned by AdvCL are clearer than those of RoCL and ACL; the converted text cannot present the scatter distribution and can only confirm this claim in the caption.
   • Feature inversion maps (FIM) (Figure 5, neuron 502): the authors state that AdvCL's neuron responses show more texture-like features, independent of the seed image choice.
   • Loss landscape (Figure 6): the authors state that AdvCL's adversarial loss landscape is flatter near local optima, and use flatness as evidence of better transferability (based on [60]).

   Other ablations and analyses

   • Fine-tuning schemes (Table 3, CIFAR-10): AdvCL under ALF has RA 52.01 and SA 79.39; the authors state that AdvCL's SLF–ALF gap is smaller than the baselines', and that AdvCL+SLF already outperforms baseline+ALF.
   • View selection (Table 4, CIFAR-10, SLF): the four views (x+δ, τ1(x), τ2(x), xh) achieve RA 50.45 and SA 80.85; replacing xh with low-frequency xl gives RA 49.51; the older practice of perturbing augmented views gives RA of only 42.12–43.51.
   • Pseudo-label settings (Table 5): without ClusterFit, RA 48.89 and SA 77.73; with fpre using ImageNet+SimCLR and K ensemble, the best (RA 50.45, SA 80.85).
   • Training time (Appendix F): AFF takes 136.08s per epoch, 24 times SLF (5.58s); pretraining time for AdvCL is 377.89s×1000, higher than RoCL's 123.15s×1000 and ACL's 126.8s×1000, and lower than AP-DPE; the authors state that pretraining can be done offline.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors only propose extending to larger datasets and models as a future direction; experiments are limited to ResNet-18 and five image datasets, and the number of repetitions is not reported.

   Limitations and future directions noted by the authors

   • Conclusion: The authors state that future work can extend the proposed self-supervised pretraining method to very large datasets and models to further improve robust transferability across datasets.
   • Appendix F: The authors write that the pretraining computational cost of AdvCL is higher than that of the contrastive learning baselines RoCL and ACL, because the contrastive loss uses more than two views and a pseudo-supervision regularization term is added; the authors also state that pretraining can usually be done offline.
   • The paper has no dedicated Limitations section and, apart from the above two items, does not explicitly list other limitations.

   Scope of experiments

   • The encoder under test is a single architecture, ResNet-18 (Sec. 5); the datasets are five image classification datasets: CIFAR-10, CIFAR-100, STL-10, SVHN, and TinyImageNet.
   • The evaluated attacks are Auto-Attack and PGD under the l∞ norm with ε=8/255 (including sweeps over steps 0–100 and ε 0–32/255, Figure 3); the paper does not mention other norms or other attack types.
   • Fine-tuning schemes cover three: SLF, ALF, and AFF; the paper explicitly does not use standard full fine-tuning (Sec. 3).
   • Pseudo-label experiments cover three fpre settings (CIFAR-10+SimCLR, ImageNet+supervised, ImageNet+SimCLR) and K ∈ {2, 10, 50, 100, 500} plus ensemble (Table 5).
   • The paper does not report multiple repeated runs, variance, or statistical significance tests; qualitative analyses (t-SNE, FIM, loss landscape) are shown only on CIFAR-10.

   Collapse this answer

6. 06Summarize the main content of the paper

   Through adversarial views, high-frequency views, and clustering pseudo-labels, AdvCL enables the robustness of contrastive pretraining to transfer downstream with only standard linear fine-tuning.

   A NeurIPS 2021 adversarial defense paper studying the problem of "cross-task robustness transfer" in self-supervised contrastive learning: adversarial robustness acquired during pretraining is easily lost during downstream supervised fine-tuning due to task mismatch.

   • Problem: Existing self-supervised robust pretraining (ACL, RoCL) relies on computationally expensive adversarial fine-tuning for downstream robustness; the authors ask whether robustness can be preserved using only standard linear fine-tuning (SLF).
   • Method: AdvCL adds two new views to SimCLR-style contrastive learning—adversarial examples generated on the original image (rather than augmented views), and the high-frequency components of the image—forming a four-view adversarial contrastive loss; it then uses ClusterFit to cluster unlabeled data and generate pseudo-labels, adds a pseudo-classification adversarial training regularization term (λ=0.2), and employs three BNs to handle the mixed data distribution.
   • Main results (ResNet-18, Table 1): Under SLF, on CIFAR-10, AA 42.57, RA 50.45, SA 80.85, all exceeding all baselines; on CIFAR-100, AA 19.78 and RA 27.67 are the highest, while SA 48.34 is lower than supervised AT and RoCL. Under AFF, some RA metrics are slightly lower than ACL.
   • Transfer and ablation: In CIFAR-10→STL-10 SLF transfer, RA 35.80 and SA 63.73 are the highest (Table 2); ablations show the high-frequency view outperforms the low-frequency view, and removing the pseudo-label regularization term lowers RA from 50.45 to 48.89 (Tables 4, 5).
   • Cost: Pretraining takes about 3 times as long as RoCL/ACL (Table A2); the authors state that pretraining can be done offline; AFF fine-tuning takes 24 times as long as SLF, highlighting the efficiency value of SLF.
   • Authors' conclusion: The authors consider view design and pseudo-supervision stimulation to be the two key factors for contrastive learning to preserve cross-task robustness; AdvCL achieves robust accuracy superior to existing self-supervised methods without sacrificing accuracy or fine-tuning efficiency, and they propose future extension to larger datasets and models.

   Collapse this answer

Related papers