How and When Does Adversarial Robustness Transfer in Knowledge Distillation?

1. 01What problem does this paper attempt to solve?

   Standard knowledge distillation cannot preserve the teacher's adversarial robustness; the authors propose input gradient alignment (KDIGA) to transfer robustness.

   Core problem: in knowledge distillation (KD), the teacher's adversarial robustness cannot be passed to the student model; how to preserve robustness during distillation.

   • Setting and importance: KD is widely used for model compression and resource-constrained deployment, but existing work mainly focuses on preserving the teacher's accuracy. The authors argue that if the student loses robustness, it creates an illusion of "successful distillation" and affects safe deployment.
   • Shortcomings of existing methods: the authors say standard KD cannot preserve adversarial robustness (Figure 1); adversarial robust distillation ARD (Goldblum et al., 2020) requires computing adversarial perturbations for all training data, which is computationally expensive on large-scale datasets (such as ImageNet).
   • Core observation: aligning logits alone is insufficient; the student's loss surface can differ from the teacher's in a neighborhood; aligning the teacher-student input gradient (input gradient alignment) is key to preserving robustness.
   • What the paper proposes: KDIGA—adding an input gradient alignment term to the KD objective; two variants combined with ARD, KDIGA-ARDC / KDIGA-ARDA; and a theoretical proof of certified robustness preservation plus a general bound based on local linearity (LLM).
   • Threat model (evaluation side):
     • Attacker's goal: cause the distilled student model to misclassify at test time.
     • Knowledge: white-box attacks—PGD (Madry et al., 2017) and AutoAttack (Croce & Hein, 2020); the paper calls AutoAttack the strongest white-box attack at the time.
     • Capability: ℓ∞ norm-bounded perturbations, pixel values scaled to 0–1, with multiple radii (ImageNet 0.001–0.01, CIFAR-10 1/255–8/255).
     • Victim system: the distilled student model (image classifiers such as ResNet, MobileNetV2, ViT).

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: robustness transfer under other settings, knowledge distillation, and adversarial training and attacks; this paper is positioned as adversarial-training-free robustness transfer under the KD setting.

   Robustness transfer in other machine learning settings

   • Hendrycks et al. (2019), Chen et al. (2020), Shafahi et al. (2019)—study adversarial robustness in transfer learning; Chan et al. (2020)—robustness transfers through input gradients in representation learning; Wang et al. (2021a)—robustness adaptation in MAML. The paper notes that its difference from these works is its focus on the knowledge distillation setting.

   Knowledge distillation

   • Hinton et al. (2015) and Gou et al. (2021)—the basic form and a survey of KD: using a CE loss plus a KL divergence with temperature T to make the student imitate the teacher's logits.
   • Zagoruyko & Komodakis (2016b), Passban et al. (2020), Mirzadeh et al. (2020)—distillation improvements aimed at preserving the teacher's accuracy. The paper notes that these works ignore other properties such as adversarial robustness.

   Adversarial training and attacks

   • Madry et al. (2017), Zhang et al. (2019), Engstrom et al. (2019)—adversarial training, obtaining robust models by adding adversarial perturbations during training; the teacher models in this paper come from the ResNet50 checkpoint of Engstrom et al. (2019) and a TRADES-trained WideResNet.
   • Goldblum et al. (2020)—ARD, which embeds adversarial training into the distillation objective. The paper notes two differences: KDIGA does not use adversarial training and is cheaper computationally; and it can be stacked with ARD to further improve the student's robust accuracy.
   • PGD (Madry et al., 2017) and AutoAttack (Croce & Hein, 2020)—used in this paper as robustness evaluation attacks; the paper calls AutoAttack the strongest white-box attack at the time.

   Baselines and benchmarks: compared methods are standard training (ST), standard KD, ARD, and this paper's KDIGA, KDIGA-ARDC, KDIGA-ARDA; datasets are ImageNet and CIFAR-10.

   The authors position this paper as: studying "how and when" robustness transfers in the KD setting, with a method that does not require adversarial training and can be combined with ARD-type methods.

   Collapse this answer

3. 03How does the paper address this problem?

   Adds a teacher-student input gradient alignment term to the KD loss (KDIGA), and proves under two assumptions that the student inherits at least the teacher's certified robust radius.

   Overall idea: standard KD aligns only teacher and student logits; KDIGA (Knowledge Distillation with Input Gradient Alignment) additionally aligns the gradients of the two cross-entropy losses with respect to the input, making the student's loss surface consistent with the teacher's within the teacher's certified radius, thereby preserving adversarial robustness; no adversarial examples are used.

   Problem Setup and Objective

   • Student f s and teacher f t are both R^D→R^N classifiers. The KDIGA objective is L_IGA=λ_CEL_CE(f^s(x),y)+λ_KLT²L_KL(f^s(x)/T,f^t(x)/T)+λ_IGA|∇_x L_CE(f^s(x),y)−∇_x L_CE(f^t(x),y)|₂, i.e., standard KD plus an ℓ2-norm term on the input gradient difference.
   • The training procedure is in Algorithm 1: each step forwards both teacher and student, computes each CE gradient with respect to the input, weights the three terms, and backpropagates.

   Two Variants Combined with ARD

   • KDIGA-ARDC: ARD's original objective (student does KL on perturbed samples against the teacher's clean samples) plus an IGA regularizer on clean samples.
   • KDIGA-ARDA: Align both predictions and input gradients on perturbed samples. In both, the inner perturbation δ is computed by the maximization in Eq. 3. The authors state they also tried other variants and observed no significant difference.

   Theoretical Results

   • Definition 1 (δ-robust): The argmax prediction is unchanged when the perturbation does not exceed δ. Two assumptions: perfect student (distillation loss is 0 everywhere); local linearity (piecewise-linear activation networks are locally linear within the certified region).
   • Proposition 1: If the teacher is δ-robust and the student is a KDIGA perfect student, the student is at least δ-robust; the proof is in Appendix A, and it explains that KD without gradient alignment cannot guarantee robustness because the first-order derivative in the neighborhood is unconstrained.
   • Proposition 2 (general bound, no assumptions): |L_CE(f^s(x+ϵ),y)−L_CE(f^t(x+ϵ),y)|≤γ^s+γ^t+ϕ, where γ is the local linearity LLM of teacher and student respectively (Definition 2, the maximum deviation of CE from its first-order Taylor expansion within the δ neighborhood), and φ contains the teacher and student CE losses and δ times the input gradient difference norm. Meaning: student robustness is jointly constrained by the teacher and student LLMs, the clean loss, and the degree of gradient alignment.

   Evaluation Setup

   • Metrics: clean accuracy and robust accuracy against PGD and AutoAttack, ℓ∞ threat model, multiple attack radii.
   • Key hyperparameters: ImageNet distillation 50 epochs, batch 128; CIFAR-10 200 epochs, batch 125; temperature T=1, λCE=λKL=0.5, λIGA=10/B (B is batch size).

   Collapse this answer

4. 04What experiments did the paper conduct?

   On ImageNet and CIFAR-10, KDIGA students have higher robust accuracy than KD/ST; on CIFAR-10 the variant combined with ARD is highest, and the LLM bound is consistent with the empirical trend.

   Experimental Setup

   • Teacher models: For ImageNet, an adversarially trained ResNet50 (Engstrom et al., 2019 checkpoint, radius 4/255) and standard-trained ViT-S/16, ViT-B/16, ViT-L/16, DEIT-S/16; for CIFAR-10, a TRADES-trained WideResNet.
   • Student models: For ImageNet, mainly ResNet18, with ResNet34/50/101 and a pretrained ViT-S/16 also tested (fine-tuning approach, marked * in the table); for CIFAR-10, MobileNetV2, all trained from scratch.
   • Baselines: Standard training (ST), standard KD, ARD (CIFAR-10 only; the authors state that ARD does not converge under the ImageNet setting and trains very slowly).
   • Attacks and metrics: ℓ∞ perturbation; for ImageNet, 40-step PGD and AutoAttack evaluated on 1000 test samples, radius 0.001–0.01; for CIFAR-10, 20-step PGD and AutoAttack evaluated on all test samples, radius 1/255–8/255. Metrics are clean accuracy and robust accuracy.
   • Training amount: All ImageNet distillation for 50 epochs; CIFAR-10 for 200 epochs. The paper does not report the number of repetitions or variance.

   Main Results

   According to Table 1 (ImageNet, 40-step PGD, teacher robust accuracy in parentheses):

   Swipe horizontally to read the table

   Model (method)	Clean	0.001	0.003	0.005	0.01
   ResNet18 (ST)	68.7	24.9	2.0	0.6	0.0
   ViT-S/16(ST)→ResNet18 (KD)	69.0	30.1	2.9	0.6	0.0
   ViT-S/16(ST)→ResNet18 (KDIGA)	60.0	51.0	32.7	18.0	3.3
   ResNet50(AT)→ResNet18 (KD)	66.3	25.7	1.5	0.0	0.0
   ResNet50(AT)→ResNet18 (KDIGA)	54.2	48.2	37.5	26.5	9.2
   ResNet50(AT)→ResNet101 (KDIGA)	60.3	55.3	44.7	33.1	12.7
   ResNet50(AT)→ViT-S/16* (KDIGA)	77.7	65.3	50.4	33.5	11.1

   • The authors state that standard KD does not preserve robustness: at radius 0.003, KD students reach only 1.5% (ResNet50(AT) teacher) and 2.9% (ViT-S/16 teacher), while the teachers are 59.4% and 24.6%, respectively.
   • The authors state that KDIGA plays a key role, and that robustness can transfer bidirectionally between CNN and ViT; after fine-tuning the pretrained ViT-S/16, clean accuracy is 77.7%, not lower than the original pretrained 77.6%.
   • Cost: KDIGA students have lower clean accuracy than KD (e.g., 54.2 vs. 66.3); the paper does not discuss this.

   CIFAR-10 Results

   According to Table 3 (WideResNet(TRADES)→MobileNetV2, 20-step PGD):

   Swipe horizontally to read the table

   Method	Clean	2/255	4/255	8/255
   ST	91.82	1.11	0.00	0.00
   KD	92.56	24.63	12.55	5.97
   ARD	91.65	68.34	49.49	20.73
   KDIGA	93.03	44.49	30.16	17.81
   KDIGA-ARDC	92.22	71.76	53.46	25.85
   KDIGA-ARDA	90.67	70.57	52.58	27.50

   • The authors state that KDIGA has the highest clean accuracy among all methods (93.03), and its robust accuracy is comparable to ARD but without generating adversarial examples and with lower computation; the paper does not report specific timings.
   • The two combined variants have higher robust accuracy than ARD alone, which the authors state shows that KDIGA can be stacked with existing methods.
   • The authors point out that ARD with λCE=0, λKL=1 has higher robust accuracy but clean accuracy drops considerably, so a fair setting of 0.5/0.5 is used uniformly.

   Local Linearity Bound and Theoretical Consistency

   • What was measured: Table 4 computes each model's LLM (radius 4/255 and 8/255), CE loss, and gradient alignment norm ‖g s − g t ‖2 on 1000 random CIFAR-10 test samples.
   • Results: ST is largest on all items (llm4/255=12.413, ‖g s − g t ‖2=4.099); KD has the lowest CE (0.218) but high values on the other items; KDIGA's bounds are close to ARD's (llm4/255 of 2.561 and 1.326, respectively); KDIGA-ARDC is smallest (1.081, 0.339).
   • Authors' interpretation: The ordering of bound magnitudes is consistent with the empirical robustness in Table 3; KD optimizing only clean accuracy can explain its failure to transfer robustness.

   Other Ablations and Analyses

   • Student capacity (Table 1): With ResNet50(AT) as teacher and radius 0.01, ResNet18/34/50/101 students have robust accuracy of 9.2/12.1/12.4/12.7, respectively; the authors state that larger capacity gives better results.
   • Non-robust teacher (Table 2): The DEIT-S/16 teacher under AutoAttack radius 0.001/0.003/0.005 is 0.4/0.0/0.0, and its KDIGA student ResNet18 is 50.0/23.7/7.8; the authors state that the student can exceed the teacher.
   • AutoAttack supplement (Table 2): Trends are consistent with PGD, e.g., ResNet50(AT)→ResNet18 at radius 0.003 is 31.9 (teacher 42.5).
   • ViT teacher variants (Table 1): ViT-B/16, ViT-L/16, DEIT-S/16 distilled to ResNet18 at radius 0.003 are 26.6, 28.6, 31.5, respectively.

   Collapse this answer

5. 05What are some points that can be further explored?

   The paper has no dedicated limitations section; the experiments cover image classification distillation on ImageNet and CIFAR-10, and some settings have a limited sample size or number of epochs.

   Limitations and future directions noted by the authors

   • The paper does not specifically discuss limitations; there is no Limitations or Future Work section. Only two shortcomings are explicitly stated in the main text: ARD does not converge under the ImageNet setting and trains very slowly, so it is compared only on CIFAR-10 (Section 4.2); the ImageNet experiments run only 50 epochs to save cost, and the authors say 100 epochs might be better (Section 4.2).

   Scope of experimental coverage

   • Tasks and data: image classification only, with datasets ImageNet and CIFAR-10 (Section 4).
   • Architectures: teachers and students cover ResNet, WideResNet, MobileNetV2, ViT/DEIT, 9 models in total; students include both training from scratch and fine-tuning from pretrained (Section 4.1).
   • Evaluation samples: ImageNet robust accuracy is based on 1000 test samples, CIFAR-10 on the full test set; the bounds in Table 4 are computed using a random 1000 test samples (Sections 4.1, 4.4).
   • Attacks: PGD (40 steps/20 steps) and AutoAttack under the ℓ∞ threat model, at multiple radii; no other norms or other attacks are used.
   • The paper does not report the number of repetitions, variance or statistical tests for each experiment, nor the specific training time of KDIGA relative to KD.

   Collapse this answer

6. 06Summarize the main content of the paper

   KDIGA transfers adversarial robustness into the student model by aligning teacher and student input gradients, validates the effect on two image datasets, and provides theoretical guarantees.

   A methods and theory paper on adversarial robustness transfer in knowledge distillation.

   • Problem: The authors state that standard KD preserves only accuracy and cannot preserve the teacher's adversarial robustness; the existing ARD method requires generating adversarial examples for all training data, which is costly on large-scale datasets.
   • Method: Add to the KD objective a term aligning the input gradients of the teacher and student CE losses (KDIGA), without using adversarial examples; two variants combined with ARD, KDIGA-ARDC / KDIGA-ARDA, are also given. Theoretically, under a perfect-student and local-linearity assumption, they prove that the student inherits at least the teacher's certified robust radius (Proposition 1), and give a general bound based on local linearity (LLM) (Proposition 2).
   • Results: On ImageNet, ResNet50(AT)→ResNet18, at PGD radius 0.003 robust accuracy rises from 1.5% for KD to 37.5% (Table 1); on CIFAR-10, WideResNet(TRADES)→MobileNetV2, at radius 8/255 it rises from 5.97% to 17.81%, and KDIGA-ARDA reaches 27.50% (Table 3).
   • Transferability: The authors state that robustness can transfer bidirectionally between CNN and ViT; after fine-tuning the pretrained ViT-S/16, robust accuracy rises to 50.4% (radius 0.003) and clean accuracy does not drop at 77.7%; larger student capacity gives better results.
   • Theory validation: On CIFAR-10, the ordering of LLM bound magnitudes across methods is consistent with empirical robust accuracy (Table 3, Table 4).
   • Authors' conclusion: KDIGA can transfer adversarial robustness from teacher to student while maintaining clean accuracy, and can be stacked with adversarial-training-type methods; the cost is that in some settings clean accuracy is lower than standard KD.

   Collapse this answer

Related papers