Skip to content
Original

CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

CARD: Closing the Teacher-Student Loop for Adversarial Robustness Distillation

arXiv: Adversarial Attacks, Backdoors & Poisoning· arXiv:2610.19911· Zhi Li, Haowei Liu, Hongchen Yang, Xiaoxuan Wang, Song Gao, Zhenli He, Wei Zhou, Shaowen YaoEngineering Research Center of Cyberspace and School of Software and AI, Yunnan University; School of Information Science and Technology, Yunnan Normal UniversityIndexed by this site · 29 days ago· Originally publishedDefenses

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe researchers propose a closed-loop adversarial robustness distillation framework, CARD: first strong-to-weak adversarial distillation, then refining the robust teacher with reliability-enhanced student feedback, and passing the teacher's gains back to the same student. The method combines reverse adversarial student feedback with teacher-student divergence maximization, and provides sufficient conditions for reliable weak-to-strong feedback transfer. On CIFAR-10, CIFAR-100, and TinyImageNet-10, CARD improves adversarial robustness in most settings while largely maintaining clean accuracy.

Questions
Adversarially robust distillation typically fixes a robust teacher and transfers one-way, discarding the distilled student's feedback. Indiscriminately aligning to weak student predictions may also transfer errors to the teacher. The authors aim to use this feedback to refine the teacher and transfer it back to the same student without changing the base distillation objective.
Methods
CARD first performs strong-to-weak distillation with an existing adversarial distillation objective; then freezes the student, supervises the teacher on clean inputs with inverse adversarial predictions, and performs robust refinement with perturbed inputs selected by divergence maximization; finally, it continues training from the original student checkpoint with the same objective.
Experiments and results
White-box and transfer black-box evaluations are conducted on CIFAR-10, CIFAR-100, and TinyImageNet-10. The authors state that robustness improves in most settings and clean accuracy is usually maintained; some PGD-20 decreases, gains vary with architecture and base objective, and a second loop is not better on all metrics.
Limitations and future work
The authors note that the evaluation is limited to relatively small-scale image classification, mainly ℓ∞ attacks, and that additional refinement and back-propagation increase training cost, and the improvements are not consistent. The experiments include these three datasets, WRN-28/34-10 teachers, and two students; the paper does not report the initial distillation epochs E1 or the number of repetitions.

Related events and news

1 events · 0 news mentions

Related papers

Both use: CIFAR-100
Show 3 more papers
Both use: CIFAR-100
Read originalarxiv.org