Towards Evaluating the Robustness of Transduction Learning Neural Networks
Towards Evaluating the Robustness of Neural Networks Learned by Transduction
The authors establish a threat model for transductive-learning defenses and propose the GMSA attack framework, stating that RMC, DENT, and URejectron can all be broken by weak instantiations.
The attacker intercepts the clean test batch V, perturbs the features within the agreed perturbation type (e.g., L∞ norm, budget ε) to obtain U′ and sends it to the defender; the defender uses the transductive learning algorithm Γ to update the model based on U′. The attacker fully knows Γ and the initial pretrained model, but does not know the private randomness that the defender samples after the attacker's action; the multi-round game is divided into states-leaking and non-leaking, and naturally ordered and adversarially ordered, and the main experiments evaluate only non-leaking and naturally ordered.
- Transductive-learning defenses dynamically update the model based on the input at test time, so the attack objective becomes a bilevel optimization; the authors believe that prior evaluation with inductive attacks such as AutoAttack would overestimate their robustness.
- The authors formalize 1-round and multi-round threat models, propose the attacking model space principle: attacking a family of models induced by the U′ neighborhood; and give a greedy instantiation GMSA (two losses, AVG/MIN), solved with PGD.
- The authors state that weak instantiations of GMSA suffice to break RMC, DENT, and URejectron: RMC+Standard on MNIST drops to 0.50% robustness under GMSA-AVG; DENT retains only a small improvement over static adversarial training; TADV instead improves relative to it.
- The authors state that future work could explore transductive defenses that resist GMSA and stronger adaptive attacks. The experiments cover MNIST, CIFAR-10, and GTSRB; RMC and DENT are each evaluated on 1000 test points, and the multi-round evaluation covers only the non-leaking setting.
- Evaluated models
- StandardMadry et al.Wu et al. (2020a)Carmon et al. (2019)Sehwag et al. (2020)Wang et al. (2020)Hendrycks et al. (2019)Wong et al. (2020)Ding et al. (2020)DANNTADVResNet18
- Benchmarks
- MNISTCIFAR-10GTSRB
- Metrics
- AccuracyRobustnesserrrej
The researchers propose the GMSA (Greedy Model Space Attack) attack framework, which can break transductive learning adversarial defenses that previously withstood attacks such as AutoAttack, even with weaker instances. The paper establishes a threat model for such defenses and points out important details in them, proposes solving the bilevel optimization attack objective with the principle of attacking the model space, and GMSA can serve as a new baseline for evaluating transductive learning defenses. The authors also report positive results for transductive adversarial training: adversarially retraining the model with fresh randomness at test time can significantly improve robustness to the considered attacks.
Paper analysis
What problem does this paper attempt to solve?
How to correctly evaluate the robustness of transductive-learning defenses: the authors argue that existing evaluations overestimate them.
Core question: how to conduct principled threat analysis and adaptive attack evaluation for transductive-learning based defenses.
- Problem setting: such defenses use a batch of test-time inputs to dynamically update the model (also called test-time adaptation, dynamic defenses); the authors state that batch prediction is common in deployment, and the defender hopes to use it to adapt to the attacker's perturbed inputs.
- Shortcomings of existing evaluations: the authors state that AutoAttack is the state-of-the-art attack under the inductive threat model but is ineffective against transductive defenses; Wang et al. report that on CIFAR-10 DENT improves over adversarial training by more than 20% absolute against AutoAttack, whereas under the authors' adaptive attack the improvement is less than 3%.
- Core observation: the attack objective is a bilevel optimization, Γ is non-differentiable, and the attack set U′ appears on both the attack and defense sides; exploiting the transferability of adversarial examples, the authors propose the principle of attacking the model space induced by the neighborhood of U′.
- Paper outputs: formalizes the 1-round and multi-round threat models, proposes the attack framework Greedy Model Space Attack (GMSA) as a new baseline, and systematically evaluates RMC, DENT, DANN, TADV, and URejectron.
- Threat model (Definition 1 and Section 4):
- Objective: within the agreed perturbation type (e.g., L∞, budget ε), perturb the features of the test batch V to obtain U′, so that the model learned by Γ(U′) errs on U′.
- Knowledge: white-box—the attacker fully knows Γ, the initial pretrained model F, and the training data D, but does not know the private randomness that the defender samples after the attacker's action.
- Capability: intercepts the clean test batch, perturbs the features, and sends them to the defender; across multiple rounds it can maintain arbitrary state; under the adversarially ordered setting it can stockpile data and then send it in batches in the worst order.
- Victim system: a defender that uses a transductive learning algorithm Γ (optionally with base model F=T(D)) to update the model at test time.
What related research exists?
Related work falls into three groups: inductive attacks and defenses, test-time defenses, and transductive-learning defenses.
Inductive adversarial robustness
- Goodfellow et al. (2015), Carlini & Wagner (2017), Kurakin et al. (2017), Moosavi-Dezfooli et al. (2016), Croce & Hein (2020b)—various attacks proposed for the inductive setting where the model is fixed at evaluation time.
- Madry et al. (2018), Zhang et al. (2019)—adversarial training, the fundamental method for obtaining robustness in this setting; AutoAttack (Croce & Hein, 2020a) is the state-of-the-art attack under the inductive threat model.
- Tramèr et al. (2020) proposed principles for adaptive attacks, and BPDA from Athalye et al. (2018) broke several defenses based on these principles.
Test-time defenses
- One line of work "purifies" test inputs with a non-differentiable function before feeding them to a pretrained model; the authors point out that most of these were broken by BPDA.
- Goodfellow (2019) proposed a research agenda for "dynamic model defense".
Transductive-learning defenses (the evaluation target of this paper)
- RMC (Wu et al., 2020b)—at test time, adapts the model on a single test point using adversarially trained data based on top-K nearest neighbors; the original source evaluated transfer attack and PGD-skip attack, and the authors classify the former as stateless and non-adaptive, and the latter as state-leaking but still non-adaptive.
- DENT (Wang et al., 2021)—stateless, starting from the original pretrained model, updates only the affine parameters of batch normalization layers, and adapts with entropy minimization plus information maximization regularization.
- Goldwasser et al. (2020)—proposed the transductive threat model and URejectron, which learns a set S of "allowed points" to make predictions with rejection, with theoretical guarantees in the bounded VC dimension case; the authors recover its threat model as a special case of the 1-round game.
Baselines and benchmarks: attack baselines are AutoAttack, PGD attack (transfer setting), and FPA; defense comparisons are the static base model (including Madry et al. adversarial training); datasets are MNIST, CIFAR-10, and GTSRB.
The authors position this paper as follows: prior work evaluated only under the naturally-ordered game, and the attacks used were mostly non-adaptive by their definitions; this paper gives principled threat modeling and an adaptive attack framework that explicitly exploits knowledge of Γ.
How does the paper address this problem?
First formalize the threat model, then propose the attacking model space principle and the greedy instantiation GMSA.
Overall approach: first model the transductive defense as an attacker–defender game and point out the modeling details, then relax the attack objective to attacking a family of models (model space), solve it with a greedy algorithm, and obtain the attack framework Greedy Model Space Attack (GMSA).
Formalizing the threat model
- 1-round game (Definition 1): the defender trains the base model F=T(D); the attacker receives the clean test set V and produces V′=A(Γ,F,D,V) (V′∈N(V)), and sends U′=V′|X; the defender produces F*=Γ(F,D,U′), which is evaluated by the referee on U′.
- Four details of the multi-round extension: the randomness that Γ samples after the attacker's action is regarded as private randomness; the defender's state across rounds is divided into states-leaking and non-leaking; attacks that explicitly exploit knowledge of Γ are called adaptive; batch order is divided into naturally ordered and adversarially ordered, and the authors point out that prior work considered only the former, with the main experiments focusing on the former and the latter evaluated only for DENT in Appendix A.7.
- Modeling capacity: without Γ it degenerates to the classical inductive threat model; it also covers test-time "purification" defenses broken by BPDA (Γ is a non-differentiable purification function that does not update the model, so it does not count as transductive-learning based).
Attack objective and difficulties
- The attack objective is maxV′ ∈ N(V), U′=V′|X La(Γ(U′), V′), i.e., the attacker seeks a perturbed test batch on which the model adapted by Γ has maximum loss; this is a bilevel optimization (Equation (4)).
- Three difficulties (listed by the authors): (D1) Γ is non-differentiable, and the authors state that existing theory such as NTK does not apply to transductive learning that handles unlabeled data, and BPDA does not apply either; (D2) U′ appears on both the attack and defense sides, making it hard to converge to a U* that can defeat the model it itself induces; (D3) Γ(U′) can be a random variable over models due to private randomness.
- FPA (Algorithm 1): each round fixes the previous round's model F(i)=Γ(U(i)) and constructs U(i+1) to defeat it; after T rounds the best attack set is selected; the authors state that FPA is weak in experiments, precisely because of (D2); transfer attack is a special case with T=0.
Attacking model space principle and GMSA
- Robust objective: exploiting the transferability of adversarial examples, it changes to maxU′ minU ∈ N(U′) La(Γ(U), U′), i.e., finding an attack set that defeats the family of models induced by the neighborhood of U′; the authors state that this relaxes the attacker-defender constraint (D2) and provides more information for handling non-differentiability (D1).
- GMSA (Algorithm 2): round i attacks all historical models {F(j)}(j=0..i), with the loss taking two instantiations—LAVG takes the average of the attack losses over the models (GMSA-AVG), LMIN takes the minimum (GMSA-MIN); solved with PGD, and after T rounds the attack set that maximizes L(F(i+1),V(i)) is selected.
- Implementation details (Appendix A.1.3): GMSA-AVG backpropagates sequentially through each historical model at each PGD step and accumulates gradients; GMSA-MIN scales the number of PGD steps in round i by a factor of i+1.
Multi-round and evaluation method
- When the defense is stateless, the 1-round attack is repeated round by round; when stateful, it is divided into leaking and non-leaking, and the experiments in Section 6 evaluate only non-leaking (harder for the attacker).
- Success criterion: reports accuracy (performance on clean test inputs) and robustness (performance under attack), both as percentages; for transductive defenses, robustness is estimated with AA, PGD, FPA, GMSA-AVG, GMSA-MIN, where AA and PGD are in the transfer setting (attacking the static base model to generate samples and then evaluating the defense); static models are always estimated with AutoAttack.
What experiments did the paper conduct?
Five defenses are evaluated on MNIST/CIFAR-10/GTSRB; the authors state that GMSA breaks RMC, DENT, and URejectron, while TADV instead improves robustness.
Experimental setup
- Defenses under test and base models: RMC (base models Standard and Madry et al.; a two-layer convolutional network for MNIST and ResNet-32 for CIFAR-10); DENT+ (CIFAR-10, base models taken from the RobustBench Model Zoo: Wu et al. (2020a), Carmon et al. (2019), Sehwag et al. (2020), Wang et al. (2020), Hendrycks et al. (2019), Wong et al. (2020), Ding et al. (2020)); DANN (Chuang et al. (2020) architecture for MNIST, ResNet-18 for CIFAR-10); TADV (LeNet for MNIST, ResNet-20 for CIFAR-10); URejectron (GTSRB, ResNet18).
- Datasets: MNIST, CIFAR-10, GTSRB (Appendix A.1.2: MNIST 5,500 training/1,000 test per class; CIFAR-10 5,000 training/1,000 test per class; GTSRB about 34,799 training and 12,630 test, augmented to 10,000 per class).
- Threat model: L∞, ε=0.3 for MNIST; L∞, ε=8/255 for CIFAR-10 and GTSRB (PGD); the defender uses private randomness different from the attacker's; RMC is evaluated in the non-state leaking setting.
- Attacks and iteration parameters: AA, PGD (transfer setting), FPA, GMSA-AVG, GMSA-MIN; T=9 for RMC/DANN/TADV and T=2 for DENT; DENT attacks use 14 PGD subroutines (5 untargeted CW loss with different random restarts + 9 targeted CW loss with different target labels) and take the best.
- Sample size and repetitions: to save computation, RMC and DENT are each evaluated on n=1000 randomly sampled test points; RMC additionally runs 5 random runs and reports mean±std (Table 6); URejectron is evaluated on x̃ consisting of 30,000 test images (50% normal + 50% adversarial).
- Metrics: accuracy and robustness (percentages); URejectron reports err (error rate on samples that pass S) and rej (rejection rate on clean inputs).
Main results
According to Table 1 (RMC, robustness column, %):
Swipe horizontally to read the table
Dataset Base model Static AA RMC AA RMC PGD RMC FPA GMSA-AVG GMSA-MIN MNIST Standard 0.00 97.70 98.30 0.60 0.50 1.10 MNIST Madry et al. 87.70 95.70 96.10 59.50 61.40 58.80 CIFAR-10 Standard 0.00 94.20 97.60 8.50 8.00 8.10 CIFAR-10 Madry et al. 44.30 77.90 71.70 40.80 42.50 39.60 - The authors state that RMC with the Standard base model is already broken by the weaker FPA; with an adversarially trained base model, RMC's robustness is instead lower than adversarial training alone (Static AA column).
- The authors state that the gap between the defense-unaware AutoAttack and GMSA is large, indicating that AA overestimates RMC.
DENT and DANN
- DENT (Table 2, CIFAR-10): robustness under DENT-AA is 62.80%–80.10%, whereas under the GMSA variants it is close to PGD and FPA, e.g., with the Wu et al. (2020a) base model GMSA-AVG is 59.60% (Static AA 58.00%), and with the Ding et al. (2020) base model GMSA-AVG is 39.10% (Static AA 39.70%). The authors state that both DENT-AA and AA overestimate DENT, and that PGD can already reduce DENT to almost the same as the static defense.
- DENT exception: the authors state that under adaptive attack DENT retains only a minor improvement over adversarial training (the introduction says less than 3%); in the adversarially-ordered game in Appendix A.7 (Table 5), DENT robustness is reduced below the static base model, e.g., in the Ding et al. (2020) row GMSA-MIN 35.00% versus Static AA 39.70%.
- DANN (Table 3): the authors state that DANN has non-trivial robustness under AA, PGD, and FPA (e.g., on MNIST AA 97.59%, FPA 96.81%), but is very low under GMSA: MNIST GMSA-MIN 6.17%, CIFAR-10 GMSA-AVG 5.53%.
TADV and URejectron
- TADV (Table 4): the attacker attacks an ensemble of T=10 adversarially trained models with independent randomness, then checks whether U′ transfers to a model trained with entirely fresh randomness. On MNIST robustness rises from 86.61% for Madry et al. (Static AA) to 94.27% under GMSA-AVG; on CIFAR-10 it rises from 45.29% to 54.12%. The authors call this phenomenon "somewhat surprising": U′ transfers poorly, and TADV improves robustness against all considered attacks.
- URejectron (Figure 1, GTSRB): (a) under PGD transfer attack the authors state that URejectron works as expected; (b) after switching to CW attack (L2, classifier robustness 0.00%), the authors state that adversarial examples can be constructed that closely resemble clean inputs, pass through S, and cause large errors; (c) benign perturbations generated with brightness corruption (severity 1, classifier accuracy 98.90%) are all rejected. The authors state that this exposes two failure modes: imperceptible perturbations slip through, and benign perturbations are rejected.
Other ablations and analyses
- Five random runs of RMC (Table 6): MNIST Standard GMSA-AVG 0.54±0.05, CIFAR-10 Standard GMSA-AVG 8.50±1.29; the authors state that the conclusions are unchanged.
- DENT adaptation hyperparameters (Appendix A.3): AdaMod six steps, learning rate 0.006, batch size 128, no weight decay, updating only BN affine parameters.
- URejectron implementation (Appendix A.6): T=1 iteration, train/validation/test split of 63,000/7,000/30,000, classifier training-set accuracy 99.90% and validation-set 99.63%.
What are some points that can be further explored?
The authors state that future work could explore defenses that resist GMSA and stronger adaptive attacks; the experiments cover three datasets and five defenses.
Limitations and future directions noted by the authors
- More robust defenses: The Conclusion states that future work could explore transductive defenses that remain robust under GMSA attacks.
- Stronger attacks: The Conclusion states that future work could explore stronger adaptive attacks that can effectively evaluate transductive defenses.
- FPA is weak: Section 5.1 states that experiments found FPA to be weak, because U(i+1) defeating F(i) does not show that it can defeat the F(i+1) it induces itself.
- Existence of non-transferable attacks: Section 5.2 states that a non-transferable "brittle" attack set may theoretically exist, but identifying it algorithmically is challenging, and the robust objective is only a natural relaxation.
- The paper has no dedicated Limitations section; all of the above comes from the Conclusion and Section 5.
Scope of experiments
- The defenses tested are five: RMC, DENT, DANN, TADV, and URejectron; the datasets are MNIST, CIFAR-10, and GTSRB; the perturbation type is L∞ (ε=0.3 or 8/255), and URejectron is additionally tested with L2 CW attacks and brightness corruption.
- The multi-round game is evaluated only under the non-leaking setting (Section 5 states this is harder for the attacker); the adversarially-ordered game is evaluated only for DENT (Appendix A.7).
- RMC and DENT are each evaluated on 1000 randomly sampled test points; RMC additionally has mean±std over 5 random runs (Table 6), and the paper does not report the number of repetitions for the other defenses.
- The DENT experiments evaluate only CIFAR-10, with base models being 7 pretrained robust models from the RobustBench Model Zoo; TADV is compared only with Madry et al. adversarial training.
- The paper does not report the number of queries or time cost for each attack; the iteration budget for GMSA is T=9 (RMC/DANN/TADV) and T=2 (DENT).
Summarize the main content of the paper
Threat modeling + GMSA attack framework: the authors state that the robustness of existing transductive defenses is overestimated, with TADV being the exception.
A paper on threat analysis and adaptive attacks against transductive-learning based defenses (ICLR 2022). Such defenses dynamically update the model at test time using a batch of inputs; the authors argue that their attack objective is a bilevel optimization, and that prior evaluations with inductive attacks overestimated robustness.
- Threat model: formalizes the 1-round and multi-round games, clarifying four previously unclear modeling details: private randomness, states-leaking versus non-leaking, adaptive versus non-adaptive, and naturally versus adversarially ordered.
- Attack principle: exploiting the transferability of adversarial examples, relaxes the objective to attacking the model space induced by the neighborhood of U′; the greedy instantiation GMSA (two losses, AVG/MIN, solved with PGD) serves as a new baseline for evaluating such defenses.
- RMC: with the Standard base model, GMSA-AVG reduces robustness from 97.70% under AA to 0.50% (MNIST) and from 94.20% to 8.00% (CIFAR-10) (Table 1).
- DENT: on CIFAR-10 the GMSA variants reduce robustness to almost the same as the static base model; the authors state that the improvement over adversarial training is less than 3%, whereas the original source reported more than 20% absolute improvement against AutoAttack; under the adversarially-ordered setting it can fall below the static base model (Table 5).
- DANN and URejectron: DANN has non-trivial robustness under AA/PGD/FPA but is broken by GMSA (MNIST GMSA-MIN 6.17%); URejectron exhibits two failure modes, slipping through and false rejection, under CW attack and image corruption respectively (Figure 1).
- TADV: retraining adversarially at test time with fresh private randomness raises MNIST robustness from 86.61% to 94.27% and CIFAR-10 from 45.29% to 54.12% (GMSA-AVG, Table 4); the authors call this result somewhat surprising.
The authors' conclusion: GMSA should serve as a new baseline for evaluating transductive defenses; existing defenses are not robust under its attack, and transductive adversarial training that exploits fresh randomness at test time is a direction worth exploring.
Related papers
Show 3 more papers
- On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial ExamplesAnalysis / interpretabilityarXiv