Faster than free: revisiting adversarial training

1. 01What problem does this paper attempt to solve?

   Adversarial training is too slow: the authors state that FGSM plus random initialization can achieve the effect of PGD training at a cost close to standard training.

   Core problem: adversarial training typically takes an order of magnitude longer than standard training because the inner loop requires multi-step attacks such as PGD to generate adversarial examples; the authors attempt to eliminate this extra cost.

   • Problem setting and importance: safety-critical applications need deep networks robust to adversarial examples; the authors say adversarial training and certified defenses are among the most reliable methods, but both incur substantial computational overhead, slowing the scaling of robustness research to large networks and rapid iteration.
   • Shortcomings of existing methods: the authors say recent acceleration work (free adversarial training, Shafahi et al., 2019; YOPO, Zhang et al., 2019) improves things but is still markedly slower than standard training—free training requires over 200 epochs and more than 10 hours on CIFAR10, and two days on ImageNet.
   • Core observation and hypothesis: the key difference between free training and FGSM training is that the perturbation's initial point is non-zero (reusing the previous minibatch's perturbation); the authors hypothesize that the main source of effectiveness is starting from a non-zero initial perturbation, independent of the specific initialization method.
   • Method proposed in the paper: FGSM adversarial training with uniform random initialization (Algorithm 3), combined with DAWNBench's standard acceleration techniques such as cyclic learning rate and mixed-precision; and identification of the "catastrophic overfitting" failure mode that caused previous FGSM training failures.
   • Threat model:
     • Defense goal: learn a classifier robust to adversarial perturbations, formalized as a robust optimization problem (Equation 1).
     • Threat model: l∞ perturbation ∆={δ:‖δ‖∞≤ε}, following the setting of Madry et al. (2017); ε=8/255 for CIFAR10, 2/255 and 4/255 for ImageNet, 0.3 for MNIST.
     • Attacker capability: at evaluation, PGD attacks (50 steps, step size 2/255, 10 random restarts) perturb inputs to fool the model.
     • Victim systems: MNIST/CIFAR10/ImageNet image classifiers (convolutional networks, PreAct ResNet18, ResNet50).

   Collapse this answer

2. 02What related research exists?

   Organized along the line of FGSM→PGD→free training, and also covers other defenses, strong attack evaluations, and DAWNBench fast training techniques.

   Evolution of adversarial training

   • Goodfellow et al. (2014)—proposed FGSM, using a single-step gradient to generate adversarial examples, and was the first to use it for adversarial training.
   • Tramèr et al. (2017)—proposed R+FGSM, adding a randomization step to FGSM; the paper notes that its initialization and step size are more restricted, and the resulting model cannot withstand full-strength PGD attacks (detailed comparison in Appendix A).
   • Kurakin et al. (2016) and Madry et al. (2017)—the Basic Iterative Method improves FGSM with multiple small steps, rendering FGSM-family adversarial training ineffective; with random restarts it forms what is now widely known as PGD adversarial training, considered an effective method for learning robust networks.

   Work on reducing the cost of adversarial training

   • Shafahi et al. (2019)—"free" adversarial training: a single backward pass updates both perturbation and weights simultaneously, combined with minibatch replay, with cost close to standard training; the main comparison target of this paper.
   • Zhang et al. (2019)—removes redundant computation during multi-step PGD backpropagation to accelerate.
   • Wang (2018)—found that larger but fewer PGD steps do not necessarily significantly change final robustness.

   Other defenses and strong-attack evaluation

   • Other defense directions: preprocessing (Guo et al., 2017, etc.), detection (Metzen et al., 2017, etc.), verification and certifiable defenses (Katz et al., 2017; Wong & Kolter, 2017, etc.); the authors state that certified defenses have been extended to networks of considerable scale, but the guarantees fall short of the empirical robustness of adversarial training.
   • Strong-attack evaluation: the authors state that weak attacks give a misleading sense of security, and historically many defenses (Papernot et al., 2016, etc.) were broken by stronger attacks (Carlini & Wagner, 2017b; Athalye et al., 2018, etc.), and Carlini et al. (2019) proposed best practices for evaluation.

   Fast standard training techniques

   • DAWNBench competition (Coleman et al., 2017)—top submissions can train CIFAR10/ImageNet in minutes/hours; common techniques include cyclic learning rate (Smith & Topin, 2018) and mixed-precision (Micikevicius et al., 2017).

   Baselines and benchmarks: The comparison baselines are PGD-7 adversarial training (Madry et al., 2017) and free adversarial training (Shafahi et al., 2019), with benchmarks on MNIST, CIFAR10, ImageNet.

   The authors position this paper as: returning to the earliest-rejected FGSM adversarial training, showing that with random initialization it is equivalent to PGD training, and converges faster with lower total cost than free training.

   Collapse this answer

3. 03How does the paper address this problem?

   FGSM adversarial training plus uniform random initialization (α=1.25ε), further combined with cyclic learning rate and mixed-precision acceleration.

   Overall approach: use FGSM with random initialization to approximate the inner maximization of robust optimization (costing only 2× standard training per epoch), then use DAWNBench's general acceleration techniques to reduce the number of epochs and per-step time, bringing total training cost close to standard training.

   Problem setting and formalization

   • Robust optimization objective: min_θ ∑_i max_{δ ∈ Δ} ℓ(f_θ(x_i+δ), y_i), i.e., minimize the worst-case loss within the threat model ∆.
   • The threat model is Δ={δ:|δ|_∞≤ϵ} (l∞ threat model), and the closed-form approximation of FGSM to the inner maximization is δ^∗=ϵ·sign(∇_x ℓ(f(x),y)).
   • PGD training requires O(MN) gradient computations per epoch (N is the number of PGD steps), about N times standard training; FGSM training requires 2 backward passes per epoch (perturbation and weights computed separately), equivalent to 2 standard training epochs.

   Core method: FGSM adversarial training with random initialization (Algorithm 3)

   • Procedure: initialize perturbation δ as Uniform(−ε, ε), take one step α·sign(∇δℓ), clip back to [−ε, ε], then update weights with this adversarial example.
   • Key design: the authors hypothesize that the main reason free training works is starting from a non-zero initial perturbation; in the experiments, all FGSM training uses random initialization with step size α=1.25ε (i.e., α=10/255 on CIFAR10).
   • Two differences from Tramèr et al. (2017)'s R+FGSM (Appendix A): initialization uses a uniform distribution over the entire perturbation domain (theirs initializes on the surface of a hypercube of radius ε/2), and step size uses α=ε (theirs uses ε/2).

   DAWNBench acceleration techniques

   • Cyclic learning rate: learning rate first rises linearly from 0 to a maximum λ (first N/2 epochs), then decreases linearly back to 0; λ is tuned per method to the maximum value that does not diverge. The ImageNet cyclic schedule decays by another 10× in the second and third stages (Figure 1).
   • Mixed-precision: uses Apex amp, O1 for ImageNet, O2 for CIFAR10 without loss scaling.
   • Additional ImageNet techniques: no weight decay on batch normalization layers; progressive image size scaling, with training in three stages (160 pixels, 352 pixels, full image).

   Failure mode and early stopping

   • The authors call the failure caused by zero initialization or too large a step size (e.g., α=2ε=16/255) "catastrophic overfitting": robust accuracy against PGD attacks (on training data) suddenly drops to 0%.
   • Detection and remedy: at the end of each epoch, run PGD with 5 steps and 1 restart on a single training minibatch; if robust accuracy drops sharply, stop early, which can recover some robustness (Table 1, Appendix D).

   Success criteria and evaluation method

   • Empirical robustness: robust accuracy is measured with a 50-step PGD attack with step size 2/255 and 10 random restarts (hyperparameters consistent with Shafahi et al. (2019) with strengthened random restarts).
   • Exact verification: on small MNIST models, MILP (Tjeng et al., 2017) is used to compute the exact verified robust rate at ε=0.1.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On CIFAR10, FGSM+random initialization reaches 46.06% PGD robust rate, training to 45% in 6 minutes; ImageNet 43.43%, 12 hours; MNIST verified robust rate 96.8%.

   Experimental setup

   • Models tested: CIFAR10 uses PreAct ResNet18 (single RTX 2080ti); ImageNet uses ResNet50 (4 RTX 2080ti); MNIST uses a small network with two convolutional layers (16, 32 filters) plus a 100-unit fully connected layer (same as Tjeng et al., 2017).
   • Datasets: MNIST, CIFAR10, ImageNet; the paper does not state the sample size of each dataset.
   • Baselines: PGD-7 adversarial training (Madry et al., 2017) and free adversarial training (Shafahi et al., 2019, m=8 or m=4).
   • Metrics: standard accuracy, PGD robust accuracy (50 steps, α=2/255, 10 random restarts; ImageNet additionally reports 1 restart), MILP-verified robust rate on MNIST, training time.
   • Training parameters: batch size 128, SGD momentum 0.9, weight decay 5×10⁻⁴; Table 1 experiments report the average of 3 random seeds; each point in Figure 2 is the average of 3 independent runs.
   • FGSM training configuration: random initialization + α=1.25ε; in Table 7, the epoch counts on CIFAR10 are FGSM 30, PGD 40, Free 96, with maximum learning rates 0.2/0.2/0.04.

   Main results

   CIFAR10, ε=8/255, standard/robust accuracy and training time of various adversarial training methods (according to Table 1):

   Swipe horizontally to read the table

   Method	Standard acc.	PGD (ε=8/255)	Time (min)
   FGSM + zero init	85.18%	0.00%	12.37
   FGSM + zero init + early stopping	71.14%	38.86%	7.89
   FGSM + previous init	86.02%	42.37%	12.21
   FGSM + random init	85.32%	44.01%	12.33
   FGSM + random init + α=10/255	83.81%	46.06%	12.17
   FGSM + random init + α=16/255	86.05%	0.00%	12.06
   Free (m=8) + DAWNBench	78.38%	46.18%	20.91
   PGD-7 + DAWNBench	82.46%	50.69%	68.8

   • The authors state that zero initialization is not robust, while random initialization or reusing the previous minibatch's perturbation both achieve robustness comparable to free and PGD training, with random initialization slightly better.
   • The authors state that increasing the step size to α=10/255 matches the best reported result of free training; α=16/255 (=2ε) causes catastrophic overfitting.
   • The original rows for Free (m=8) and PGD-7 in the table (46.33%/785 min, 45.80%/4965.71 min) are the values reported in the original papers under different network architectures and a weaker attacker (Table 1 footnote).

   MNIST exact verification

   • What was measured: small convolutional networks trained with FGSM and PGD adversarial training (ε=0.3, PGD training uses 40 steps, step size 0.01), with exact robust rate computed via MILP.
   • Results (Table 2): FGSM-trained model standard 99.20%, PGD(0.1) 97.53%, PGD(0.3) 88.77%, verified(0.1) 96.8%; PGD-trained model correspondingly 99.20%, 97.66%, 89.90%, 96.7%.
   • The authors state that FGSM training on MNIST shows almost no difference from PGD training in empirical and verified robustness; ε=0.3 and larger datasets cannot be exactly verified due to the MILP scale being too large.

   Fast CIFAR10 and ImageNet training

   • CIFAR10 (Table 3): DAWNBench+FGSM trains to 45% robust accuracy with 15 epochs, 6.34 minutes; DAWNBench+PGD-7 is 10 epochs, 17.49 minutes; DAWNBench+Free(m=8) is 80 epochs, 17.44 minutes. The authors state that free training benefits less from cyclic learning rate and requires more epochs.
   • ImageNet (Table 4, ε=2/255): FGSM reaches standard 60.90%, PGD+10 restarts 43.43% in 12.14 hours; Free(m=4) reaches 64.37%, 43.28% in 52.20 hours. At ε=4/255, FGSM is 55.45%, 30.18%, and Free(m=4) is 60.42%, 31.08%.
   • ImageNet timing (Table 5): FGSM mixed-precision 15 epochs totals 12.14 hours (three stages 2.01+5.34+4.80); Free(m=4) mixed-precision 92 epochs totals 38.76 hours. The authors state that mixed-precision speeds up free training by 25%, but it is still slower than FGSM.

   Other ablations and analysis

   • R+FGSM comparison (Table 6, MNIST, 10 random seeds): this paper's version (Uniform(0.3) initialization + step size 0.30) has robust rate 86.21±0.75%; original R+FGSM 34.58±36.06%; uniform initialization only 72.92±10.40%; full step size only 26.53±32.48%.
   • Step size sweep (Figure 3, CIFAR10, 30 epochs, 3 seeds): robust rate rises with step size up to α=10/255 and then no longer improves; larger step sizes tend to overfit to the perturbation domain boundary.
   • Catastrophic overfitting (Figure 4): during zero-initialization FGSM training, PGD test error suddenly rises while FGSM training error stays low; Figure 5 shows that the absolute values of PGD perturbations for the overfitted model fall more in the interval near the origin, while the robust model's perturbations are closer to the boundary (near 8).
   • Early stopping remedy (Figure 6): step sizes greater than 11/255 originally have 0% robust rate, and early stopping can recover part of the robust accuracy.
   • Free+DAWNBench on ImageNet (Table 9, Appendix F): with m=3, 30 epochs (equal compute to FGSM 15 epochs), PGD+10 is 27.08%–27.81%, lower than the original Free(m=4)'s 31.08%; the authors state this is not a claim that the two are incompatible, only that free training was not tuned for.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that exact verification cannot scale to large datasets, and propose a theoretical direction for explaining when approximate inner solutions suffice; experiments cover three image benchmarks.

   Limitations and future directions noted by the authors

   • Scalability of exact verification: on MNIST at ε=0.3 and on larger datasets, the MILP is too large to solve in reasonable time, and robustness of larger networks can only be assessed by empirical testing (Section 5.1, footnote 4).
   • Theoretical direction: the authors say this paper exposes a new direction—explaining more rigorously when approximate solutions to the inner optimization problem suffice for robust optimization and when they fail (Conclusion).
   • Combining free training with DAWNBench: the authors say they cannot claim that free training and DAWNBench techniques are incompatible on ImageNet, and that more epochs might recover performance, but tuning for free training is not the goal of this paper (Appendix F).
   • The authors believe free training on ImageNet may need more than 15 epochs (30 epochs at equal compute) to reach best performance (Appendix F).

   Scope of experiments

   • 3 models tested: PreAct ResNet18 (CIFAR10), ResNet50 (ImageNet), and a small convolutional network (MNIST).
   • Benchmarks are MNIST, CIFAR10, ImageNet; threat model is only l∞, with ε of 0.1/0.3 (MNIST), 8/255 (CIFAR10), 2/255 and 4/255 (ImageNet).
   • Evaluation attack is PGD (50 steps, 10 random restarts; ImageNet additionally reports 1 restart), and MNIST additionally has MILP exact verification (only ε=0.1); the paper does not report attack evaluations other than PGD.
   • Number of repetitions: Table 1 and Figures 2/3 use 3 random seeds, Table 6 uses 10 random seeds; the paper does not report the number of repetitions for ImageNet experiments.
   • Comparison baselines are PGD-7 (Madry et al., 2017) and free adversarial training (Shafahi et al., 2019); the paper does not report runtime comparisons with certified defense training methods.

   Collapse this answer

6. 06Summarize the main content of the paper

   FGSM adversarial training plus random initialization can match PGD training: training a robust classifier on CIFAR10 in six minutes and on ImageNet in twelve hours.

   An ICLR 2020 paper on accelerating adversarial training, addressing the problem that: adversarial training is about an order of magnitude slower than standard training because the inner loop requires multi-step PGD attacks, while the cheapest FGSM adversarial training was previously considered ineffective against PGD attacks.

   • Method highlights: The authors found that the key difference between free training and FGSM training is the non-zero initial perturbation, so they add a uniform random initialization to the FGSM perturbation (step size α=1.25ε); combined with DAWNBench's cyclic learning rate and mixed-precision (ImageNet additionally uses progressive scaling and removing weight decay from BN layers) to compress training time.
   • CIFAR10: FGSM+random initialization+α=10/255 reaches 46.06% PGD robust accuracy (Table 1); with 15 epochs, trains to 45% robust accuracy in 6.34 minutes, whereas the original PGD-7 implementation requires 4965.71 minutes and free training requires 674.39 minutes (Table 3).
   • ImageNet: At ε=2/255, FGSM reaches 43.43% in 12.14 hours (PGD+10 restarts), free(m=4) requires 52.20 hours to reach 43.28% (Tables 4, 5).
   • MNIST verification: The FGSM-trained model has a MILP-verified robust rate of 96.8% at ε=0.1, versus 96.7% for PGD training (Table 2).
   • Failure mode: Zero initialization or α=2ε triggers "catastrophic overfitting", with PGD robust accuracy dropping to 0.00% (Table 1); this can be detected with 5-step PGD on a single minibatch and recovered via early stopping.
   • Authors' conclusion: In robust optimization, the inner maximization only needs a rough approximation, and defenders do not need a strong attacker during training; the authors believe this will make robustness research iterate faster, and propose a theoretical direction for explaining when an approximate inner solution is sufficient.

   Collapse this answer

Related papers