Distributed Adversarial Training: Large-Scale Enhancement of Deep Neural Network Robustness
Distributed Adversarial Training to Robustify Deep Neural Networks at Scale
The authors propose a distributed adversarial training framework DAT, combining LALR to solve large-batch min-max optimization; on ImageNet/ResNet-50, RA is comparable to centralized AT and training is faster.
Defender scenario: train robust DNNs against l∞-norm-bounded (radius ε) adversarial perturbations; evaluation attacks are PGD (CIFAR 20 steps, ImageNet 10 steps) and AutoAttack, training uses PGD/FGSM to generate perturbations; victim systems are image classifiers (ResNet-18/50, Wide ResNet-28-10).
- Adversarial training (AT) is computationally expensive and hard to scale; directly scaling the batch to distributed multi-machine setups noticeably degrades robust and standard accuracy (Figure 1), and the paper addresses how to scale robust training in a distributed environment.
- Proposes a distributed adversarial training framework DAT: under a parameter server architecture, each worker generates perturbations with PGD or FGSM, with optional gradient quantization, the server aggregates gradients, and layerwise adaptive learning rates (LALR/LAMB) are used to handle large-batch min-max optimization; theoretically gives an O(1/√T) convergence rate in the non-convex setting.
- On ImageNet/ResNet-50, DAT-PGD's RA of 38.45% is close to centralized AT's 40.38%, and per-epoch time drops from 6022 seconds to 1960 seconds; effectiveness is reported in semi-supervised, randomized smoothing certified robustness, and pretraining fine-tuning scenarios; 8-bit one-sided quantization reduces RA by only 0.55%.
- The authors state that future work is worth examining for acceleration effects under extreme settings (more attack steps, more machines, larger models). The experiments are limited to image classification and l∞ perturbations; the RA variance of DAT-FGSM is about 1.5%, higher than the about 0.5% of DAT-PGD; the paper does not report further evaluations related to adaptive attacks.
- Evaluated models
- ResNet-50Pre-act ResNet-18Wide ResNet-28-10
- Benchmarks
- ImageNetCIFAR-10CIFAR-10080 Million Tiny Images
- Metrics
- TARAAAcertified accuracycommunication time per epochtraining time per epoch
The researchers propose distributed adversarial training (DAT), a multi-machine large-batch adversarial training framework that extends robust training to large-scale models and datasets. DAT supports labeled and unlabeled data, multiple attack generation methods, and gradient compression, and under standard conditions gives a rate of convergence to first-order stationary points in the non-convex case. Experiments show that its robust accuracy matches or exceeds the current best level, and it achieves good training speedup on ResNet-50 on ImageNet.
Paper analysis
What problem does this paper attempt to solve?
Solves the problem that adversarial training is hard to scale to distributed multi-machine: directly enlarging the batch loses accuracy, so the authors propose the DAT framework.
Core problem: adversarial training (AT) is computationally intensive and hard to scale, and directly moving it to a distributed large-batch setting causes significant accuracy loss; how to design a distributed adversarial training framework with theoretical guarantees.
- Problem setting and importance: The authors state DNNs are easily manipulated by adversarial perturbations, and AT is currently the mainstream robust training method, but it is far more computationally intensive than standard training and hard to scale; when data cannot be stored centrally due to scale or privacy, or when large batches are needed to improve training scalability, distributed AT is needed.
- Shortcomings of existing methods: The authors state Fast AT has problems such as unstable robust learning, excessive sensitivity to learning rate schedules, and catastrophic forgetting against strong attacks (citing Li et al., 2020; Rice et al., 2020; Andriushchenko and Flammarion, 2020); the few existing works on multi-machine AT scaling (Xie et al., 2019 et al.) are limited to specific use cases and lack systematic theoretical and practical study.
- Core observation: Figure 1 shows that on ImageNet/ResNet-50, as the number of nodes and batch size scale from 1×128 to 6×512, both RA and TA of regular AT drop substantially (per the authors), indicating naive distributed scaling is infeasible.
- What the paper proposes: Proposes the DAT (distributed adversarial training) large-batch adversarial training framework, supporting labeled/unlabeled data, multiple attack generators and gradient compression; theoretically gives an O(1/√T) rate of convergence to a first-order stationary point in the non-convex setting; experiments cover ImageNet robust training, randomized smoothing, semi-supervised, pretraining+fine-tuning and multiple communication protocols.
- Threat model:
- Attacker goal: Add l∞-norm-bounded (radius ε) perturbations to inputs to change the classification result.
- Knowledge and capability: Evaluation uses PGD (CIFAR 20 steps, ImageNet 10 steps) and AutoAttack; training uses PGD or single-step FGSM to generate perturbations.
- Victim system: Image classification DNNs (ResNet-18/50, Wide ResNet-28-10), with the defender being the trainer.
What related research exists?
Related work falls into three groups: AT and its accelerated variants, distributed large-batch training and communication compression, and distributed min-max optimization; the authors state that this paper is the first to systematically study large-batch distributed min-max robust training.
Robust classifier training
- AT (Madry et al., 2017): The first defense based on min-max optimization, and the object this paper aims to distribute; it has spawned TRADES (Zhang et al., 2019b), distributionally robust training (Sinha et al., 2018), semi-supervised robust training (Carmon et al., 2019; Stanforth et al., 2019), etc.
- Fast AT variants: Free AT (Shafahi et al., 2019), YOPO (Zhang et al., 2019a), FGSM-based Fast AT (Wong et al., 2020) speed up training by simplifying the inner maximization, but the authors point out they are designed for centralized training, and Fast AT has stability issues.
- Attempts to scale AT across machines: Xie et al., 2019, Kang et al., 2019, Qin et al., 2019 did empirical scaling, which the authors criticize as limited to specific use cases and lacking theoretical and practical study of "when and why distribution helps".
Distributed model training
- Large-batch training: Dean et al., 2012, Goyal et al., 2017 et al. demonstrated that distributed optimization is effective for standard training; You et al., 2017b/2018/2019 proposed adjusting the learning rate as batch grows and layerwise adaptive learning rates (LARS/LAMB), and the authors state this paper brings this idea into DAT.
- Communication-efficient optimization: QSGD (Alistarh et al., 2017), double quantization (Yu et al., 2019), signSGD (Bernstein et al., 2018) et al. use gradient quantization/sparsification to reduce communication overhead; this paper's quantizer is taken from Alistarh et al., 2017 and Yu et al., 2019.
Distributed min-max optimization
- Srivastava et al., 2011, Tsaknakis et al., 2020, Liu et al., 2019a/b et al. studied distributed min-max or decentralized GAN training; the authors point out that none of these works studied large-batch min-max optimization and its application to robust DNN training, and that robust training requires inner maximization per sample, which is fundamentally different from GAN's empirical maximization.
Baselines and benchmarks: Experimental baselines are AT, Fast AT, DAT w/o LALR, DAT-LSGD (Xie et al., 2019); datasets are ImageNet, CIFAR-10/100, 80 Million Tiny Images; also compared with the reported values of Carmon et al., 2019 on RobustBench (Croce et al., 2020).
Positioning: The authors state this paper provides a general DAT framework and proves under standard optimization conditions that its convergence rate matches the classical rate of minimization-only SGD, which is the previously missing large-batch min-max convergence analysis.
How does the paper address this problem?
DAT combines sample-level attack generation, optional gradient quantization, and layer-wise adaptive learning rate (LALR) under a parameter-server framework, and provides a non-convex convergence analysis.
Overall approach: write AT's min-max problem as a distributed robust objective across M workers, decompose the computation with a parameter server architecture, then use LALR (layerwise adaptive learning rate) to solve the min-max optimization difficulty brought by large batches, and support gradient quantization to reduce communication overhead.
Problem setting and formalization
- AT review: Standard AT solves minθ 𝔼(x,y) max|δ|∞ ≤ ϵ ℓ(θ, x+δ; y), i.e., the loss under the worst perturbation for each sample.
- DAT objective: M workers each hold local data D(i), minimizing (1/M)∑i fi, where fi = 𝔼[λ ℓ(θ;x,y) + max|δ|∞≤ϵ ϕ(θ,δ;x,y)]; λ≥0 balances the training loss and the robust regularization term. When M=1, λ=0, φ=ℓ it reduces to AT.
- Two settings: With labeled data, φ is the CE loss on perturbed samples; with unlabeled data, it uses the pseudo-label worst CE loss (Carmon et al., 2019) or the TRADES regularizer (Stanforth et al., 2019).
Algorithm framework (Algorithm 1 / A1)
- Block 1 (worker): Each worker calls the inner maximization oracle on each sample in the batch to generate perturbations, computes the local gradient, optionally quantizes it and sends it to the server.
- Block 2 (server): The server aggregates the workers' gradients (optionally with secondary quantization) and broadcasts.
- Block 3 (worker): Each worker updates the model with the outer minimization oracle (LAMB) using the received gradients; All-reduce is a special case where each worker also acts as server.
Key designs
- LALR: The update scales θ layerwise, τ(‖θ_i‖)=min{max{‖θ_i‖, c_l}, c_u}, with c_l=0, c_u=10 in experiments; the authors state the layerwise adaptive learning rate smooths the optimization trajectory so that a large learning rate under large-batch min-max does not fall into sharp optima. u_t is given by Adam/LAMB-style momentum.
- Two inner oracle choices: Iterative PGD or single-step FGSM; FGSM is chosen because it is computationally lightest, and the paper states other simplification techniques could also be used but are not elaborated.
- Gradient quantization: Uses a stochastic quantizer (Alistarh et al., 2017; Yu et al., 2019), b-bit quantization, supporting one-sided (Ring-AllReduce) and two-sided (parameter server) quantization.
Convergence analysis
- Assumptions: A1 layerwise Lipschitz gradient; A2 φ is strongly concave in δ with Lipschitz gradient (the authors state this can be argued via Sinha et al., 2018 et al.); A3 stochastic gradients have unbounded bias and bounded variance σ²; the appendix additionally has A4 bounded quantization error variance.
- Main theorem (Theorem 1): When the inner oracle reaches an ε-approximate solution and the learning rate η_t∼O(1/√T), DAT converges to a first-order stationary point at an O(1/√T) rate; the error has four terms, corresponding to the large-batch benefit O(σ/√MB), the quantization error min{√d/4^b, d/2^b}, the number of iterations T and the inner accuracy ε.
- Technical points: The authors state the difficulty lies in the nonlinear coupling of LALR's biased adaptive gradients with the alternating-update error; the paper proposes a new descent lemma (Appendix Lemma 2) to handle it, and proves that large batches can compensate for the bias introduced by layerwise normalization.
What experiments did the paper conduct?
On ImageNet/ResNet-50, DAT-PGD's RA of 38.45% is close to centralized AT's 40.38%, and per-epoch time drops from 6022 seconds to 1960 seconds; also covers randomized smoothing, semi-supervised, fine-tuning and quantization ablations.
Experimental Setup
- Models tested: Pre-act ResNet-18 (CIFAR), ResNet-50 (ImageNet), Wide ResNet-28-10 (semi-supervised comparison).
- Datasets: ImageNet (supervised DAT); CIFAR-10 + 500K unlabeled images from 80 Million Tiny Images (semi-supervised); CIFAR-10/100 as fine-tuning target datasets.
- Baselines: AT, Fast AT (both centralized), DAT w/o LALR, DAT-LSGD (Xie et al., 2019).
- Training parameters: training ε=8/255 (CIFAR), 2/255 (ImageNet); PGD generates 10 steps (CIFAR), 4 steps (ImageNet); 100 epochs (CIFAR-10), 30 epochs (ImageNet); ImageNet uses 6×512=3072 batch, 36 GPUs.
- Evaluation: RA against PGD (CIFAR 20 steps, ImageNet 10 steps, ε same as training), AA against AutoAttack, TA is standard accuracy; all experiments run with 3 random seeds and averaged.
- Communication protocols: Ring-AllReduce (one-sided quantization), parameter server (two-sided quantization), HPC (no quantization); 1Gbps Ethernet between nodes.
Main Results
Table 1 (ImageNet, ResNet-50, 6×6 configuration, batch 6×512; AT/Fast AT are 1×6, batch 512):
Swipe horizontally to read the table
Method TA (%) RA (%) AA (%) C (s) T (s) AT 62.70 40.38 37.46 NA 6022 DAT-PGD w/o LALR 57.09 34.02 30.98 865 1932 DAT-PGD 63.75 38.45 36.04 898 1960 Fast AT 58.99 40.78 37.18 NA 1544 DAT-FGSM w/o LALR 55.04 35.03 32.16 863 1080 DAT-FGSM 58.02 40.27 36.02 859 1109 - Authors' interpretation: Direct distribution without LALR causes large drops in RA/TA; with LALR, DAT-PGD/DAT-FGSM are comparable to centralized AT/Fast AT, and training is faster.
- Source of speedup: The authors compute the computational speedup of DAT-PGD excluding communication as 6022/(1960−898)=5.67, close to the ideal 6×; the failure to reach linear speedup is mainly due to communication overhead.
- Cost of FGSM: DAT-FGSM has about 5% lower standard accuracy than DAT-PGD, and RA variance of about 1.5% (DAT-PGD about 0.5%); the authors state FGSM training is more unstable, consistent with Li et al., 2020.
Scalability and Attack Strength
- Figure 2: DAT-FGSM and DAT-LSGD compared under 1×6, 3×6, 6×6 configurations; the authors state DAT scales more smoothly with the number of nodes and has smaller accuracy loss; they also note this paper's setting is harder (36 GPUs, batch 85 per GPU, whereas Xie et al. use 128 GPUs, batch 32 per GPU, 30-step PGD training).
- Figure 3: On ImageNet/ResNet-50, RA curves of DAT-PGD, DAT-FGSM and AT under different perturbation sizes (x-axis 0–10, divisor 255) and different PGD steps (0–30); the authors state DAT matches AT at all ε and step counts.
Randomized Smoothing Certified Robustness (Table 2)
- Setup: CIFAR-10/ResNet-18, Smooth-AT baseline with N=2 noise copies, Smooth-DAT with N=20; two settings σ=0.12 and 0.25, evaluating certified accuracy (CA) at each l2 radius r.
- Results: At σ=0.12, DAT (N=20) has CA 0.838 at r=0.05 (baseline 0.832), slightly higher than the baseline at all radii; at σ=0.25, DAT is 0.514 at r=0.5 (baseline 0.498).
- Authors' interpretation: DAT makes Smooth-AT with large N feasible, thereby improving certified robustness; CA decreases as r increases, and large σ involves a trade-off between large and small radii (consistent with Cohen et al., 2019).
Semi-supervised and Pretraining Fine-tuning
- Semi-supervised (Table 3): CIFAR-10 + 500K unlabeled Tiny Images, 12 machines with 1 GPU each. ResNet-18 (batch 12×2048): DAT-PGD TA 87.00/RA 47.34/AA 45.23; Wide ResNet-28-10 (batch 12×128): DAT-PGD TA 89.37/RA 62.06/AA 58.35, which the authors state is close to Carmon et al., 2019's 89.69% TA and 59.53% AA on RobustBench.
- Pretraining + fine-tuning (Figure 4): ResNet-50 pretrained with DAT on ImageNet, fine-tuned to CIFAR-10/100; the authors state TA and RA quickly surpass the from-scratch (no pretraining) control within 3 epochs.
Other Ablations and Analyses
- Quantization (Table 4, ImageNet/ResNet-50): DAT-PGD 32→16→8 bit (one-sided) RA is 38.45→38.40→37.90; 8-bit two-sided drops to 34.59; DAT-FGSM 8-bit two-sided RA 33.20, the authors state DAT-FGSM is more sensitive to quantization.
- HPC: Under NVLink/IB interconnect, DAT-PGD communication time drops from 898 seconds to 15 seconds with no loss in RA 38.55 (Table 4).
- Cyclical learning rate (Figure A1): Fast AT + CLR on CIFAR-10/ResNet-18 degrades markedly in accuracy as batch increases from 128 to 12288; the authors state this is because under large batch the number of iterations is insufficient to complete the cyclical schedule, so DAT uses piecewise decay and early-stop instead.
- 8-bit quantization supplement: The main text states 8-bit DAT's RA is only 0.55% lower than 32-bit (corresponding to the one-sided quantization row in Table 4).
What are some points that can be further explored?
The authors only note that future work could examine acceleration under extreme training settings; the experimental coverage is limited to image classification, l∞ perturbations, and three model architectures.
Limitations noted by the authors and future directions
- Speedup in extreme cases (Conclusions): The authors state it is worth examining in the future the speedup of DAT in extreme training cases, e.g., using significantly more attack steps and distributed machines, and extremely large models.
- The paper has no dedicated Limitations section; the above is the only explicitly stated future direction in the conclusion.
Scope of experimental coverage
- Models tested: Pre-act ResNet-18, ResNet-50, Wide ResNet-28-10, 3 architectures in total, all image classification networks (Sec 6.1).
- Datasets and tasks: ImageNet, CIFAR-10/100, 80 Million Tiny Images; scenarios include supervised AT, semi-supervised AT, randomized smoothing certified robustness, pretraining+fine-tuning (Sec 6).
- Perturbations and attacks: Training and evaluation both use l∞ perturbations (ε=8/255 or 2/255); evaluation attacks are PGD (up to 30 steps, Figure 3) and AutoAttack; certified robustness uses l2 radius (Table 2).
- Scale upper bound: At most 36 GPUs (6 nodes×6 V100), batch 3072; semi-supervised uses 12 machines with 1 GPU each; HPC experiments use 6 of a 57-node cluster (Appendix 4.2).
- Repetitions: All experiments average over 3 random seeds (Sec 6.1); the paper does not report statistical significance tests for differences between DAT and baselines.
Summarize the main content of the paper
DAT extends adversarial training to distributed multi-machine: LALR solves large-batch min-max, and on ImageNet RA is comparable to AT while each epoch is about 3× faster.
A paper on systematizing robust training: extending adversarial training (AT) from single-machine to distributed multi-machine large-batch settings, while maintaining accuracy and giving convergence guarantees.
- Problem: AT is computationally intensive and hard to scale; Figure 1 shows that after naively enlarging the batch, AT's RA/TA drop substantially, and existing fast AT and multi-machine attempts each have shortcomings.
- Method: Proposes the DAT framework—under a parameter server architecture, each worker generates sample-level perturbations with PGD or single-step FGSM, with optional stochastic gradient quantization, the server aggregates, and the outer layer updates with LAMB-style layerwise adaptive learning rates (LALR); theoretically proves convergence to a first-order stationary point at O(1/√T) in the non-convex setting, with the error decomposed into a large-batch benefit term, a quantization error term and an inner oracle accuracy term.
- Main results (Table 1, ImageNet/ResNet-50, 6×6): DAT-PGD's RA 38.45%, TA 63.75%, close to centralized AT's 40.38%/62.70%; total time per epoch 1960 seconds vs AT's 6022 seconds, and the authors compute a computational speedup of 5.67× excluding communication.
- Extended scenarios: Semi-supervised DAT on Wide ResNet-28-10 has AA 58.35%, which the authors state is close to the 59.53% reported on RobustBench; Smooth-DAT (N=20) has higher certified accuracy than the N=2 baseline; ImageNet pretraining+fine-tuning surpasses from-scratch training within 3 epochs.
- Costs and exceptions: DAT-FGSM has about 5% lower TA than DAT-PGD and RA variance of about 1.5%; 8-bit two-sided quantization reduces DAT-PGD's RA from 38.45% to 34.59%.
- Authors' conclusion: DAT is general (supports labeled/unlabeled, multiple attack generators, gradient compression) and has convergence guarantees; in the future it is worth examining the speedup in extreme cases with more attack steps, more machines and larger models.
Related papers
- Semantic-aware Adversarial Fine-tuning for CLIPDefensesarXiv