On adaptive attacks against adversarial example defenses
On Adaptive Attacks to Adversarial Example Defenses
The authors construct adaptive attacks against each of 13 adversarial example defenses from ICLR/ICML/NeurIPS one by one, and the authors claim all can be bypassed, with most accuracies dropping to near the baseline.
All defenses claim white-box robustness: the attacker knows the model structure, weights, and necessary additional information; evaluated according to the threat model each defense claims, all are lp norms (p≥0), mostly l∞ (e.g., CIFAR-10 ε=8/255, MNIST ε=0.3), with l0 also present; the attack objective is untargeted (targeted for the audio defense).
- Adaptive attacks have become the standard for evaluating adversarial example defenses, but the authors believe that typical adaptive evaluations are incomplete: many papers merely reuse "adaptive" attacks aimed at old defenses without adjusting them for the new defense, leading to overestimation of the defense.
- For each of the 13 defenses, the full process is followed: read the paper and source code, propose failure hypotheses, design targeted loss functions and attack algorithms, iterate until the attack succeeds, and summarize six general attack themes (e.g., attack the complete defense, ensure loss consistency).
- The authors claim all 13 defenses were bypassed: the k-WTA adversarially trained model's accuracy dropped to 0.16%, Ensemble Diversity to 0%, EMPIR to 1.5%, and several detectors' detection rates to 0%; most defenses' accuracies were reduced to near the baseline model.
- The authors do not discuss limitations in a separate subsection; the conclusion states that the evaluation is empirical and not certified, and puts forward an informal "no-free-lunch" view: any given attack can be blocked by a defense that is not robust. The experiments cover 13 defenses, mainly on CIFAR-10/MNIST, with different settings in each section.
- Evaluated models
- k-Winners Take All (vanilla ResNet-18)k-Winners Take All (adversarially trained ResNet-18)The Odds are Odd (wide ResNet)Generative Classifiers (MNIST)Generative Classifiers (CIFAR-10)Robust Sparse Fourier TransformRethinking Softmax Cross EntropyError Correcting CodesEnsemble Diversity (ResNet-20)EMPIRTemporal DependencyMixup Inference (MI-OL)ME-Net (USVT)ME-Net (USVT_ADV)ME-Net (nuclear-norm)Asymmetrical Adversarial Training (integrated classifier)Asymmetrical Adversarial Training (generative classifier)Turning a Weakness into a Strength
- Benchmarks
- CIFAR-10MNISTImageNetautomatic speech recognition
- Metrics
- robust accuracydetection ratemedian distortionFalse-Positive-Rate
The paper by the four researchers Tramer, Carlini, Brendel and Madry shows that thirteen adversarial example defenses published at ICLR, ICML and NeurIPS, although almost all of them had been evaluated with adaptive attacks, were all still bypassed, with accuracy far below the originally reported values. Unlike previous work that reported only attack results, the paper publicly presents, case by case, the complete process from reading the original work and proposing hypotheses to constructing a successful attack, and summarizes six recurring attack themes. The authors point out that the PGD, C&W, BPDA, EOT and other techniques needed to bypass the defenses already existed; the problem lies in the evaluation methodology rather than the techniques; adaptive attacks cannot be automated and must be customized for each defense. For example, on CIFAR-10, when attacking the k-Winners Take All defense with ℓ∞ perturbation 8/255, the accuracy of the normally and adversarially trained ResNet-18 dropped to 0% and 0.16% respectively.
Why it was selectedThe paper dissects case by case the full process by which thirteen top-conference defenses were bypassed by adaptive attacks, and provides transferable defense evaluation methods and attack design principles.
Related events and news
0 events · 3 news mentions
News that mentions this paper
- SPY Labcites the paper's id or linkSPY Lab 研究:机器学习隐私防御的评测存在误导
They use weak, non-adaptive attacksSimilar concerns have previously been raised for defenses against adversarial examples..
- Nicholas Carlini Writingcites the paper's id or linkNicholas Carlini 公开其 2016–2019 年研究想法日志文件
I remember considering working on this paper in the summer of 2017 and deciding that it probably wasn't actually something that would be interesting...
- Nicholas Carlini Writingcites the paper's id or linkNicholas Carlini 等攻破 13 项对抗样本防御,质疑防御研究是否在进步
We have a new paper out as a result of these attacks.
Paper analysis
What problem does this paper attempt to solve?
The authors believe that adaptive attacks in current defense evaluations are generally incomplete, and that the 13 defenses that have already undergone adaptive evaluation can still be bypassed.
Core question: are adversarial example defenses that have undergone adaptive attack evaluation really robust? The authors' answer is no.
- Problem setting: over the past five years the community has proposed many adversarial example defenses, but many defenses only tested static weak attacks and were subsequently bypassed by stronger attacks. The authors note that the proportion of defenses that perform adaptive attack evaluation has risen from near zero in 2017 to one third in 2018, and now to nearly all.
- Shortcomings of existing evaluations: the authors find that the common practice is to reuse off-the-shelf "adaptive" attacks that previously bypassed other defenses, without considering how to modify them for the new defense; and previous breaking papers usually only present the final successful attack, without describing the methodology for discovering the attack.
- Core observation: the authors believe the problem lies in methodology rather than technique—the tools needed to build stronger attacks already exist in the literature; and no single attack strategy works against all defenses, adaptive attacks cannot be automated, and must be manually adjusted for each defense.
- Paper contributions: a complete record of the evaluation process for 13 defenses (selected from ICLR, ICML, NeurIPS), from first reading the paper and forming hypotheses to the final attack, distilling six reusable attack themes (Section 4).
- Threat model:
- Attacker goal: generate adversarial examples that cause the classifier to err; the paper uniformly adopts an untargeted objective (targeted for the audio defense section).
- Attacker knowledge: all defenses claim white-box robustness, and the attacker knows the model structure, weights, and necessary additional information.
- Attacker capability: evaluated according to the threat model each defense itself claims, all are lp norms (p≥0), mostly l∞ (e.g., CIFAR-10 ε=8/255), with l0 and unbounded attacks also present.
- Victim systems: 13 published defense models or detectors.
What related research exists?
Related work falls into four categories: adversarial examples and attack techniques, prior defense evaluation, defense ideas that were broken, and automated attack tools.
Adversarial examples and basic attacks
- Szegedy et al. (2014), Biggio et al. (2013)—proposed the adversarial example phenomenon; Madry et al. (2017) proposed PGD and adversarial training; Carlini & Wagner (2017b) proposed the C&W family of attacks. These are the tool basis for the attacks in this paper.
- Athalye et al. (2018a) proposed BPDA, and Athalye et al. (2018b) proposed EOT; this paper directly uses these two techniques in many places to handle non-differentiable and randomized defenses.
Prior defense evaluation work
- Carlini & Wagner (2017a) bypassed ten detection methods; Athalye et al. (2018a) bypassed multiple defenses at ICLR 2018. The paper notes that these works mainly present final results and do not describe the methodology for arriving at the attacks; the difference of this paper lies precisely in fully documenting the evaluation process.
- Carlini et al. (2019) gave a general checklist for defense evaluation; this paper uses it to diagnose problems such as gradient masking.
The 13 evaluated defenses (grouped as in the paper)
- Gradient masking type: k-WTA (Xiao et al., 2020) deliberately uses discontinuous activations to break gradients.
- Detection type: The Odds are Odd (Roth et al., 2019), Temporal Dependency (Yang et al., 2019b), Turning a Weakness into a Strength (Hu et al., 2019), Asymmetrical Adversarial Training (Yin et al., 2020).
- Ensemble and training objective type: Ensemble Diversity (Pang et al., 2019), EMPIR (Sen et al., 2020), Error Correcting Codes (Verma & Swami, 2019), Rethinking Softmax Cross Entropy (Pang et al., 2020a).
- Preprocessing and generative model type: Robust Sparse Fourier Transform (Bafna et al., 2018), ME-Net (Yang et al., 2019a), Mixup Inference (Pang et al., 2020b), Generative Classifiers (Li et al., 2019).
Automated attack tools
- Croce & Hein (2020)'s AutoAttack: the authors state that among four identical defenses it only partially succeeded on two, and that it cannot be directly used for detection-type defenses, which shows that automated tools cannot replace manual adaptive attacks.
Baselines and benchmarks: Each section uses the numbers reported in the defense's original paper as a comparison (e.g., k-WTA claims 50.0%, Ensemble Diversity reports 48.4%, EMPIR reports 13.5%), and repeatedly compares with a pure adversarial training baseline; the datasets are CIFAR-10, MNIST, ImageNet, and one speech recognition task. The authors position this paper as: not proposing new attack techniques, but giving a methodology and case studies for doing adaptive evaluation.
How does the paper address this problem?
For each defense, follow the complete process of "read the paper → form hypotheses → design targeted losses and attacks → iterate," and distill six attack themes.
Overall approach: rather than presenting the "final blow," the evaluation of each defense is made into a case study, recording the entire process from first reading to breaking it. The core method is manually designing a consistent, easy-to-optimize loss function for each defense, then solving it with the simplest possible optimization algorithm (mostly PGD variants).
Evaluation process (Section 3)
- One section per defense, with a fixed structure: defense principle → reason for selection → initial hypotheses and experiments → final robustness evaluation → lessons learned.
- Selection criteria: published at ICLR/ICML/NeurIPS, and representative of some defense idea; all defenses have public or obtainable source code.
- The process faithfully reflects the actual steps, without post-hoc embellishment of conclusions.
Six attack themes (Section 4, Table 1)
- T0 Pursue simplicity: attacks stay as close as possible to "gradient descent + suitable loss," adding complexity only when simple attacks fail.
- T1 Attack the complete defense: if end-to-end differentiable, attack the whole thing; preprocessing steps must be included.
- T2 Target key components: complex defenses often have only one or two components that are critical; attack only these.
- T3 Adjust the objective to simplify the attack: e.g., use targeted/multi-targeted attacks, feature adversaries (matching another class's internal representation).
- T4 Ensure loss consistency: an increase in loss must correspond to a stronger attack, otherwise even the strongest optimizer will fail.
- T5 Change the optimization method: white-box gradients are not always suitable; score-based/decision-based black-box attacks can optimize hard-to-differentiate losses.
- T6 Adversarial training needs strong attacks: if the attack during training is too weak, combined defenses can be worse than adversarial training alone.
Core attack construction (examples)
- General form: xi+1 = Proj(xi + α · normalize(∇xi L(xi, y))), i.e., projected gradient ascent, where the design of the loss L is key.
- For detection-type defenses, use logit/feature matching: ||z(x′) − z(xt)||22, making the adversarial example's logit consistent with another class's clean sample (Section 6, Equation (1)).
- For randomized defenses, use EOT to average gradients; for non-differentiable preprocessing, use BPDA approximation; for gradient masking, use finite differences to estimate gradients (k-WTA uses M=20000 random perturbations, 100 steps).
Success criteria
- The goal is not to reduce accuracy to 0%, but to demonstrate that the original adaptive evaluation is insufficient and that accuracy can be reduced below the originally reported value; in most cases the authors claim it can be reduced to near the baseline model.
- Detection-type defenses report detection rate (at a fixed false positive rate); unbounded attacks (audio) report the median distortion required for success rather than the success rate.
What experiments did the paper conduct?
13 defenses evaluated one by one, the authors claim all were bypassed; most accuracies dropped to near the baseline, and several detectors' detection rates dropped to 0%.
Experimental setup
- Subjects: 13 defenses, selected from ICLR/ICML/NeurIPS, covering gradient masking, detection, ensembles, preprocessing, generative models, and adversarial training variants (Table 1 lists all).
- Datasets: CIFAR-10 (most), MNIST, ImageNet (Section 17), speech recognition (Section 13).
- Threat model: the lp threat model each defense itself claims; common settings are CIFAR-10 l∞ ε=8/255 or 0.031, MNIST ε=0.3, l0 (Section 8).
- Models: ResNet-18 (k-WTA, ME-Net), ResNet-20 (Ensemble Diversity), wide ResNet (Odds are Odd), VGG features (generative classifier), etc., all using the models and code released by the authors.
- Metrics: robust accuracy, detection rate (at a fixed false positive rate), median distortion (unbounded attacks).
- Attack implementation: based on Foolbox; PGD steps and repetitions differ by section (e.g., 400 steps, 250 steps, 100 steps, repeated 3–5 times).
Main results
The paper has no summary table; the table below is compiled from the text of each section (all are the authors' reported final attack results; conditions are noted):
Swipe horizontally to read the table
Defense (section) Setting Originally reported After this paper's attack k-WTA (5) CIFAR-10 l∞ 8/255, vanilla / adversarially trained ResNet-18 13.1% / 50.0% 0% / 0.16% Odds are Odd (6) CIFAR-10 l∞ 8/255, detection rate 71.4% 0% (below false positive rate) Generative Classifier (7) MNIST/CIFAR-10, classifier + detector still effective under large perturbations CIFAR-10 100% success; MNIST 90% (100% after fixing the seed) Sparse Fourier (8) MNIST l0 claimed >75% at l0=55 median distortion 14.8 pixels MMC loss (9) CIFAR-10 ε=0.031 — <0.5% Error Correcting Codes (10) CIFAR-10 ε=0.031 57% <5% Ensemble Diversity (11) CIFAR-10 l∞ 0.01 48.4% 0% EMPIR (12) CIFAR-10 ε=0.031 13.5% (at ε=0.1) 1.5% - The authors emphasize that the goal is not to reduce to 0%, but to demonstrate that the original evaluation is insufficient; however, most defenses were reduced to near the baseline model.
- The authors claim that AutoAttack only partially succeeded on two of four of these defenses, to illustrate the limitations of automated tools.
Detection and audio defenses
- Asymmetrical Adversarial Training (16): on CIFAR-10 l∞ 8/255 at a 5% false positive rate, the integrated classifier's robust accuracy dropped from 30% to 11%, and the generative from 55% to 37% (below full PGD adversarial training); the authors acknowledge they did not reduce it to 0%, and list four reasons for believing the evaluation is sufficiently thorough (the defense is simple, interpretable, the attack is simple, and standard checks only reduce it by a further 1%).
- Temporal Dependency (13): targeted unbounded attack on speech recognition, 100% success in fooling both the classifier and the detector; the baseline median l∞ distortion is 41 (16-bit range 65536), 46 with the defense added, and the authors state the improvement over baseline defenses (e.g., bit quantization) is not large.
- Turning a Weakness into a Strength (17): CIFAR-10 accuracy reduced to 0%, ImageNet <1%, detection rates all 0% (the original paper's threat model); the authors also found that the original paper's PGD implementation lacks the sign operator, and the same is true of the #Steps check inside the defense.
Randomization and preprocessing defenses
- ME-Net (15): CIFAR-10 l∞ 8/255; PGD+BPDA (200 steps) reduces USVT to 35% and USVTADV to 50%; BPDA+EOT (40 gradient averages) further reduces them to 15% and 32%; the nuclear-norm variant is reduced to 13% with the same attack. The authors note that USVTADV's 32% is below the 40%+ of pure adversarial training with the same architecture.
- Mixup Inference (14): CIFAR-10 l∞ 8/255, MI-OL originally reported about 57% (adaptive); after incorporating the mixing step into the attack, PGD with 50 steps reduces it to 43.9%, close to the 42.5% of the undefended base.
Other ablations and analyses
- k-WTA: PGD remains unreliable when increased from 40 steps to 400 steps; Figure 1 shows the loss surface exhibits plateau-like jumps (the x-axis maximum perturbation changes the input by only 0.1%).
- Ensemble Diversity: step size ×3 reduces accuracy from 48% to 26%; 50 steps → 20%; 250 steps → 10%; repeated 3 times → 7%; switching to the B&B attack → 5% → 0%.
- MMC loss: dividing the logit by a constant reduces CIFAR-10 accuracy from 24% to 18%, but is insufficient to break it.
- Generative Classifier: pure random perturbation + restarts reduces MNIST accuracy below 50%; repeated classification of clean inputs has a 5%–10% misclassification rate.
- ME-Net: attacking the base ResNet-18 alone without preprocessing, accuracy cannot go below 10%; the authors believe the base itself has some robustness.
- Odds are Odd: with logit matching alone, the detection rate is 17% for a single noise source and 10% for 10 noise sources; with EOT added, 0%.
What are some points that can be further explored?
The authors do not include a limitations section; the conclusion presents a "no-free-lunch" view, and each section ends with reader exercises as directions for follow-up.
Limitations and future directions noted by the authors
- The evaluation is empirical: Section 16 explicitly says that the defense is not certified and that evaluation will "ultimately always be empirical"; the authors use this to explain why they believe the results are close to true robustness.
- Informal no-free-lunch: The Conclusion proposes that for any given attack one can construct a defense that blocks it but is not robust, so the authors call for not copying the attacks in this paper as templates.
- Attacks cannot be automated: The Conclusion states that no automated tool can comprehensively evaluate defenses, and that adaptive attacks must be manually designed for the specific defense.
- Each section's "Exercises for the reader" lists open questions, for example: whether a smooth differentiable approximation of k-WTA exists (Section 5); whether the logit matching attack is still effective against robust classifiers (Section 6); whether there are defense ideas specific to l0 (Section 8); whether EMPIR can be reduced to 0% with a more consistent loss (Section 12); the source of the robustness of the ME-Net base (Section 15).
- The paper does not discuss its own limitations in a separate Limitations subsection.
Scope of the experiments
- There are 13 defenses tested, all selected from ICLR/ICML/NeurIPS, with one section per defense and different settings for each (Section 3).
- The datasets are CIFAR-10, MNIST, ImageNet (Section 17 only), and one speech recognition task (Section 13); the threat models cover l∞, l0, and unbounded attacks.
- The evaluation of each defense relies on code and pretrained models released by the original authors; the nuclear-norm adversarial training variant of ME-Net was not evaluated because training cost is about 10 CPU years (Section 15).
- The paper does not report sample sizes or confidence intervals for the experiments; the number of repetitions is stated only in some sections (e.g., Ensemble Diversity repeated 3 times, Error Correcting Codes repeated 5 times).
- The authors state that before submission they contacted the authors of all 13 defenses, and that except for one that did not reply, all the rest confirmed the attacks were effective (Introduction).
Summarize the main content of the paper
All 13 defenses that underwent adaptive evaluation were bypassed by manual adaptive attacks; the problem lies in methodology rather than technique.
A methodology paper on how to correctly evaluate adversarial example defenses, using the complete breaking process of 13 defenses as case studies.
- Problem: adaptive attack evaluation has become standard, but the authors find that the typical practice merely reuses old attacks without adjusting them to the new defense, resulting in incomplete evaluation.
- Method: for each defense, follow the process of "read the paper and source code → propose failure hypotheses → design a consistent and easy-to-optimize loss → choose a suitable optimizer → iterate," and summarize six themes: attack the complete defense, target key components, adjust the attack objective, ensure loss consistency, change the optimization method, and adversarial training needs strong attacks.
- Results (reported by the authors): the k-WTA adversarially trained model drops from 50.0% to 0.16% on CIFAR-10 l∞ 8/255; Ensemble Diversity drops from 48.4% to 0%; EMPIR drops to 1.5%; the detection rates of Odds are Odd and Turning a Weakness into a Strength drop to 0%; most defenses are reduced to near the baseline model.
- Exception: Asymmetrical Adversarial Training was not fully broken (the generative variant drops from 55% to 37%), and the authors give four reasons for believing the evaluation is sufficient.
- The authors' conclusion: building stronger attacks does not require new techniques; the problem is methodological; adaptive attacks cannot be automated, and AutoAttack only partially succeeded on two of four identical defenses; attacks should be as simple as possible and manually designed against the weakest link of the defense.
Related papers
- Theoretically Principled Trade-off for Stateful Defenses against Query-Based Black-Box AttacksAnalysis / interpretabilityarXiv