DeepMind paper: adversarial risk and the danger of evaluating with weak attacks
Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
The authors propose evaluating defenses within a framework of adversarial risk and "stealthiness to the attacker," and use stronger attacks such as SPSA to reduce the adversarial accuracy of multiple defenses to near 0%.
The attacker's goal is to find inputs within the ℓ∞ neighborhood (CIFAR-10 ε=8, ImageNet ε=2) that cause the classifier to misclassify; attack methods include white-box PGD, gradient-free SPSA, and transfer-based attacks transferred from substitute models; the victim systems are image classifiers with defenses on CIFAR-10 and ImageNet.
- Many adversarial example defenses are evaluated only under the weaker attacks used in the original paper; the authors argue that this may merely make adversarial examples harder for the attack to find rather than genuinely eliminating them, leading to overestimated robustness.
- The authors define commonly used attacks as a computable surrogate objective for true adversarial risk, propose the concept of "obscurity to the attacker", and re-evaluate various defenses using gradient-free SPSA and improved gradient attacks.
- On CIFAR-10 and ImageNet, defenses such as JPEG compression, PixelDefend, randomization, and High-level Guided Denoiser drop to 0%–10% accuracy under stronger attacks; the adversarially trained model maintains 51% accuracy against SPSA.
- The authors state that SPSA is several orders of magnitude less efficient than gradient methods, and PixelDefend is evaluated on only 100 images due to compute; the experiments are limited to CIFAR-10 and ImageNet image classification, and the paper does not report the number of repetitions or confidence intervals.
- Evaluated models
- PixelDefend (Song et al., 2017)Adversarial Training (Madry et al., 2017)Non-differentiability (Guo et al., 2017)Stochasticity (Xie et al., 2017)High-level Guided Denoiser (Liao et al., 2017)PixelCNNResNet-50VGG-like fully convolutional model
- Benchmarks
- CIFAR-10ImageNet
- Metrics
- adversarial accuracyobscurity boundattack success rate
DeepMind researchers propose an adversarial risk framework, pointing out that commonly used attacks and evaluation metrics are only proxy objectives for true adversarial risk, and that a high evaluation score does not mean true robustness. The paper formalizes the concept of a model's obscurity to a specific attacker, and adapts the gradient-free optimization method SPSA into an adversarial attack for validation. On ImageNet and CIFAR-10, the adversarial accuracy of PixelDefend drops from 75% to below 10% (CIFAR-10, ε=8), the JPEG compression defense drops from 15% to 0%, the high-level guided denoiser drops from 75% to 0%, and the random padding and resizing defense drops from 36% to below 1% (the latter three are all ImageNet, ε=2).
Why it was selectedThe authors use the SPSA gradient-free attack to drive the adversarial accuracy of multiple defenses close to zero, suggesting that evaluating only against weak attacks overestimates model robustness.
Paper analysis
What problem does this paper attempt to solve?
The authors argue that many adversarial defenses are merely "obscure" to the attacks used in evaluation, and that stronger attacks can reduce their accuracy to near 0%.
Core problem: the reported robustness of many recently proposed adversarial example defenses may come from "obscurity to weak attacks" (security via obscurity) rather than genuinely eliminating adversarial examples.
- Problem setting: neural networks produce incorrect outputs under small input perturbations, and a large body of defense work (Guo et al., 2017; Song et al., 2017; Xie et al., 2017; Liao et al., 2017) reports robustness to commonly used attacks. The authors ask: do these models truly have no adversarial examples, or are they merely robust to current attack methods.
- Shortcomings of existing evaluations: the authors argue that commonly used attacks and evaluation metrics define only one computable surrogate objective for true adversarial risk, and defenses may merely optimize this surrogate objective; surrogate adversarial risk is downward-biased relative to true adversarial risk, so a high evaluation score does not imply true robustness.
- Core hypothesis: many defenses make specific attacks fail through gradient masking, non-differentiability or randomness, but adversarial examples still exist on the decision boundary; the authors propose "obscurity to attacker f" to characterize the gap between evaluation metrics and true adversarial risk.
- Paper contributions: formalize the concepts of adversarial risk and obscurity, give heuristic criteria for identifying obscure models, adapt gradient-free optimization (SPSA) into an attack, and empirically reduce the adversarial accuracy of various defenses to near 0%.
- Threat model:
- Goal: find inputs within the ℓ∞ neighborhood (CIFAR-10 ε=8, ImageNet ε=2, pixel range [0, 255]) that cause the classifier to misclassify, using a margin-based loss (Carlini and Wagner, 2017b).
- Knowledge and capability: white-box attacks can directly take gradients (PGD); gradient-free attacks (SPSA) only require querying model outputs; transfer-based attacks transfer adversarial examples from a known surrogate model.
- Victim systems: image classifiers on CIFAR-10 and ImageNet with various defenses added.
What related research exists?
Related work covers four groups: adversarial attacks, gradient masking in defense evaluation, black-box robustness, and verified robustness.
Adversarial attacks
- Goodfellow et al. (2014), Carlini and Wagner (2017b)—white-box attacks that use optimization methods to directly find perturbations that maximize loss for the model.
- Papernot et al. (2016; 2017)—black-box / oblivious attacks that require no direct access to the model; Kurakin et al. (2016b), Brown et al. (2017), and Athalye et al. (2018b) further demonstrate attacks on printed images and 3D objects.
- Transfer attacks—Szegedy et al. (2013) and Papernot et al. (2017) show adversarial examples transferring from substitute models; the paper notes that the success of transfer attacks depends heavily on the similarity between the substitute model and the target model.
Defense evaluation and gradient masking
- Papernot et al. (2017), Goodfellow and Papernot (2017)—propose the gradient masking phenomenon: removing or shrinking gradients or adding noise makes gradient attacks fail, which may lead practitioners to mistakenly believe the model is robust.
- Athalye et al. (2018a)—independent contemporaneous work with this paper, reaching similar conclusions: many defenses merely obscure the gradient; the paper notes that its attack methods differ, but the results likewise emphasize evaluating defenses with targeted attacks.
Black-box robustness and competitions
- Ensemble Adversarial Training (Tramèr et al., 2017) and High-level Guided Denoising (Liao et al., 2017)—the paper acknowledges real progress in black-box robustness and points to the NIPS adversarial defense competition (Kurakin and Goodfellow, 2017) as a comparison under the black-box threat model.
Verifiable robustness and transparency
- Katz et al. (2017), Bunel et al. (2017), Carlini et al. (2017)—neural network verification, computing provable worst-case adversarial examples.
- Kolter and Wong (2017), Raghunathan et al. (2018), Sinha et al. (2017)—training procedures that optimize robustness certificates by relaxing the original adversarial optimization problem and solving it efficiently.
Baselines and benchmarks: The paper does not propose new defenses for horizontal comparison but directly attacks existing defenses; the defenses attacked are the input transformations (JPEG compression) of Guo et al. (2017), PixelDefend of Song et al. (2017), the randomization of Xie et al. (2017), the High-level Guided Denoiser of Liao et al. (2017), and the adversarial training of Madry et al. (2017); the datasets are CIFAR-10 and ImageNet.
The authors position this paper as: similar in conclusions to Athalye et al. (2018a) but with different attack methods, emphasizing evaluating defenses with stronger attacks, and using imperceptibility as a quantitative metric connecting adversarial robustness research and model transparency research.
How does the paper address this problem?
Formalize the evaluation problem in terms of adversarial risk and obscurity, then use stronger attacks such as SPSA to empirically measure the true adversarial accuracy of various defenses.
Overall idea: treat commonly used attacks as a surrogate objective for true adversarial risk, define "obscurity to the attacker" to quantify the gap between evaluation metrics and true risk, then empirically test various defenses with gradient-free and improved gradient attacks. The core concepts are adversarial risk and obscurity.
Formalization: adversarial risk and obscurity
- Expected risk and worst-case risk: standard supervised learning minimizes the expected risk E(x,y)∼ D ℓ(mθ(x), y); the authors argue that high-risk settings require also controlling worst-case risk, i.e., taking the supremum of the loss over the support of the data distribution.
- Local adversarial risk: L(θ) = E(x,y)∼ D supx′ ∈ Nϵ(x) ℓ(mθ(x′), y), i.e., taking the maximum loss within the ε neighborhood of each input and then taking the expectation; it is a lower bound on worst-case risk and can be approximated with off-the-shelf optimizers such as PGD.
- Surrogate adversarial risk and obscurity: approximate the inner maximization with a specific attacker f to obtain L̂(θ, f), i.e., the evaluation metric reported in the paper; Obscurity(θ, f) = L(θ) − L̂(θ, f), measuring how many adversarial examples of the model f cannot find. If there exists a computationally efficient f that makes obscurity low, the authors call the model transparent.
Attack design
- Optimization objective: untargeted, white-box margin-based loss (Eq. 6), negative when misclassified, with ℓ∞ norm constraint; the authors follow Carlini and Wagner (2017b) in arguing that logits are more "linear" and easier to optimize than output probabilities.
- Gradient attack: PGD with Adam updates, with random initialization within N_ε(x0), to counter local minima caused by gradient masking.
- Gradient-free attack: SPSA (Algorithm 1), estimating the gradient by finite differences along random Rademacher directions, perturbation δ=0.01, batch size 8192, at most 100 iterations (Table 3); the authors state that the Rademacher distribution was chosen because SPSA requires the inverse of v to have finite first and second moments, and that averaging gradient estimates is for GPU implementation efficiency and faster convergence.
- Adaptations for specific defenses: for the randomized defense (Xie et al.), optimize instead the expected objective E[J_adv] over the model's internal randomness; for the PixelCNN detection defense, add λ·lCNN to the objective to jointly optimize misclassification and likelihood.
Heuristic criteria for obscurity
- Whether the surrogate objective converges to the true objective: in transfer attacks, unless the surrogate model and target model are guaranteed to be similar, adversarial examples may not transfer.
- Whether the true objective behaves well: vanishing gradients, discrete or highly nonlinear operations can make gradient attacks fail, but do not eliminate adversarial examples.
- The authors emphasize: heuristics can only approximately predict obscurity; confirming high obscurity still requires empirical testing with stronger attacks.
Success criterion
- Report adversarial accuracy (1 − L̂, i.e., accuracy under 0-1 loss), and give the obscurity bound: the difference between the original evaluation accuracy and the accuracy after this paper's attack (Table 1).
- The attack stops when the margin-based objective is less than −5.0; any attacker gives a valid lower bound on true adversarial risk.
What experiments did the paper conduct?
Under stronger attacks, the accuracy of five defenses mostly drops to 0%–10%, while the adversarially trained model maintains 51% against SPSA.
Experimental setup
- Models tested: ResNet-50 on ImageNet (clean accuracy 75%), a VGG-like fully convolutional model on CIFAR-10 (clean accuracy 94.5%) (Appendix A).
- Defenses tested: JPEG compression (Guo et al., 2017, quality 75), PixelDefend (Song et al., 2017, ε_defend=16), randomized padding/resize (Xie et al., 2017), High-level Guided Denoiser (Liao et al., 2017), adversarial training (Madry et al., 2017).
- Datasets and perturbations: CIFAR-10 (ε=8) and ImageNet (ε=2), pixel range [0, 255].
- Attacks: PGD (Adam, random initialization), SPSA (δ=0.01, batch 8192, at most 100 iterations), FGSM/PGD transfer attacks, expected-objective attack against the randomized defense.
- Sample size: 1000 random test images each for CIFAR-10 and ImageNet; PixelDefend evaluated on only 100 images due to compute constraints (Appendix B); the paper does not report the number of repetitions.
- Metrics: adversarial accuracy (1 − L̂ under 0-1 loss) and obscurity bound; stopping condition is margin objective < −5.0。
主结果
据 Table 1(“Original Evaluation”为原论文评估设置下作者复现的准确率,“Adversarial Accuracy Bound”为本文攻击后的准确率上界):
Swipe horizontally to read the table
数据集 (ε) 防御 原评估准确率 本文攻击后 Obscurity Bound CIFAR-10 (8) PixelDefend 75% <10% >65% CIFAR-10 (8) Adversarial Training 47% <47% >0% ImageNet (2) Non-differentiability (JPEG) 15% 0% 15% ImageNet (2) Stochasticity 36% <1% >35% ImageNet (2) High-level Guided Denoiser 75% 0% 75% - The authors state that most defenses appear robust under the attacker used in the original paper's evaluation, but remain vulnerable to stronger attacks, with true adversarial accuracy near 0%.
- The authors note that the numbers in the left column differ slightly from the original papers because evaluation conditions were unified across defenses.
- Adversarial training is the only defense the authors did not suspect a priori of having high obscurity; its obscurity bound is only >0%.
Attacks on generative-model detection defenses (Section 6.2.1)
- What was tested: Song et al. (2017) use a PixelCNN for likelihood-threshold detection; the authors reproduced it and then applied a PGD attack that jointly optimizes misclassification and likelihood.
- Results: at a threshold with 1% false positive rate (4.68 bits/dim), 99.7% of the original PGD adversarial examples were detected; the adapted attack reduced the detection rate to 0%, and the adversarial examples had higher likelihood than natural examples (Figure 1).
- Authors' interpretation: the authors state that this shows the intuition that "adversarial examples lie in low-probability regions" does not hold for an attacker who knows the defense.
Attacks on randomized and ensemble defenses (Section 6.3)
- What was tested: the randomized padding/resize model of Xie et al. (2017), originally evaluated with an ensemble-pattern attack averaging k=21 sub-models; the authors reproduced an accuracy of 36% (ImageNet, ε=2).
- Results: after switching to stochastic gradient descent on the objective E[J_adv] that takes the expectation over the model's internal randomness, accuracy dropped to 1%.
- Authors' interpretation: the authors argue that even when the surrogate model is an ensemble, adversarial examples may still overfit to specific sub-models, and success of a transfer attack does not guarantee target model robustness.
Obscurity check on adversarial training (Section 6.4, Appendix C)
- What was tested: whether adversarially trained models learned gradient masking through the training process, comparing SPSA with PGD.
- Results: the model's accuracy against SPSA was 51% (CIFAR-10, ε=8); only on 0.2% of images did SPSA succeed while PGD did not; in Figure 3 the margin objective values obtained by the two attacks mostly lie close to the y=x line.
- Authors' interpretation: the authors state that this (though far from conclusive) supports the picture that PGD is close to an optimal attacker for differentiable image classifiers, and note that adversarial training is currently one of the few techniques that appears able to improve true adversarial risk.
Other ablations and analyses
- Gradient-free attack comparison (Table 2, ImageNet, ResNet-50): at batch 256, accuracy after attack was 29% for SPSA, 30% for NES, 72% for ZOO; at batch 8192, both SPSA and NES dropped to 0%, and ZOO to 10%.
- Convergence speed (Figure 2): large-batch attacks start finding adversarial examples later but are more reliable as the number of model evaluations increases; on CIFAR-10 all attacks reduce accuracy to 0% within 32768 model evaluations, and SPSA drops below 5% after 2048 evaluations.
- Different perturbation sizes (Figure 4, Appendix D): four non-adversarially-trained defenses remain robust to the original evaluation attacks across a range of ε, but their accuracy against stronger attacks all drops to near 0 (only axes and legend are visible in the figure; specific values are not labeled).
- PixelDefend computational cost: the authors state that attacking PixelDefend with SPSA is computationally expensive, and that the optimization process in its inference itself constitutes a form of obscurity.
What are some points that can be further explored?
The authors note that SPSA is inefficient and some results are inconclusive; the experiments are limited to five types of defenses on two image datasets.
Limitations and future directions noted by the authors
- SPSA is inefficient: The authors state that the gradient-free optimizer is several orders of magnitude less efficient than gradient methods, so adversarial training in its current form is infeasible on non-differentiable models (Section 6.4).
- The conclusion on adversarial training is uncertain: The authors state that the comparison between SPSA and PGD is "far from conclusive" and only adds to the evidence that PGD is close to an optimal attacker (Section 6.4).
- Heuristics cannot replace empirical testing: The authors state that ultimately confirming the high stealthiness of a defense still requires experimental evaluation with stronger attackers, and heuristic reasoning can only approximately predict it (Section 5).
- Local adversarial risk is only a proxy: The authors state that local adversarial risk provides a certificate only on a fixed set of points, not the overall worst-case risk; the ultimate goal should be a model with both low worst-case risk and low expected risk (Sections 3.1, 7).
- Future directions: The authors hope that rigorous evaluation against strong attackers and a better understanding of stealthiness can drive progress toward models without adversarial examples (Section 7).
Scope of experiments
- The models tested are ResNet-50 on ImageNet and a VGG-like fully convolutional model on CIFAR-10, 2 in total (Appendix A).
- There are 5 categories of defenses tested: JPEG compression, PixelDefend, randomization, High-level Guided Denoiser, and adversarial training (Section 6).
- The sample size is 1000 random test images per dataset, and 100 for PixelDefend due to compute constraints (Appendix B).
- The perturbation settings are CIFAR-10 ε=8 and ImageNet ε=2, ℓ∞ norm (Table 1).
- The paper does not report the number of attack repetitions, confidence intervals, or statistical significance tests; the gradient-free attack comparison in Appendix A uses 100 ImageNet test images.
Summarize the main content of the paper
The authors formalize the concepts of adversarial risk and obscurity, and use stronger attacks to reduce the adversarial accuracy of various defenses to near 0%.
A paper on adversarial robustness evaluation methodology, whose core claim is: many defenses are merely "obscure" to the weak attacks used in evaluation and do not genuinely eliminate adversarial examples.
- Problem: commonly used attacks and evaluation metrics are only a computable surrogate objective for true adversarial risk, and are downward-biased relative to true risk; defenses may make specific attacks fail through gradient masking, non-differentiability or randomness, thereby obtaining inflated evaluation scores.
- Method: the authors define local adversarial risk L and "obscurity to attacker f" (the difference between L and the surrogate risk L̂), give heuristic criteria for identifying obscure models, and adapt gradient-free SPSA into an attack, together with an expected-objective attack against randomized defenses and joint likelihood optimization against detection defenses.
- Main results (Table 1): under ImageNet ε=2, the adversarial accuracy of JPEG compression and High-level Guided Denoiser drops from 15% and 75% to 0%, and the randomized defense from 36% to below 1%; under CIFAR-10 ε=8, PixelDefend drops from 75% to below 10%.
- Detection defense: the PixelCNN likelihood detector originally detected 99.7% of adversarial examples at a 1% false positive rate; after jointly optimizing misclassification and likelihood, the detection rate dropped to 0% (Figure 1).
- Positive exception: the adversarially trained model (Madry et al., 2017) maintains 51% accuracy against SPSA, with SPSA succeeding while PGD did not on only 0.2% of images; the authors state that this supports PGD being close to an optimal attacker, but is far from conclusive.
- Authors' conclusion: evaluation metrics are meaningful only when the attacker can find the adversarial examples that exist; the authors hope the obscurity framework and strong-attack evaluation can drive the design of transparent defenses, moving toward models with genuinely no adversarial examples.
Related papers
- Theoretically Principled Trade-off for Stateful Defenses against Query-Based Black-Box AttacksAnalysis / interpretabilityarXiv