Obfuscated gradients give a false sense of security: bypassing adversarial example defenses

1. 01What problem does this paper attempt to solve?

   The authors argue that many adversarial defenses create false robustness through obfuscated gradients, and propose corresponding attacks to bypass them.

   Core problem: many defenses against adversarial examples appear to withstand iterative optimization attacks; the authors argue that this apparent robustness comes from "obfuscated gradients" rather than true robustness, and that these defenses can be bypassed.

   • Problem setting: adversarial example defenses commonly use iterative optimization attacks (PGD, Carlini & Wagner, etc.) for benchmark evaluation; when the defense makes gradients nonexistent, incorrect, or unusable, gradient-based attacks fail and the defense appears robust.
   • Shortcomings of existing evaluations: the authors say obfuscated gradients are a special case of gradient masking, and gradient masking is known to be an incomplete defense; but the authors observe that 7 of the 9 non-certified white-box defenses at ICLR 2018 rely on this effect.
   • Core observation: the authors divide obfuscated gradients into three categories—shattered gradients (non-differentiable operations or numerical instability cause gradients to be nonexistent or incorrect), stochastic gradients (test-time randomness), vanishing/exploding gradients (multiple iterations of network evaluation unroll into an extremely deep computation).
   • The paper proposes: attack techniques BPDA, EOT, and reparameterization for the three categories of obfuscated gradients respectively, and uses the ICLR 2018 defenses as a case study; the authors say they completely bypass 6 and partially bypass 1; they also give recommendations for defense evaluation (Section 6).
   • Threat model:
     • Attacker goal: construct adversarial examples within the threat model claimed by the defense (e.g., l∞ distortion ε=0.031) to make the model misclassify.
     • Attacker knowledge: white-box—full access to the classifier architecture, weights, and defense details, but does not know the specific values of test-time randomness (only the distribution).
     • Victim systems: the 9 non-certified defenses accepted at ICLR 2018 that claim white-box robustness (including one detection method).

   Collapse this answer

2. 02What related research exists?

   Related work covers adversarial examples and defenses, gradient masking, and the prior attack techniques cited by the authors.

   Adversarial examples and defenses

   • Szegedy et al. (2013), Biggio et al. (2013)—discovered that neural networks are vulnerable to adversarial examples, the starting point of defense research.
   • Goodfellow et al. (2014b)—proposed adversarial training; Madry et al. (2018)—used PGD to solve the inner maximization of the min-max problem for adversarial training, which this paper evaluates as a defense that does not rely on obfuscated gradients.
   • Na et al. (2018)—cascaded adversarial training, first using an iterative method to generate adversarial examples to train the first model, then using a single-step method to train the second model.

   Gradient masking and prior criticism

   • Papernot et al. (2017)—defined gradient masking (a defense "has no usable gradient") and pointed out that it is an incomplete defense against adversarial examples; the authors of this paper define obfuscated gradients as a special case of it.
   • Tramèr et al. (2018)—disrupts gradient descent by learning to make gradients point in the wrong direction; the authors distinguish their work from such "learned to break gradients" defenses: obfuscated gradients refer to defenses that by design necessarily lead to gradient masking.

   Sources of attack methods

   • Kurakin et al. (2016a), Madry et al. (2018), Carlini & Wagner (2017c)—iterative optimization attacks; this paper uses PGD (l∞) and Carlini & Wagner's Lagrangian relaxation (l2) as base attacks.
   • Athalye et al. (2017)—Expectation Over Transformation (EOT), which this paper uses to attack randomized defenses.
   • Bengio et al. (2013)—straight-through estimator; the authors state that a special case of BPDA amounts to applying it to construct adversarial examples.
   • Carlini & Wagner (2017a)—high-confidence adversarial examples, which this paper uses to bypass LID detection.

   Defenses evaluated (also used as comparison objects): Buckman et al. (2018) thermometer encoding, Guo et al. (2018) input transformation, Ma et al. (2018) LID detection, Dhillon et al. (2018) SAP, Xie et al. (2018) randomization, Song et al. (2018) PixelDefend, Samangouei et al. (2018) Defense-GAN; benchmarks are MNIST, CIFAR-10, and ImageNet. The authors omit two defenses with provable security claims (Raghunathan et al., 2018; Sinha et al., 2018) and one defense that only claims black-box security (Tramèr et al., 2018).

   The authors position this paper as: revealing that the robustness of these defenses comes from the common cause of obfuscated gradients, and providing attack techniques and an evaluation checklist to help future defenses avoid the same evaluation pitfalls.

   Collapse this answer

3. 03How does the paper address this problem?

   For each of the three types of obfuscated gradients, the three attacks BPDA, EOT, and reparameterization are given respectively, along with a behavioral checklist for identifying obfuscated gradients.

   Overall approach: first give the characteristic behaviors that identify obfuscated gradients, then propose one attack technique for each of the three causes, making gradient-based iterative optimization attacks usable again. Method names: Backward Pass Differentiable Approximation (BPDA), Expectation Over Transformation (EOT), reparameterization.

   Problem setting and identification

   • Setting: classifier f(·), adversarial example x′ satisfies that D(x, x′) is small and c(x′) ≠ c*(x); distortion is measured with l∞ ([0,1] normalized, 0.031 corresponds to 8/256) and l2 (root mean square normalized by pixel count).
   • Identification checklist (Section 3.1, the authors say these behaviors suggest gradients are obfuscated): single-step attacks outperform iterative attacks; black-box attacks outperform white-box attacks; unbounded attacks fail to reach 100% success; more than 10^5 random samples can find adversarial examples that gradient attacks cannot find; increasing the distortion upper bound does not improve the success rate.

   BPDA: handling shattered gradients

   • Special case (straight-through estimator): when the defense has the form f̂(x) = f(g(x)) and g(x) ≈ x, approximate the derivative of g with the derivative of the identity function: ∇_x f(g(x))|_x=x̂ ≈ ∇_x f(x)|_x=g(x̂), i.e., compute normally in the forward pass and replace g with the identity function in the backward pass.
   • General form: for a non-differentiable layer f^i, find a differentiable approximation g(x) ≈ f^i(x); the forward pass still goes through f^i, and only the backward pass replaces it with g; the authors report it still works when gradients are inexact, but requires more iterations, and replacing only in the backward pass is necessary (replacing in both passes is completely ineffective against Song et al. and many times worse against Buckman et al.).

   EOT: handling stochastic gradients

   • Approach: for defenses with a random transformation t(·) ~ T, optimize the expectation Et~T f(t(x)), using ∇E f(t(x)) = E ∇f(t(x)), and approximate the expectation with samples at each gradient descent step.
   • Parameters: when attacking SAP, take k=10 random samples per iteration to estimate the gradient; the authors say they also need to handle the numerical instability of gradient explosion caused by dividing by numbers close to 0.

   Reparameterization: handling vanishing/exploding gradients

   • Approach: when the defense contains an optimization loop g(·), change variables x = h(z) so that g(h(z)) = h(z) and h is differentiable, thereby directly computing the gradient with respect to f(h(z)) and avoiding differentiating through the unrolled extremely deep network.

   Attack implementation and success determination

   • Base attacks: for l∞ use PGD restricted to the ε ball, for l2 use the Lagrangian relaxation of Carlini & Wagner (2017c); iterate 100 to 10,000 times until convergence.
   • Determination: for MNIST/CIFAR-10, generate untargeted adversarial examples over the entire test set and report model accuracy; for ImageNet, generate targeted adversarial examples with random target classes on 1000 random test images and report attack success rate and accuracy. Each defense is evaluated under the threat model of its original paper.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Attacking the nine white-box defenses at ICLR 2018 one by one, seven rely on obfuscated gradients, six were completely bypassed and one partially bypassed.

   Experimental setup

   • Subjects: the 9 non-certified defenses accepted at ICLR 2018 that claim white-box robustness (including Ma et al.'s LID detection method); 2 provable defenses and 1 defense that claims only black-box security are skipped.
   • Models: a 5-layer CNN for MNIST (99.3% accuracy); Wide ResNet for CIFAR-10 (95% accuracy); InceptionV3 for ImageNet (78.0% top-1, 93.9% top-5); Guo et al.'s original paper used ResNet-50, and the authors reproduce it with InceptionV3.
   • Datasets and scale: the entire test set for MNIST and CIFAR-10 (untargeted); 1000 random test images for ImageNet (targeted, random target classes).
   • Metrics: model accuracy (lower means easier to attack), attack success rate; distortion measured with l∞ (ε=0.031 or 0.015) and l2 (normalized root mean square).
   • Attack parameters: PGD / Carlini & Wagner attacks, 100–10,000 iterations; EOT with k=10 on SAP; Defense-GAN's Evaluation A runs 50k iterations per sample (less than a minute per instance).
   • Repetition and review: the paper does not report the number of repetitions; no LLM or human review, determination is by misclassification.

   Main results

   According to Table 1 (each defense under its original paper's threat model, model accuracy after the authors' attacks):

   Swipe horizontally to read the table

   Defense	Dataset	Distortion	Accuracy
   Buckman et al. (2018)	CIFAR	0.031 (l∞)	0%*
   Ma et al. (2018)	CIFAR	0.031 (l∞)	5%
   Guo et al. (2018)	ImageNet	0.005 (l2)	0%*
   Dhillon et al. (2018)	CIFAR	0.031 (l∞)	0%
   Xie et al. (2018)	ImageNet	0.031 (l∞)	0%*
   Song et al. (2018)	CIFAR	0.031 (l∞)	9%*
   Samangouei et al. (2018)	MNIST	0.005 (l2)	55%**
   Madry et al. (2018)	CIFAR	0.031 (l∞)	47%

   • Table note: * indicates that the defense proposes to combine with adversarial training, and the table shows the numbers for the defense alone; ** indicates that in principle the accuracy of this defense should be 0%, and actual imperfection causes the theoretically optimal attack to fail (see §5.4.2). Na et al. (2018) has 15% accuracy on CIFAR under l∞ 0.015 (Table 1).
   • The authors say that of the 9 defenses, 7 rely on obfuscated gradients, 6 are completely bypassed, and 1 (Defense-GAN) is partially bypassed; Madry et al.'s adversarial training does not cause obfuscated gradients, and the authors' attack succeeds with some probability but does not overturn the original claim.
   • The authors caution that cross-dataset comparison is difficult: two defenses claim robustness on ImageNet, and one on MNIST.

   Attack details for shattered-gradient defenses

   • Thermometer encoding (Buckman et al.): the original defense reports 50% accuracy under LS-PGA (ε=0.031), and 80% after adversarial training; the authors use BPDA (g(x)=τ̂(x)) to drop it to 1% at ε=0.015 and 0% at ε=0.031 without adversarial training; for the thermometer-encoding + adversarial-training model, they reproduce the 80% claim under LS-PGA, but their own attack drops it to 30%, lower than the Madry model without thermometer encoding. Figure 1 shows that in the 0–0.03 distortion range, thermometer encoding alone has limited value, and combined with adversarial training it is instead worse than adversarial training alone (the authors say).
   • Input transformations (Guo et al.): use EOT for cropping and scaling, BPDA (identity approximation) for bit-depth reduction and JPEG compression, and EOT+BPDA for total variation minimization and image stitching; under the original paper's minimum perturbation budget (root mean square 0.05, normalized l2 0.01), the strongest defense reaches 100% targeted attack success rate and accuracy drops to 0%.

   Randomization and generative-model defenses

   • SAP (Dhillon et al.): the authors use EOT with k=10 to estimate gradients, and accuracy drops to 9% at ε=0.015 and 0% at ε=0.031; even requiring all 10 samples to be misclassified for success, accuracy at ε=0.031 is still below 10%.
   • Randomization (Xie et al.): the original authors' strongest attack drops InceptionV3 to 32.8% top-1; the authors use EOT to optimize over the distribution of discrete transformations, and even by the criterion that all 10 samples must be wrong for success, they still reach 100% targeted success rate, with accuracy dropping from 32.8% to 0.0% (l∞ ε=0.031).
   • PixelDefend (Song et al.): the original defense claims 46% accuracy at ε=0.031 (vanilla ResNet); the authors use BPDA for an end-to-end attack (replacing PixelCNN with PixelCNN++), dropping a naturally trained classifier with 95% accuracy to 9%; the authors report that PixelDefend combined with adversarial training is no more robust than adversarial training alone.

   LID detection and Defense-GAN (Appendices A, B)

   • LID (Ma et al.): the authors do not attack the detector directly, but generate high-confidence adversarial examples completely unaware of the detector (minimizing l∞), dropping model accuracy to 2% within ε=0.015; LID judges 97% of them as benign (the benign rate on unmodified test data is 98%). The authors say an adaptive attack that directly performs gradient descent on the LID loss gets stuck in local optima and fails.
   • Defense-GAN (Samangouei et al., MNIST): Evaluation A demonstrates that adversarial examples exist on the generator manifold—the reparameterization attack succeeds 100%, with average l2 distortion 0.0027 (0.0019 for the undefended classifier, 1.46×); Evaluation B uses BPDA against the actual implementation, dropping accuracy to 55% at normalized distortion 0.0051, and the main text separately states a success rate of 45%.

   Other ablations and analyses

   • SAP drop probability: the authors say the robustness numbers for different drop probabilities are similar.
   • BPDA replacement location: replacing in both forward and backward passes is completely ineffective against Song et al. and many times worse against Buckman et al. (§4.1.2).
   • Na et al. (cascaded adversarial training): the authors cannot reduce it below its claim, but point out that the claim is weak (16% accuracy at ε=0.015, whereas Madry adversarial training exceeds 70% under the same budget).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out remaining issues such as the attack on Defense-GAN only partially succeeding; the experiments are limited to three image datasets and nine defenses.

   Limitations and future directions noted by the authors

   • Defense-GAN is only partially bypassed: BPDA attack success rate is 45% (§5.4.2); the Table 1 note explains that in principle the accuracy of this defense should be 0%, and actual imperfection causes the theoretically optimal attack to fail.
   • Ilyas et al.'s extension is not evaluated: Ilyas et al. (2017), contemporaneous with this work, developed an almost identical method to Defense-GAN and combined it with adversarial training to improve robustness; the authors explicitly write "We do not evaluate this extended approach" (Appendix B).
   • Open question about LID: the authors say the results of this paper raise the question of whether a large LID is an essential feature of adversarial examples or a byproduct of certain attacks, and future work should carefully evaluate whether LID-based detectors are robust to adversarial examples explicitly targeting them (Appendix A).
   • Guo et al.'s threat model is retained: the authors relay that this defense may be stronger under a threat model where the attacker does not fully know the details of the stitching process (§5.2.2 footnote, from personal communication with the original authors).
   • Recommendations for defense evaluation (Section 6): define realistic threat models (at minimum grant knowledge of the model architecture, training algorithm, and query access; do not artificially limit the attacker's compute), give concrete testable claims (fully disclose hyperparameters, code, and pretrained models), and evaluate adaptive attacks (report the mean of the strongest attack per image).

   Scope of experiments

   • The defenses tested are the 9 non-certified white-box defenses accepted at ICLR 2018; 2 provable defenses and 1 black-box defense are skipped (Section 5).
   • The datasets are MNIST, CIFAR-10, and ImageNet; the models are a 5-layer CNN, Wide ResNet, and InceptionV3 (§2.3).
   • MNIST/CIFAR-10 use the entire test set for untargeted attacks; ImageNet uses 1000 random images for targeted attacks (§2.3).
   • Attacks iterate 100–10,000 times; the paper does not report the number of repetitions or variance for each experiment.
   • If a defense claims robustness on both MNIST and other datasets, the authors evaluate it only on the larger dataset (§2.3).

   Collapse this answer

6. 06Summarize the main content of the paper

   Obfuscated gradients make defenses appear robust; the authors use three attacks to bypass seven of the nine white-box defenses at ICLR 2018.

   An attack paper on the evaluation of adversarial example defenses, whose core point is that the apparent robustness of many defenses comes from obfuscated gradients—the defense makes gradients nonexistent, random, or vanishing/exploding, causing gradient-based iterative attacks to fail, rather than being truly robust.

   • Problem: the authors say that of the 9 non-certified white-box defenses at ICLR 2018, 7 rely on obfuscated gradients, which are a special case of the known-incomplete gradient masking.
   • Method: divide by cause into three categories and give one attack for each—shattered gradients use BPDA (normal forward pass, replace non-differentiable layers with a differentiable approximation in the backward pass); stochastic gradients use EOT (expected gradient over the distribution of random transformations); vanishing/exploding gradients use reparameterization to avoid differentiating through the unrolled computation. The base attacks are PGD (l∞) and Carlini & Wagner (l2).
   • Main results (Table 1, under each defense's original threat model): Buckman et al. (CIFAR, l∞ 0.031) accuracy drops to 0%; Guo et al. (ImageNet, l2 0.005) 0%; Dhillon et al. (CIFAR, l∞ 0.031) 0%; Xie et al. (ImageNet, l∞ 0.031) 0%; Song et al. (CIFAR, l∞ 0.031) 9%; Ma et al.'s LID detection is judged benign by high-confidence adversarial examples at a 97% rate; Defense-GAN (MNIST, l2 0.005) drops to 55%, which the authors say counts only as a partial bypass.
   • Defenses not broken: Madry et al.'s adversarial training (CIFAR, 47% accuracy under l∞ 0.031) does not cause obfuscated gradients; the authors cannot reduce Na et al.'s cascaded adversarial training below its claim, but point out that the claim itself is weak.
   • The authors' conclusion: defenses must withstand future attacks rather than only existing attacks; the authors recommend defining realistic threat models, giving concrete testable claims, evaluating adaptive attacks, and have released reproduction and attack code for each defense.

   Collapse this answer

Related papers