Defensive distillation is not robust to adversarial examples
Defensive Distillation is Not Robust to Adversarial Examples
The authors state that defensive distillation cannot resist adversarial examples: after improving Papernot's L0 attack, they achieve 96.4% targeted misclassification on the T=100 distilled network on MNIST.
Targeted misclassification attack: the attacker seeks an adversarial example close to the original image in L0 distance (few changed pixels) that makes the network classify it as a specified target class. The attack uses network gradients (differentiating with respect to logits or the softmax output); the victim system is a 9-layer feedforward neural network on MNIST (an undistilled baseline and a T=100 defensively distilled network).
- Defensive distillation was proposed to make feedforward neural networks resist adversarial examples; the authors test whether it can truly resist targeted misclassification attacks.
- The authors analyze why Papernot's L0 attack fails on the distilled network: distillation magnifies the logits by roughly a factor of T, causing vanishing gradients. The improved attack instead takes gradients with respect to the softmax output, artificially divides the logits by T before computing, and switches to α−β for pixel selection.
- On MNIST, the improved attack achieves a 96.4% targeted misclassification success rate on the T=100 distilled network, changing on average 36.4/768 (4.7%) pixels; on the undistilled network the success rate is 86%, changing 45 pixels on average.
- The experiments were conducted only on MNIST and one 9-layer network architecture, with the attack being a customized modification targeting distillation. The authors suggest that defenders analyze the reasons for attack failure and evaluate attacks specific to this defense, rather than only testing existing attacks.
- Evaluated models
- distilled network (defensive distillation, T=100)
- Benchmarks
- MNIST
- Metrics
- targeted misclassification success ratenumber of pixels changed (L0)
Nicholas Carlini and David Wagner published a paper showing through experiments that defensive distillation is not secure; with slight modifications to standard attacks, adversarial examples can be found on defensively distilled networks, and its resistance to targeted misclassification attacks is no stronger than that of an unprotected neural network. Defensive distillation amplifies the magnitude of the softmax layer input by a factor of T on average, rendering the unmodified Papernot attack ineffective (success rate only 0.45% at T=100). The authors instead take gradients with respect to the network's actual output, artificially divide the softmax input by T to avoid vanishing gradients, and select modification points pixel by pixel by the maximum value of α−β. On the MNIST handwritten digit task, against a defensively distilled network with T=100, the attack achieves a 96.4% targeted misclassification success rate by modifying 4.7% of pixels on average (36.4 out of 768 pixels), whereas the same attack achieves an 86% success rate against an undistilled standard network.
Why it was selectedThe paper shows that slight modifications to standard attacks can bypass defensive distillation, suggesting that defense designers should analyze the specific reasons why attacks fail rather than only testing known attacks.
Paper analysis
What problem does this paper attempt to solve?
Test whether defensive distillation can resist adversarial examples: the authors state that after improving the existing attack, the distilled network is no longer more robust than the unprotected network.
Core question: Does defensive distillation actually make neural networks resistant to adversarial examples? The authors' answer is no.
- Problem setting: Adversarial examples are inputs very close to normal samples that are nonetheless misclassified by the network; making neural networks robust to adversarial examples is an open problem. Defensive distillation (Papernot et al. [5]) is a defense proposed for this purpose.
- Shortcomings of existing evaluations: Distillation increases the magnitude of the softmax layer's inputs (logits), causing the unmodified attack to fail—the authors state that Papernot's attack achieves only a 0.45% success rate on the T=100 distilled network. The authors believe this failure may be due only to superficial reasons.
- Core observation: Training at temperature T and testing at temperature 1 magnifies the logits by roughly a factor of T, causing vanishing gradients and attack failure; artificially shrinking the logits restores the attack.
- Paper's contribution: Small modifications to Papernot's L0 attack restore its effectiveness on the distilled network, verified on MNIST.
- Threat model:
- Attacker's goal: Targeted misclassification—find x′ such that the network classifies x′ as a specified target class t, with x′ very close to x.
- Distance metric: L0 distance, i.e., few changed pixels.
- Attacker's capability: The attack uses network gradients (differentiating with respect to logits or the softmax output); the paper does not explicitly use terms such as white-box.
- Victim system: A 9-layer feedforward neural network on the MNIST digit recognition task, including an undistilled baseline and a T=100 distilled network.
What related research exists?
The paper unfolds along three lines: adversarial examples, Papernot's L0 attack, and distillation as a defense; this paper is an offensive test of that defense.
Adversarial examples
- Szegedy et al. [6] (ICLR 2013)—proposed the adversarial example phenomenon: instances x′ that are very close to normal samples but are misclassified by the network. This paper follows their definition of adversarial examples.
Attack methods
- Papernot et al. [4] (EuroS&P 2016)—proposed a targeted attack for L0 distance: iteratively greedily select pixel pairs, using network gradients (differentiating with respect to logits) to estimate pixel importance, until classified as the target class or more than 112 pixels have been changed. This paper's attack is a modification of it.
Defense methods
- Papernot et al. [5] (IEEE S&P 2016)—proposed defensive distillation: train a teacher network at temperature T to produce soft labels, then train a distilled network. The paper relays their results: at T=100 the Papernot attack's success rate drops to 0.45%, and the pixels needed to change the classification rises from 2% to 14%.
Baseline and benchmark
- Baseline: the undistilled standard network (same 9-layer architecture), tested with the same improved attack; the paper notes this is not a fully fair comparison, because the modifications were made for the distilled network.
- Benchmark: the MNIST handwritten digit dataset [2].
The authors position this paper as a case study of the specific defense of defensive distillation: by making small modifications to an existing attack, they show that "resisting existing attacks" does not mean the defense is effective.
How does the paper address this problem?
Analyze why distillation makes the Papernot attack fail (logits magnified by a factor of T, vanishing gradients), then make three small modifications to the attack to restore its effectiveness.
Overall approach: first explain the mechanism by which defensive distillation makes Papernot's L0 attack fail, then modify the attack accordingly. The method is not separately named; it can be summarized as "an improved L0 attack that takes gradients with respect to the softmax output and artificially shrinks the logits."
Background: distillation and its side effects
- Three steps of distillation: Train a teacher network at temperature T (softmax(Z/T)); use the teacher to produce soft labels on the training set; train the distilled network at temperature T on the soft labels, using T=1 at classification time.
- Logit magnification: Training at temperature T and testing at temperature 1 makes the absolute value of the logits roughly T times larger. The authors report experimental verification: the mean L1 norm of the logits of the undistilled network is 5.8 (standard deviation 6.4), and that of the T=100 distilled network is 482 (standard deviation 457).
Why the attack fails on the distilled network
- Recap of the Papernot attack: Iteratively greedily select pixel pairs (p*, q*), scoring with α_pq (gradient with respect to the target class logits) and β_pq (sum of gradients with respect to the other classes' logits), choosing the pixel pair with the largest −α·β and α>0, β<0, setting it fully on or fully off; until classified as the target class or more than 112 pixels have been changed.
- Failure mechanism: The attack takes gradients with respect to the logits and treats all changes equally; after distillation magnifies the logits by a factor of T, the authors state this amplifies the optimality flaw of that approximation, making the attack "fail thoroughly."
Three modifications to the attack
- Take gradients with respect to the output: Instead of taking gradients with respect to the softmax input (logits), take gradients with respect to the network's actual output.
- Artificially shrink the logits: Define F̂(θ, x) = softmax(Z(θ, x)/T), i.e., divide the logits by T before computing gradients, solving the vanishing gradient; the meaning is to shrink the artificially magnified logits back to a comparable scale.
- Change the pixel selection criterion: Replace the original −α·β product with the maximum of α−β; the authors state that searching a single pixel at a time (rather than pixel pairs) does not reduce the success rate and is 768 times more efficient.
Experimental setup and success criterion
- Implementation: TensorFlow reproduction of distillation and the attack; the same 9-layer architecture as Papernot et al., with a slightly smaller learning rate; MNIST dataset.
- Temperature choice: T=100, which the authors state is the hardest to attack among the temperatures proposed in the original distillation paper; and they verify the attack is effective for all T from 1 to 100.
- Success criterion: Targeted misclassification succeeds when the network classifies the modified image as the specified target class; success rate and number of changed pixels are recorded.
What experiments did the paper conduct?
On MNIST, the improved attack has a success rate of 96.4% on the T=100 distilled network (average 4.7% of pixels modified), and 86% on the undistilled network.
Experimental setup
- Networks under test: two models with the same 9-layer architecture—an undistilled standard network (baseline) and a T=100 defensive distillation network.
- Dataset: MNIST handwritten digit recognition.
- Clean accuracy: baseline network 99.4%, distilled network 99.1%.
- Attack method: the paper's improved Papernot L0 attack (gradient with respect to softmax output, logits divided by T, α−β pixel selection, pixel-by-pixel search).
- Metrics: targeted misclassification success rate; number of modified pixels (L0).
- Sample size and number of repetitions: the paper does not report the number of images used for the attack or the number of repetitions.
Main results
Target network Attack success rate Average modified pixels Distilled network (T=100) 96.4% 36.4/768 (4.7%) Undistilled standard network 86% 45 pixels (According to Section 3 of the main text; the paper has no results table)
- The authors state that the distilled network is not more secure against adversarial examples than the undistilled network.
- The authors note that the baseline comparison is not entirely fair: the attack's modifications were designed for the distilled network.
- As a control, the paper restates that the unmodified Papernot attack has a success rate of only 0.45% on the T=100 distilled network.
Distribution of pixel counts
- Figure 1: cumulative distribution function (CDF) of the number of modified pixels required for a successful attack, with the horizontal axis being the number of modified pixels (0–100) and the vertical axis being success probability (0–100%); the pdftotext text contains only axis ticks, and the specific curve shape cannot be read from the text.
- The main text reports an average of 36.4 modified pixels (4.7%).
Validation of the temperature range
- What was tested: whether the improved attack is effective only for T=100.
- Result: the authors state that the attack is effective for all T settings from 1 to 100, the same range studied in the original distillation paper; specific numbers for each T are not given.
Other ablations and analyses
- Logit magnitude measurement: the mean L1 norm of logits for the undistilled network is 5.8 (standard deviation 6.4); for the T=100 distilled network the mean is 482 (standard deviation 457), used to support the explanation that "logits are amplified by about a factor of T."
- Pixel-by-pixel vs pixel pairs: after switching to the α−β criterion, pixel-by-pixel search does not reduce the success rate, and the authors state it is 768 times more efficient; numbers for each of the two settings are not given.
- Negative results/exceptions: the success rate on the undistilled network (86%) is lower than on the distilled network (96.4%); the authors do not explain the reason and only note that the comparison is not entirely fair.
What are some points that can be further explored?
The authors did not list limitations, but suggest that defense research analyze the reasons for attack failure and evaluate attacks specific to this defense; the experiments cover only the single MNIST architecture.
Limitations and future directions noted by the authors
- How to evaluate defenses (Conclusion): the authors believe that merely showing a defense can block existing attacks is insufficient; one should also analyze why the attack fails and whether the defense is effective against attacks specifically targeting it.
- Seeking impossibility arguments (Conclusion): the authors encourage defense designers to seek arguments that "existing attacks cannot be adapted"; they also acknowledge that it is impossible to test against all future attacks.
- Warning about suboptimal attacks (Conclusion): the authors point out that Papernot's attack does not claim to be optimal, and defending against a suboptimal attack does not mean being able to defend against other attacks.
- Open problem (Conclusion): the authors state that defending against adversarial examples remains an open problem in the field.
- The paper has no dedicated Limitations section; the above are advisory statements in the Conclusion.
Scope of experimental coverage
- The networks under test are two models with the same 9-layer architecture: an undistilled baseline and a T=100 distilled network.
- The dataset is only MNIST; the attack type is only the targeted misclassification L0 attack (an improved version of the Papernot attack).
- Temperature coverage is 1 to 100, but specific numbers are not reported except for T=100.
- The paper does not report the number of image samples used for the attack, the number of repetitions, or statistical significance tests.
- The paper does not evaluate other attack algorithms or other defenses.
Summarize the main content of the paper
A 4-page attack short paper: small modifications to Papernot's L0 attack suffice to break defensive distillation, with a 96.4% targeted misclassification success rate on MNIST.
This is an offensive case study targeting defensive distillation, by Carlini and Wagner.
- Problem: Defensive distillation was proposed to resist adversarial examples; the authors test whether it can truly resist targeted misclassification attacks.
- Mechanism analysis: Training at temperature T and testing at temperature 1 magnifies the logits by roughly a factor of T (measured mean L1 norm rises from 5.8 to 482), causing the logits-gradient-based Papernot attack to fail due to vanishing gradients—the authors consider this only a superficial reason.
- Method: Three modifications to the attack—take gradients with respect to the softmax output, artificially divide the logits by T before computing, and select pixels one by one using the α−β criterion.
- Main results: On MNIST, the improved attack achieves a 96.4% targeted misclassification success rate on the T=100 distilled network, changing on average 36.4/768 (4.7%) pixels; on the undistilled network the success rate is 86% (45 pixels on average); the authors state the attack is effective for all temperatures from 1 to 100.
- Authors' conclusion: Defensive distillation is no more secure against adversarial examples than an unprotected network; when evaluating a defense one should understand why attacks fail and test attacks specific to that defense—resisting existing (possibly suboptimal) attacks does not mean the defense is effective.
Related papers
Both use: MNIST
Show 3 more papers
Both use: MNIST