Distillation as a defense against adversarial perturbations of deep neural networks
Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks
The authors propose defensive distillation, distilling a DNN at high temperature with the same architecture; the authors report that the adversarial example crafting success rate on MNIST drops from 95.89% to 0.45% (T=100).
The attack occurs at test time (training-time attacks are not considered), and the goal is source-target misclassification (chosen target attack): adding a perturbation to samples of any source class so that they are classified into a specified target class. The attacker can access the trained classification DNN (directly accessing the architecture and parameters, or approximately copying the model as an oracle); the paper considers this assumption acceptable because adversarial examples are transferable.
- DNNs are vulnerable to adversarial example attacks: adding a small perturbation to the input can make the model output a class specified by the attacker, threatening safety-sensitive scenarios such as autonomous driving and content filtering, while existing defenses require architecture changes or only partially take effect.
- Proposes defensive distillation: first train an initial network with a softmax temperature T>1, then use the probability vectors it outputs as soft labels and train a distilled network with the same architecture and temperature, with the temperature lowered back to 1 at test time, thereby smoothing the model and suppressing adversarial gradients.
- On two 9-layer CNNs for MNIST and CIFAR10, the authors report the adversarial example crafting success rate drops from 95.89% to 0.45% and from 87.89% to 5.11% respectively (T=100), accuracy changes by no more than 1.37%, and the robustness metric improves by 790% and 556% respectively.
- The authors note that the defense applies only to models that can produce energy-based probability distributions and for which a temperature can be defined; it is not compared with traditional regularization methods; only one attack algorithm was tested, and the defense's effectiveness against other attacks such as L-BFGS and FGSM is left to future work.
- Evaluated models
- F_MNIST (MNIST architecture DNN)F_CIFAR10 (CIFAR10 architecture DNN)
- Benchmarks
- MNISTCIFAR10
- Metrics
- 对抗样本制作成功率分类准确率对抗梯度平均幅度鲁棒性 ρ_adv(平均最小扰动特征比例)预测置信度 C(X)
Papernot et al. propose defensive distillation, which on the DNNs studied reduces the success rate of adversarial example construction from 95% to below 0.5%. The authors attribute this effect to distillation shrinking the gradients used to construct adversarial examples by a factor of 10^30; on one DNN tested, the minimum number of features that must be modified to construct an adversarial example increases by about 800% on average. The paper theoretically analyzes the generalization and robustness brought by defensive distillation, and empirically tests the defense on two DNNs placed in an adversarial setting. The paper's motivation is that adversarial examples can compromise the security of systems supported by DNNs, for example causing autonomous vehicles to crash, illegal content to bypass content filtering, or biometric authentication to be manipulated to allow unauthorized access.
Paper analysis
What problem does this paper attempt to solve?
Against the threat of test-time adversarial examples to DNN classifiers, proposes defensive distillation that does not modify the architecture and does not reduce accuracy.
Core problem: DNN classifiers are vulnerable to adversarial example attacks at test time, requiring a defense mechanism that does not modify the architecture and does not sacrifice accuracy.
- Problem scenario: the authors point out that adversarial examples are carefully chosen small perturbations superimposed on legitimate inputs that make the DNN output a result chosen by the attacker, with the perturbation small enough that humans can still correctly process the sample; in safety-sensitive scenarios such as autonomous driving, content filtering, and biometrics, the consequences are severe.
- Shortcomings of existing methods: the authors state that existing defense schemes have flaws—either they require modifying the DNN architecture, or they can only partially prevent adversarial examples from taking effect (references [9], [17]).
- Core observation: crafting adversarial examples relies on estimating the gradient of the network's sensitivity to each input dimension (which the authors call the adversarial gradient); the larger the gradient magnitude, the easier it is for a small perturbation to cause an output change. Therefore the defense idea is to smooth the model learned during training and reduce the adversarial gradient magnitude.
- Method proposed in the paper: defensive distillation—changing distillation from cross-architecture compression to feeding back knowledge as probability vectors on the same architecture, to improve robustness to perturbations; the paper also provides an analysis based on learning theory and an empirical evaluation on two DNNs.
- Threat model:
- Attacker goal: source-target misclassification (also called chosen target attack), i.e., perturbing samples of any source class into a specified target class.
- Attacker knowledge: can access the trained classification DNN—either directly accessing the architecture implementation and parameters, or accessing the network as an oracle and approximately copying the model; the authors consider this assumption acceptable because of the transferability of adversarial examples.
- Attack timing: test-time attack; the paper explicitly does not consider training-time attacks, leaving them for future work.
- Victim system: a DNN classifier with softmax as the output layer.
What related research exists?
Related work is divided into four groups: adversarial example attacks, distillation techniques, existing defense attempts, and machine learning security and stability theory.
Adversarial example attacks (source of this paper's attack framework)
- Szegedy et al. [8] and Goodfellow et al. [9]—found that DNNs are easily fooled by adversarial examples; [9] proposed the fast sign gradient method, adding a small perturbation along the gradient sign direction to all input dimensions (minimizing Euclidean distance).
- Papernot et al. [7]—proposed the forward derivative (Jacobian) plus saliency map method, perturbing only a few input dimensions; the attack implemented in this paper's experiments is this method.
- Miyato et al. [29]—proposed the KL-divergence-based Local Distribution Smoothness sensitivity measure, but the authors point out that it does not give adversarial example crafting results, but is used as a training regularizer.
Distillation technique (basis of this paper's method)
- Ba and Caruana [18] proposed the intuition of knowledge transfer, and Hinton et al. [19] formally proposed distillation: training a smaller network with the probability vectors output by a large network under high-temperature softmax, to reduce computational cost and facilitate deployment to resource-constrained devices. This paper changes the goal of distillation from compression to defense, and keeps the same architecture for both training runs.
Existing defense attempts (the paper explicitly criticizes their shortcomings)
- Goodfellow et al. [9]—radial basis activation functions are more resistant to perturbation, but the authors point out that deploying them requires substantial modification of existing architectures.
- Gu and Rigazio [17]—use a denoising autoencoder to remove adversarial noise, but the authors point out that their stacked architecture can still be bypassed by adversarial examples; their subsequent Deep Contractive Networks use a Jacobian penalty, but the authors point out that this penalty limits network capacity.
- Warde-Farley and Goodfellow [38]—inspired by an earlier preprint of this paper, independently tested label smoothing and found that it can partially resist adversarial examples crafted by FGSM; the authors' explanation is that label smoothing is sufficient to defend against simple, cheap attack methods, but not against the stronger iterative method used in this paper.
Machine learning security and theoretical tools
- Machine learning security and attack taxonomy: Barreno et al. [39], Huang et al. [12], Biggio et al. [42] (a security evaluation framework for binary classifiers such as SVM, not involving deep learning), etc.
- Theoretical tools: Fawzi et al. [30]'s definition of classifier robustness (which this paper extends); Shalev-Schwartz et al. [34]'s theorem on the equivalence of learnability and stability (borrowed in this paper's analysis).
Baselines and benchmarks: the experimental baseline is a model of the same architecture trained without distillation (two 9-layer CNNs, MNIST and CIFAR10); the datasets used are MNIST [20] and CIFAR10 [21].
The authors position this paper as: previous attack papers only gave defense suggestions and left verification for future work, whereas this paper proposes and fully evaluates a defense mechanism.
How does the paper address this problem?
Defensive distillation: on the same architecture, retrain with probability-vector soft labels under high-temperature softmax, smoothing the model and lowering the adversarial gradient.
Overall idea: adapt distillation from cross-architecture compression into same-architecture defensive training—use the probability vectors output by the first network as soft labels to train a second network of the same architecture, leveraging the inter-class relative information in the soft labels to make the model generalize better around training points, thereby reducing its sensitivity to input perturbations. The method is named defensive distillation.
Problem formalization and defense requirements
- Adversarial examples are formalized as an optimization problem: argminδ X |δ X| s.t. F(X+δ X) = Y∗, i.e., minimizing the perturbation subject to making the model output the target class Y∗.
- Robustness is defined as ρadv(F) = Eμ[Δadv(X,F)], i.e., the expected minimum perturbation required for a sample to be misclassified into each of the other classes; the larger this value, the more robust the model (Figure 4 gives an intuitive illustration).
- The authors propose four defense design requirements: small changes to the architecture; no reduction in classification accuracy (excluding L1/L2 weight decay, on the grounds that it causes underfitting); no significant impact on test-time speed (the authors note that Jacobian regularization such as double backpropagation harms training performance); effective against adversarial examples close to the training set.
Defensive distillation procedure
- Input a training set with hard labels (one-hot indicator vectors).
- Train a network F with a softmax output layer on this training set, with softmax temperature T (requiring T greater than 1).
- Replace the hard labels with the probability vector F(X) output by F for each training sample, forming a new training set (X, F(X)).
- Train the distilled network F^d with the new training set, the same architecture, and the same temperature T; at test time the temperature is set back to 1.
- Key difference from Hinton distillation: the original network and the distilled network use the same architecture, because the goal is to improve robustness rather than compress the model.
- Design rationale (per the authors): soft labels carry more inter-class relative information than hard labels (for example, the structural similarity of 7 and 1), preventing the model from fitting the training data too tightly and improving generalization around training points.
Theoretical analysis
- Effect on training: hard-label training constrains only the correct-class neuron, forcing the model to make overconfident predictions; soft labels make training constrain all output neurons in proportion to the class likelihoods. The authors argue that the distilled model should in theory converge to F (cross-entropy is minimized when F^d(X)=F(X)), but because optimization is non-convex and samples are finite, in practice F^d performs better in adversarial settings.
- Effect on sensitivity: the authors derive the softmax layer Jacobian components, noting that their absolute values are inversely proportional to temperature T, and that the logits are first divided by T before exponentiation, so raising the temperature systematically reduces the model's sensitivity to small input changes; the authors believe that lowering the temperature back to T=1 at test time does not affect the smoothness already encoded in the weights, but acknowledge that further formal analysis is needed and leave it to future work.
- Effect on generalization: drawing on the stability theorem of Shalev-Schwartz et al., the authors argue that with an appropriate setting of T, distilled training satisfies the stability condition and minimizes empirical risk, so the distilled model generalizes well; at the same time they explicitly state that they have not rigorously proven that the distilled model generalizes better than the non-distilled model.
Attack implementation and success criterion
- The attack implementation used for evaluation is from [7]: compute the Jacobian of the network output with respect to the input, rank input features with a saliency map, and perturb first the features more likely to change the output; perturbed features are set to 1 on MNIST and to 2 on CIFAR10; two steps (direction sensitivity estimation, perturbation selection) are iterated until the sample is classified into the target class.
- The upper limit on the number of perturbed features is 112, on the grounds that larger perturbations would be noticed by humans or anomaly detection systems; success is judged as the sample being classified into the target class specified by the attacker.
What experiments did the paper conduct?
Attack success rate, accuracy, gradient magnitude, robustness, and confidence are measured on two 9-layer CNNs, MNIST and CIFAR10.
Experimental setup
- Models under test: two 9-layer CNN architectures (Table I). MNIST architecture: 32+32 convolution, pooling, 64+64 convolution, pooling, two 200-unit fully connected layers, 10-class softmax; CIFAR10 architecture: 64+64, 128+128 convolution, two 256-unit fully connected layers. Each architecture has an original model and distilled models at different temperatures.
- Training parameters (Table II): MNIST learning rate 0.1, momentum 0.5; CIFAR10 learning rate 0.01, momentum 0.9 (both with decay); dropout 0.5, batch 128, 50 epochs. Original model accuracy: MNIST 99.51%, CIFAR10 80.95%.
- Datasets: MNIST (70,000 images, 60,000 training / 10,000 test); CIFAR10 (60,000 images, 50,000 training / 10,000 test).
- Attack: implements the iterative saliency map attack of [7], with an upper limit of 112 perturbed features; the authors report that this attack previously achieved a 97% success rate on MNIST with an average of 4.02% feature perturbation, and in this paper it achieves 92.78% on the CIFAR10 test set.
- Metrics and sample size: the adversarial example crafting success rate is measured on 100 random test samples × 9 target classes = 900 samples per model; accuracy is measured on all 10,000 test samples; gradient magnitude and confidence are measured on all 10,000 CIFAR10 test samples.
- Distillation temperature: T takes {1, 2, 5, 10, 20, 30, 50, 100}; at test time the temperature is set back to 1. The paper does not report the number of repetitions.
Main results
Adversarial example crafting success rate (900 targets, perturbation limit 112 features; per main text V-B and Figure 7):
Swipe horizontally to read the table
Model No distillation Distillation (main text T=20) Distillation T=100 MNIST architecture 95.89% 1.34% 0.45% CIFAR10 architecture 89.9% (elsewhere in the main text 87.89%) 16.76% 5.11% - The authors state that distillation reduces the success rate by 98.6% on MNIST and by 81.36% on CIFAR10 (T=20 setting).
- The authors state that increasing temperature makes adversarial example crafting more difficult, and that there is an elbow point after which the success rate is roughly flat (about 0% on MNIST, about 5% on CIFAR10); the authors verify the elbow hypothesis using the CIFAR10 average maximum probability: 0.72 at T=1, 0.14 at T=20, 0.11 at T=40, approaching 1/N=0.1.
- Note: the CIFAR10 no-distillation baseline given in two places in the main text is inconsistent (89.9% and 87.89%), and the T=20 and T=100 numbers correspond to different passages respectively.
Effect on accuracy
- Accuracy change before and after distillation (Figure 8, 10,000 test samples): the MNIST model's accuracy drops by no more than 1.28% at all temperatures (e.g., 99.05% at T=20); the CIFAR10 model drops by at most 1.37% (81.39% at T=20), and at some temperatures the change is positive.
- The authors consider the accuracy change mild, that the defense meets the design requirement of "not reducing accuracy," and that the positive change on CIFAR10 may come from the generalization ability brought by distillation.
Gradient magnitude and robustness
- Gradient magnitude (Figure 9, CIFAR10 10,000 test samples): without distillation, 4,763 samples have an average adversarial gradient magnitude greater than 0.001, whereas with T=100 distillation only 172 do; samples with magnitude less than 10^-40 increase from 8 to 7,908. The authors state that the factor by which gradient magnitude is reduced is greater than 10^30.
- Robustness (Figure 10, approximated with perturbed feature proportion as distance, on 10,000 test samples × 9 targets): the MNIST model rises from 1.55% (no distillation) to 13.79% (T=20), which the authors state is an increase of 790%; the CIFAR10 model rises from 0.39% to 2.56% (T=50), which the authors state is an increase of 556%. The authors point out that a perturbation of 13.79% is already large enough to possibly change the true class or be detected by anomaly detection.
- Confidence (Table III, CIFAR10): the average confidence C(X) rises monotonically from 71.85% at T=1 to 81.06% at T=20; the authors state that distillation improves prediction confidence, and point out that on MNIST no conclusion can be drawn because confidence is already close to 99%.
Other ablations and analyses
- Soft label substitution: training the MNIST model with soft labels (0.9 for the correct class, 1/(10N) for each incorrect class) instead of probability vectors yields an adversarial example misclassification rate of 86.00%, whereas the distilled model is less than 1%; the authors believe the reason is that probability vectors encode relative information between classes whereas soft labels do not.
- Overfitting: the authors state that under overfitting the adversarial gradient magnitude gradually rises again, and that 50 epochs is sufficient to avoid gradient "explosion."
- Temperature selection: the authors state that deployment requires an empirical trade-off between robustness and accuracy, and that the MNIST model can choose T=20 according to Figures 7 and 8.
What are some points that can be further explored?
The authors point out limitations such as applicability and attack coverage; the experiments cover only two datasets, one attack, and two CNN architectures.
Limitations and future directions noted by the authors
- Limited applicable models (Discussion): defensive distillation applies only to models that can produce energy-based probability distributions and for which a temperature can be defined; applying it to machine learning models other than DNNs requires additional research.
- No comparison with traditional regularization (Discussion): this paper does not compare the defense with traditional regularization techniques such as dropout and weight decay; the authors' stated reason is that adversarial examples are not a traditional overfitting problem.
- Attack coverage (Discussion): the defense is effective against the attack used in this paper, but may still be vulnerable to attacks based on L-BFGS, the fast gradient sign method, and genetic algorithms; the authors cite preliminary results in [38] suggesting distillation may also defend against such techniques, leaving this to be explored.
- Distance metric (Discussion): this paper uses the number of perturbed features as the distance; future work should study other metrics such as L1 and L2, especially in domains beyond vision such as malware.
- Theoretical analysis (Section IV-B): the explanation for why lowering the temperature at test time does not affect sensitivity requires further formal analysis, which the authors plan to advance in future work.
- Extension directions (Conclusion): future work should evaluate the effect of distillation on other DNN models and other adversarial example crafting algorithms, extend to tasks beyond classification (which requires finding a substitute for probability vectors), and explore other definitions of robustness.
Scope of experimental coverage
- The models tested are two 9-layer CNN architectures (one each for MNIST and CIFAR10), each architecture including the original model and distilled models at 8 temperatures.
- The attack evaluation covers only one attack, the iterative saliency map attack of [7], with a perturbation limit of 112 features.
- Success rate measurements are based on 100 random test samples per model × 9 target classes (900 in total); the gradient, robustness, and confidence experiments are based on 10,000 test samples.
- The paper does not report the number of experimental repetitions, nor does it report statistical significance tests.
- The gradient magnitude and confidence experiments were conducted only on the CIFAR10 model; for MNIST, the authors state that the confidence analysis could not reach a conclusion because the values were already close to 99%.
Summarize the main content of the paper
Defensive distillation: high-temperature distillation with the same architecture smooths the DNN; the authors report the adversarial example success rate drops substantially while accuracy remains almost unchanged.
A defense paper: against source-target misclassification attacks on DNN classifiers by test-time adversarial examples, it proposes defensive distillation—first train an initial network with a softmax at temperature T>1, then use its probability vectors as soft labels and train a distilled network with the same architecture and temperature, with the temperature lowered back to 1 at test time. The authors' theoretical analysis holds that the inter-class relative information carried by soft labels improves generalization, and that softmax Jacobian components are inversely proportional to temperature, so high-temperature training systematically reduces the model's sensitivity to input perturbations.
Main experimental results (two 9-layer CNNs, attack is the saliency map method of [7], 900 targets):
- Attack success rate: MNIST model from 95.89% (no distillation) down to 0.45% (T=100); CIFAR10 model from 87.89% down to 5.11% (T=100).
- Accuracy cost: accuracy change of the two models before and after distillation does not exceed 1.37%, and at some temperatures CIFAR10 even shows a positive change.
- Gradients and robustness: the authors report the adversarial gradient magnitude is reduced by a factor greater than 10^30; the robustness metric (average minimum perturbation feature proportion) on MNIST rises from 1.55% to 13.79% (T=20, an increase of 790%), and on CIFAR10 from 0.39% to 2.56% (T=50, an increase of 556%).
- Ablation: when soft labels are replaced by probability vectors, the adversarial example misclassification rate on MNIST is 86.00%, far higher than the less than 1% of the distilled model.
The authors' conclusion: defensive distillation is simple to implement, has low training overhead, does not modify the architecture and does not reduce accuracy; the authors call it a new foundation for protecting deep learning-based systems. The authors also note that it applies only to models with probability outputs for which a temperature can be defined, and leave the defense's effectiveness against other attack algorithms and other tasks to future work.
Related papers
Show 3 more papers
- The Space of Transferable Adversarial ExamplesAnalysis / interpretabilityarXiv