Adversarial examples are not easy to detect: bypassing ten detection methods

1. 01What problem does this paper attempt to solve?

   Are the ten adversarial example detection methods effective? The authors state that all can be bypassed by targeted attacks.

   Core question: can recently proposed adversarial example detection methods actually detect adversarial examples, and do adversarial examples have intrinsic properties that distinguish them from natural images.

   • Background: neural networks are easily fooled by adversarial examples, and existing defenses that attempt to correctly classify adversarial examples are mostly ineffective, so recent research has turned to detecting and rejecting adversarial examples.
   • Shortcomings of existing methods: the authors consider these detection schemes to lack rigorous security evaluation, most being tested only with weaker attacks such as fast gradient sign and JSMA, and mostly evaluated only on MNIST.
   • Core hypothesis: the authors question the hypothesis that "adversarial examples have intrinsic differences from natural images," and consider that the observed differences may come from dataset artifacts or the attack method rather than intrinsic properties.
   • Work of the paper: reproduce ten detection methods from seven papers, evaluate them under a unified framework with three threat models, and construct a dedicated attacker loss function for each defense to carry out white-box attacks.
   • Threat models (using the paper's terminology):
     • Zero-Knowledge: the attacker generates adversarial examples on the undefended model F without knowing that the detector D exists.
     • Perfect-Knowledge: the attacker knows the detection scheme D and its model parameters and can attack F and D simultaneously (white-box adaptive).
     • Limited-Knowledge: the attacker knows the defense type and training method but does not have access to the trained detector D or the exact training data, and relies on a substitute model for transfer attacks.
     • Victim systems: image classification networks and their detectors on MNIST and CIFAR-10.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three categories: adversarial example generation and defenses, the ten evaluated detection methods, and attack algorithms.

   Adversarial examples and existing defenses

   • Biggio et al. (2013) and Szegedy et al. (2014)—demonstrated test-time evasion attacks on neural networks, generating visually similar images with different labels.
   • Defenses aimed at correct classification (Bastani 2016, Gu 2014, Papernot 2016 distillation, etc.)—the authors state that no defense so far can correctly classify adversarial examples, so research has turned to detection.

   The eleven evaluated detection methods (seven papers)

   • Binary classification: Grosse et al. (2017) adversarial retraining plus an N+1 class, Gong et al. (2017) training a binary classifier to distinguish natural from adversarial examples, Metzen et al. (2017) attaching a detection network to an intermediate convolutional layer of the classification network.
   • PCA-based: Hendrycks & Gimpel (2017) PCA on input images, Bhagoji et al. (2017) dimensionality reduction followed by training a classifier, Li & Li (2016) PCA on convolutional layer outputs plus a cascaded SVM.
   • Distribution statistics and normalization: Grosse et al. (2017) MMD statistical test, Feinman et al. (2017) kernel density estimation (KDE) and dropout uncertainty, Li & Li (2016) 3×3 mean blur.

   Attack algorithms (baseline attacks)

   • Evaluation mainly uses the Carlini & Wagner (2017) L2 attack, which the authors consider superior to other published attacks; fast gradient sign (Goodfellow 2014) and JSMA (Papernot 2016) are used only as weak attacks by the defenders, and the authors do not use them for evaluation.
   • Transferability (Szegedy 2014, Papernot 2016)—the black-box Limited-Knowledge attack relies on this property.

   Positioning: the authors state that this paper is the first work to make a unified comparison of these ten detection methods and evaluate each with a targeted white-box attack; the datasets are MNIST and CIFAR-10.

   Collapse this answer

3. 03How does the paper address this problem?

   Reproduce ten detection methods and attack them one by one under three threat models using the C&W attack plus a loss function customized for each defense.

   Overall approach: combine each detector with the classifier into one differentiable objective, and use a strong iterative attack to fool both simultaneously. The attack framework is based on the C&W L2 attack; the key is to construct a suitable attacker loss function for each defense.

   Attack basics and success criterion

   • C&W L2 attack: minimize |x′ − x|₂² + c · ℓ(x′), where ℓ(x′) = max(max{Z(x′)_i : i ≠ t} − Z(x′)_t, −κ); κ controls confidence, c is chosen by binary search, and the loss acts on logits Z rather than probabilities F.
   • Robustness measure: the L2 distance to the nearest adversarial example (average perturbation) measures the difficulty added by the defense; success means the generated adversarial example is both misclassified and not detected by the detector.

   Evaluation procedure under three threat models

   • Zero-Knowledge: directly use the C&W attack to generate adversarial examples and see whether the defense can detect them; the authors consider that defenses failing this test need not be tested on the latter two.
   • Perfect-Knowledge: construct a new loss function for the defense and generate examples that fool both the classifier and the detector; the authors emphasize that the choice of loss function is crucial, and intuitive choices often make a defense appear more robust than it is.
   • Limited-Knowledge: split the training set in two, have the attacker perform a white-box attack on a substitute model, then test whether the adversarial examples transfer to the defender's model.

   Loss function design for each type of defense

   • Binary classification (Grosse/Gong/Metzen): construct a combined function G that outputs class N+1 when the detector judges the input malicious and otherwise outputs the original classification result; G is designed to be approximately linear so that a suitable value of the constant c in the C&W attack exists.
   • KDE defense: add a penalty term ℓ2(x′) = max(−log(KDE(x′)) − ϵ, 0) to the objective, with ϵ set to the median of −log(KDE) over the training set; optimize in two stages, first generating an adversarial example and then reducing its detectability; the inner loop estimates the KDE using only 100 randomly sampled training instances (<5%).
   • dropout uncertainty: abandon a single combined function (because U(x) is computed from probabilities F and no suitable constant c exists), and instead sample K deterministic networks with different dropout randomness, average the K losses, and optimize.
   • mean blur: treat the 3×3 mean filter as the first convolutional layer of the network and apply the C&W attack directly to F′(x) = F(blur(x)).

   Reproduction and setup

   • All ten defenses were reimplemented by the authors (the authors state that seven of them had no public code); MNIST uses a standard convolutional network (99.4% accuracy), CIFAR-10 uses the ResNet of Metzen et al. (91.5% accuracy); the Bhagoji defense uses a 100-100-10 fully connected network, with 26 models trained for K from 9 to 784.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Ten defenses are each evaluated under three threat models: all can be bypassed under white-box attacks, and only dropout and KDE increase the perturbation on MNIST.

   Experimental setup

   • Models tested: standard convolutional network on MNIST (99.4% accuracy); ResNet of Metzen et al. on CIFAR-10 (91.5% accuracy); 100-100-10 fully connected network for the Bhagoji defense (26 different K values).
   • Datasets: MNIST (70,000 28×28 grayscale images), CIFAR-10 (60,000 32×32 color images).
   • Attack: C&W L2 attack (targeted attack); weak attacks FGS/JSMA are used only to reproduce the defenders' original results.
   • Metrics: attack success rate, average L2 perturbation, true positive rate/false positive rate (TPR/FPR) of detection.
   • Sample sizes: most evaluations generate an adversarial example for each image in the test set; the MMD test uses 100 images (MNIST) and at most 10,000 images (CIFAR); the dropout defense uses L = 30 samples and threshold τ = 0.02.
   • Number of repetitions: not reported in the paper.

   Main results

   Performance of each defense under Perfect-Knowledge (white-box) attack; the no-defense baseline average L2 perturbation is MNIST 2.05 and CIFAR 0.169 (according to the sections of the main text):

   Swipe horizontally to read the table

   Defense	Dataset	Attack success rate	Average L2 perturbation	Relative to baseline
   Grosse (adversarial retraining)	MNIST	100%	2.26	+10%
   Gong (binary classifier)	MNIST	100%	2.31	about +10%
   Metzen (convolutional layer detection)	CIFAR	not reported	0.227	+34%
   Bhagoji (dimensionality reduction)	MNIST	not reported	see Figure 3	still lower than the no-defense CNN
   Feinman KDE	MNIST	not reported	3.81	increases (baseline 2.05)
   Feinman dropout	MNIST	98%	3.68	baseline 2.05
   Feinman dropout	CIFAR	not reported	1.1	5×
   Li (mean blur)	MNIST	not reported	no increase	—

   • The authors state that five defenses add almost no robustness, three add only slight robustness, and the last two (KDE, dropout) are effective only on the simpler dataset.
   • The authors state that on CIFAR the adversarial examples remain imperceptible after all defenses are bypassed; on MNIST perturbations of 3.68–3.81 begin to approach human perceptibility (Figure 1).
   • The abstract and introduction give the summary: all ten methods are broken under white-box (and black-box) attacks.

   Zero-Knowledge evaluation

   • What was tested: the ability of each defense to detect C&W adversarial examples when the attacker does not know the defense exists.
   • Results: Grosse detects 98.5%, Gong 98% (MNIST); Metzen on CIFAR reaches at best only 81% TPR (28% FPR); the first layer of Li's cascaded SVM reaches 62% TPR (37% FPR) on MNIST and 56% TPR (40% FPR) on CIFAR, with the overall cascade reaching 92%/98% FPR; MMD cannot reject the null hypothesis on either 100 MNIST samples or 10,000 CIFAR samples (p > 0.05); dropout detects 75% on MNIST and 95% on CIFAR; mean blur causes 80% of low-confidence adversarial examples to fail.
   • Authors' interpretation: six defenses are weaker under zero knowledge than previously thought; weak attacks (FGS/JSMA) give an overly optimistic impression of robustness.

   Limited-Knowledge (black-box transfer) evaluation

   • What was tested: split the training set in half, have the attacker perform a white-box attack on substitute model R1, and measure the transfer success rate to the defender's model R2.
   • Results: under the Grosse defense, transfer success rate 98%, average L2 perturbation 5.3 (no-defense baseline 98%, 5.1); under the Metzen defense, low-confidence samples transfer 84%, high-confidence up to 98% (perturbation increases by a further 28%); under the KDE defense, the attacker uses only 5% of the training data and still succeeds with unchanged perturbation; under the dropout defense, transfer success rate 90%, and reaching 98% requires a 15% increase in perturbation to 4.23.
   • Authors' interpretation: dropout makes white-box attacks nearly as hard as black-box attacks, and the authors consider it the only defense not fully broken (although on CIFAR the perturbation remains imperceptible).

   Other ablations and analyses

   • MNIST artifact: the differences detected by the Hendrycks defense stem from MNIST border pixels being almost always 0 in natural images (Figure 2); on CIFAR there is no detectable difference, and the authors state that this defense is effective only on MNIST.
   • Dimensionality reduction dimension: the hardest-to-attack model of the Bhagoji defense retains only the first 25 principal components, with robustness about 3 times that of retaining all 784 dimensions, but still worse than the no-defense convolutional network (Figure 3).
   • KDE direction reversal: on CIFAR, in 80% of cases the KDE score of the adversarial example is higher than that of the original image, on which basis the authors state that this defense cannot hold on CIFAR.
   • Initial dropout attack failure: when attacking the dropout defense directly with the combined function G, 60% are still detected; the authors attribute this to U(x) being computed from probabilities, leaving no constant c to choose, and succeed after switching to the multi-network average loss.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that randomized defenses combined with eliminating transferability are the most promising direction; the experiments cover only MNIST and CIFAR-10.

   Limitations and future directions noted by the authors

   • Randomization plus eliminating transferability: the authors state that if a method to eliminate transferability could be found, randomization-based defenses might detect adversarial examples, and "we believe this is the most promising direction for future work" (Section 7.1).
   • Harder evaluation datasets: the authors state that as defenses improve, future work may need to evaluate on harder datasets such as ImageNet (Section 7.2).
   • Limitations of statistical tests: the authors state that MMD is one of the strongest multidimensional statistical tests, and even it is ineffective, so "without significant modification, statistical tests will not be able to detect adversarial examples" (Section 5.1).
   • Evaluation baseline recommendations: the authors propose that defense evaluation should follow this paper's methodology as a first step, including using the strongest attack, showing that white-box attacks fail, reporting TPR/FPR, evaluating beyond MNIST, and releasing code (Section 7.2).

   Scope of experiments

   • The evaluated defenses are ten detection methods from seven papers, on the two image classification datasets MNIST and CIFAR-10.
   • The attack algorithm is the C&W L2 attack; FGS and JSMA are used only to reproduce the defenders' original results and are not used for this paper's evaluation.
   • Robustness is measured by L2 distance; the paper does not report evaluation results under L0 or L∞ metrics.
   • In the black-box evaluation, transfer tests were performed for four defenses: Grosse, Metzen, KDE, and dropout; the paper does not report Limited-Knowledge evaluation for defenses such as Bhagoji and Li (cascaded SVM).
   • The paper does not report the number of repetitions or statistical significance tests for each experiment; the Li et al. defense was originally evaluated on ImageNet, and this paper demonstrates its ineffectiveness only on MNIST/CIFAR.

   Collapse this answer

6. 06Summarize the main content of the paper

   All ten adversarial example detection methods can be bypassed under targeted white-box attacks, and the authors give recommendations for defense evaluation.

   An attack/evaluation paper: it systematically examines the defense line of "detecting adversarial examples," and the authors state that all ten published detection methods can be bypassed by targeted attacks.

   • Problem: Adversarial examples are difficult to classify correctly, and recent work has turned to detection; the authors question the effectiveness of these detection schemes and the assumption that "adversarial examples have intrinsic distinguishable properties."
   • Method: Reproduce ten detection methods from seven papers and evaluate them under three threat models: Zero-Knowledge, Perfect-Knowledge, and Limited-Knowledge; the core technique is to construct an attacker loss function specifically for each defense, using the C&W L2 attack to simultaneously fool the classifier and the detector.
   • Main results: Under white-box, the attack success rate against the Grosse and Gong defenses is 100% (MNIST average L2 perturbation 2.26/2.31, baseline 2.05); for the Metzen defense on CIFAR, the perturbation rises only from 0.169 to 0.227; KDE and dropout on MNIST raise the perturbation to 3.81 and 3.68, and are the only two defenses that increase difficulty on the simple dataset.
   • Key observations: The authors state that after all defenses on CIFAR are bypassed, the adversarial examples remain imperceptible; the difference detected by the Hendrycks defense on MNIST stems from a dataset artifact in which boundary pixels are always zero, and there is no such difference on CIFAR.
   • Authors' conclusion: Adversarial examples are harder to detect than previously recognized, and properties believed to be intrinsic are in fact not intrinsic; dropout randomization is the only defense not fully broken, and if transferability can be eliminated, the randomization line may be effective.
   • Recommendations: Defense evaluation should use the strongest attacks, demonstrate white-box attack failure, report TPR/FPR, evaluate on datasets beyond MNIST, and release code.

   Collapse this answer

Related papers