Limitations of local intrinsic dimensionality (LID) in characterizing the adversarial example subspace
On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial Examples
The authors use two new sets of experiments on MNIST and CIFAR-10 to test LID's ability to characterize adversarial subspaces, and the authors say its performance is limited under confidence variation and transfer attacks.
- LID was proposed to characterize the subspace of adversarial examples, but whether it remains effective under variation in attack confidence and under black-box transfer attacks has not been covered by existing analyses.
- The authors conduct two new sets of experiments on MNIST and CIFAR-10: oblivious attacks at different confidences (including ensemble-confidence training) and transfer attacks crafted from another DNN, using C&W and EAD attacks.
- The authors report: in the oblivious setting the detection rate is mostly below 50% (CIFAR-10); ensemble-confidence training is instead worse; under transfer attacks the detection rate drops further, and EAD is more transferable.
- The authors conclude that LID has limited ability to characterize EAD's L1 adversarial examples, ensemble-confidence training, and transferable adversarial examples; the experiments cover only MNIST and CIFAR-10 and two attacks, C&W and EAD.
- Evaluated models
- LID detectorDNN models (provided by Ma et al. 2018)
- Benchmarks
- MNISTCIFAR-10
- Metrics
- AUC scoredetection rateclassification rate (post detection)classification rate w/o detection
Through two new sets of experiments on MNIST and CIFAR-10, the paper points out that local intrinsic dimensionality (LID) has limitations in characterizing the adversarial example subspace. The first set of experiments shows that LID is very sensitive to the confidence parameter used by the attack, and performs poorly when adversarial examples with different confidence levels are mixed; the second shows that when adversarial examples are generated by another DNN model, LID cannot effectively characterize their subspace. The authors therefore conclude that LID has limited ability to characterize the adversarial example subspace.
Paper analysis
What problem does this paper attempt to solve?
Test LID's ability to characterize adversarial subspaces under confidence variation and transfer attacks; the authors report its limitations.
Core question: whether LID (local intrinsic dimensionality) can still effectively characterize the subspace of adversarial examples under variation in attack confidence and under black-box transfer attacks.
- Background and motivation: understanding the subspace in which adversarial examples lie helps study the robustness of DNNs to adversarial perturbations. Ma et al. (2018) proposed using LID on the layer-by-layer hidden representations of DNNs to characterize the adversarial subspace, and claimed it is discriminative for five attacks.
- Gap in existing analyses: the authors point out two sets of experiments missing from existing LID analyses: (i) oblivious attacks using adversarial examples of different confidences and LID analysis; (ii) black-box transfer attacks.
- Core observations: the authors find that LID's performance is very sensitive to the attack's confidence parameter; LID trained on an ensemble of adversarial examples of different confidences instead performs worse; when adversarial examples are generated from another DNN, LID cannot effectively characterize their subspace.
- Paper's output: two new sets of experiments on MNIST and CIFAR-10 with two attacks, C&W and EAD, reporting LID's limitations.
- Threat model:
- Attacker's goal: generate adversarial examples that bypass the LID detector (untargeted attacks).
- Knowledge: in the oblivious setting, adversarial examples are generated on an undefended DNN without knowledge of the deployed defense; in the transfer setting, adversarial examples are generated from another DNN with similar classification accuracy.
- Capability: control the confidence parameter κ of the C&W and EAD attacks (range [0, 40]).
- Victim system: the DNN model provided by Ma et al. (2018) and its LID detector (trained with all DNN layers, k=20, minibatch 100).
What related research exists?
The work is organized around four categories: adversarial examples, LID subspace characterization, EAD attacks, and transferability.
Existence of and defenses against adversarial examples
- Szegedy et al. (2013), Goodfellow et al. (2015)—pointed out that well-trained DNNs are vulnerable to adversarial examples.
- MagNet (Meng & Chen, 2017)—defends against adversarial examples with a detector and a data reformer learned from the manifold of natural sample data; the paper claims it has robust defense performance against C&W attacks with different confidence levels under the oblivious setting.
LID subspace characterization (the object directly tested in this paper)
- Ma et al. (2018)—proposed using LID to characterize adversarial subspaces on layer-by-layer hidden representations of DNNs, and claims strong discriminative power against five attacks; its C&W detector is trained only with low-confidence samples at κ=0. This paper points out that it lacks two sets of experiments: confidence variation and transfer attacks.
- The LID estimation method itself comes from Karger & Ruhl (2002), Houle et al. (2012).
- Athalye et al. (2018)—showed that high-confidence (larger κ) adversarial examples can bypass the same detector; the paper states that it is still unclear whether training with high-confidence samples can strengthen the detector, and this paper tests this.
Attack methods
- C&W attack (Carlini & Wagner, 2017)—an L2 distortion attack, used in this paper for comparison.
- EAD (Chen et al. 2017)—an elastic-net attack based on L1 distortion, which degenerates to C&W when β=0; the paper states that a larger β can generate more effective and more transferable adversarial examples (citing Sharma & Chen, 2017). EAD is an attack not tested by Ma et al. (2018).
Transferability of adversarial examples
- Papernot et al. (2016), Liu et al. (2016)—pointed out that adversarial examples can transfer from one DNN to another, supporting the black-box transfer attack setting.
Baselines and benchmarks: This paper does not propose a new detector as a baseline for comparison, but directly evaluates Ma et al. (2018)'s LID detector (default settings); the datasets are MNIST and CIFAR-10.
The authors position this paper as a supplementary test of Ma et al. (2018)'s LID analysis: under the two settings it did not cover—oblivious confidence variation and transfer attacks—they report the limitations of LID in characterizing adversarial subspaces.
How does the paper address this problem?
Evaluate the AUC and detection rate of the LID detector under two new settings (varying-confidence oblivious, transfer attacks).
Overall approach: reuse the LID detector of Ma et al. (2018) and its default training settings, measure AUC and detection rate under two attack settings it did not cover, and test the boundary of LID's ability to characterize adversarial subspaces.
Definition of LID
- Given a reference sample x generated by a distribution P, the MLE estimate of LID is LID(x) = −[1/k∑i=1klog(ri(x))/(rk(x))]−1, where r_i(x) is the distance from x to its i-th nearest neighbor and r_k(x) is the maximum distance among the k nearest neighbors.
- Following Ma et al. (2018), LID is applied separately to the layer-by-layer hidden representations (or the final logits layer) of adversarial examples and normal examples; the detector also adds randomly perturbed samples to distinguish random perturbations from adversarial perturbations.
Experimental setup
- Data and targets: MNIST and CIFAR-10, untargeted attacks; 1000 correctly classified images are randomly selected from the test set as attack targets.
- Model sources: the oblivious setting uses the DNN model provided by Ma et al. (2018); the transfer setting uses a different DNN model from Carlini & Wagner (2017) to generate adversarial examples (the paper says these models have similar classification accuracy).
- Attack methods: C&W (L2, default settings) and EAD (default implementation, L1 regularization coefficient β=0.1); for EAD, two decision rules (elastic-net (EN) or L1 distortion) for selecting adversarial examples are additionally reported.
- Confidence: κ for C&W and EAD is taken from [0, 40], and the experiments use {0, 10, 20, 30, 40}.
- LID detector training: uses the default settings of Ma et al. (2018), all DNN layers, number of neighbors k=20, minibatch 100; unless otherwise stated, the detector is trained with the same attack and the same confidence, which the authors say already weakens the attack capability to the greatest extent.
Design of the two sets of experiments
- Oblivious attacks: the detector is trained with the same attack and the same κ, and AUC and detection rate are measured at different κ; then a detector trained with samples from κ={0,10,20,30,40} ensembled (5 times the training samples of a single κ) is compared.
- Transfer attacks: adversarial examples are generated on another DNN, and the detector is still trained on the target model with the same attack and confidence; detection rate, classification rate after detection (post detection), and classification rate without detection are measured.
Evaluation metrics
- AUC score: follows the metric of Ma et al. (2018).
- Detection rate: the proportion of attacks detected by the detector.
- Classification rate (post detection / w/o detection): reported in the transfer experiments, denoting respectively the classification rate of undetected attacks after detection and the classification rate when no detector is deployed.
What experiments did the paper conduct?
Two sets of experiments: oblivious attacks with varying confidence (including ensemble training) and transfer attacks; the detection rate is mostly below 50%.
Experimental setup
- Subject under test: the LID detector of Ma et al. (2018) (all DNN layers, k=20, minibatch 100), installed on the DNN model provided by Ma et al. in the oblivious setting.
- Datasets: MNIST, CIFAR-10; for each, 1000 randomly selected correctly classified test images are used as attack targets.
- Attacks: C&W(L2) (default settings) and EAD (β=0.1, with two decision rules, EN and L1), κ ∈ {0,10,20,30,40}, untargeted.
- Metrics: AUC score (%), detection rate (%); the transfer experiments additionally report classification rate (post detection) and w/o detection (%).
- Default condition: except in the ensemble experiment, the detector is trained with the same attack and the same confidence; the paper does not report the number of repetitions.
Main results
Transfer attacks (the detector is trained on the target model with the same attack and confidence, and the adversarial examples come from another DNN; according to Table 1, units %):
Swipe horizontally to read the table
Dataset Attack κ Detection rate Classification rate (post detection) Classification rate (w/o detection) MNIST C&W(L2) 0 25.90 73.48 98.89 MNIST EAD(EN) 40 31.25 0.80 1.81 MNIST EAD(L1) 40 31.35 0.90 1.91 CIFAR-10 C&W(L2) 0 1.01 95.28 96.30 CIFAR-10 C&W(L2) 40 17.70 72.73 84.45 CIFAR-10 EAD(EN) 40 23.43 57.19 69.29 CIFAR-10 EAD(L1) 40 22.42 58.47 70.70 (Table 1 contains all 30 rows for κ=0–40; some rows are omitted here; see Table 1 of the paper for the complete numbers.)
- The authors say transfer attacks further reduce the LID detection rate, even when the detector knows the attack and confidence.
- The authors say that on CIFAR-10, measured by classification rate after detection, EAD's attack transferability is better than C&W's.
- Figure 1 shows visualizations of adversarial examples for different attacks in the κ=20 transfer setting, with unsuccessful attacks (detected by the detector) marked with a red cross; the authors say EAD generates highly transferable, visually similar examples, while C&W has more difficulty bypassing the detector.
Oblivious attacks: single-confidence training
- Results (Table 2): on CIFAR-10 the detection rate is below 50% in most cases, and on MNIST about 35–60%; at κ=0 on CIFAR-10, the C&W detection rate is 89.17% and AUC 99.05, while the EAD(EN) detection rate is only 42.92% and AUC 82.22.
- Variation with confidence: the authors say AUC and detection rate decrease as κ increases, especially at κ=20 and 30; for example, on CIFAR-10 the C&W detection rate drops from 89.17% at κ=0 to 43.18% at κ=20.
- Authors' interpretation: even when the detector is trained with the same attack and confidence, at low confidence it is effective only against C&W and much less effective against EAD.
Oblivious attacks: ensemble-confidence training
- Setup: the detector is trained with samples from the same attack with κ={0,10,20,30,40} ensembled (the training sample size is 5 times that of a single κ).
- Results (Table 3): on both datasets, the detection rate for all attacks is consistently below 45% and AUC consistently below 85%; MNIST's AUC (73.49–82.75) is lower than CIFAR-10's (81.49–84.18).
- Authors' interpretation: training with more samples of different confidences instead harms detector performance; the authors call this result unexpected.
Other ablations and analyses
- The two EAD decision rules (EN vs L1) give close numbers in the three tables, and the paper does not offer additional interpretation of their difference.
- The paper does not report ablations on other hyperparameters.
What are some points that can be further explored?
The authors summarize three limitations; the experiments cover only two datasets and two attacks, and do not report the number of repetitions.
Limitations noted by the authors and future directions
- Even at low confidence and with the full attack parameters given for analysis, LID still cannot effectively characterize the subspace of L1-distortion adversarial examples generated by EAD (Conclusion, item 1).
- LID subspace analysis trained with ensemble confidence performs worse (Conclusion, item 2).
- LID's ability to characterize the subspace of transferable adversarial examples is limited (Conclusion, item 3).
- In the oblivious setting, if the attack parameters are unknown, the performance of the LID detector may drop severely; in the transfer setting, even if the attack parameters are known, LID appears to ignore the subspace of transferable adversarial examples (end of Introduction, in the authors' voice).
- The paper does not propose future research directions.
Experimental coverage
- There are two datasets, MNIST and CIFAR-10; for each dataset, 1000 randomly selected, correctly classified test images are used as attack targets (Section 2.1).
- The attack methods are C&W(L2) and EAD (β=0.1, with two decision rules, EN/L1), confidence κ takes {0,10,20,30,40}, and all are untargeted attacks.
- In the transfer experiments, the adversarial examples are generated on the DNN model of Carlini & Wagner (2017), which the paper says has classification accuracy close to that of the target model, but it does not give the specific architecture name or accuracy numbers.
- The LID detector is fixed to the default settings of Ma et al. (2018) (all layers, k=20, minibatch 100); the paper does not report ablations on k or other hyperparameters.
- The paper does not report the number of experimental repetitions or variance, nor does it report human evaluation.
Summarize the main content of the paper
Two new sets of experiments test LID's ability to characterize adversarial subspaces; the authors report its limitations under confidence variation and transfer attacks.
An ICLR 2018 workshop paper that tests the ability of the LID method proposed by Ma et al. (2018) to characterize adversarial subspaces under two settings it did not cover.
- Problem: LID is used to distinguish adversarial from normal samples on layer-by-layer hidden representations of DNNs, but existing analyses did not test two settings: changes in attack confidence (oblivious) and black-box transfer.
- Method: On MNIST and CIFAR-10, 1000 correctly classified test images were taken from each, and untargeted attacks were performed with C&W(L2) and EAD (β=0.1), with κ ∈ {0,10,20,30,40}; the LID detector follows Ma et al.'s default settings and by default is trained with the same attack and the same confidence.
- Oblivious results: On CIFAR-10, detection rates are mostly below 50%; at κ=0, the C&W detection rate is 89.17%, while EAD(EN) is only 42.92% (Table 2).
- Ensemble training results: Ensemble training with 5 times the samples and κ={0,…,40} is instead worse, with detection rates consistently below 45% and AUC below 85% (Table 3).
- Transfer results: When adversarial examples come from another DNN, detection rates drop further; on CIFAR-10 at κ=0, the C&W detection rate is only 1.01%; on MNIST at κ=40, the post-detection classification rate for EAD attacks that were not detected is below 1% (Table 1).
- Authors' conclusion: LID has limited ability to characterize EAD's L1 adversarial examples, ensemble-confidence training, and transferable adversarial examples; when attack parameters are unknown, its detection performance may drop severely.