Skip to content

The space of transferable adversarial examples

The Space of Transferable Adversarial Examples

arXiv· arXiv:1704.03453· Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, Patrick McDanielStanford University; Pennsylvania State University; Google BrainIndexed by this site · 21 days ago· Originally publishedSelected AttacksSelectedAttention40

Quick read

Analysis and theory

Based on the paper PDF (AI-generated); refer to the original

The authors propose the GAAS method to estimate the dimensionality of the adversarial example subspace, find that on MNIST an average of 24.87 directions transfer between fully connected networks, and give sufficient conditions and a counterexample for transferability.

Questions
Adversarial examples often transfer between models, making black-box attacks possible, but the dimensionality of adversarial subspaces and the similarity of decision boundaries across different models previously lacked quantitative characterization.
Methods
Propose the GAAS method to find multiple orthogonal adversarial directions to estimate subspace dimensionality; define legitimate, adversarial, and random directions to measure decision boundary distances between models; and give a sufficient condition for the transferability of model-agnostic class-mean-shift perturbations along with an XOR counterexample.
Experiments and results
On MNIST, an average of 24.87 directions transfer between fully connected networks; for most model pairs the distance between boundaries is smaller than the distance from data points to the boundary; adversarial training only increases the distance between boundaries in the adversarial direction from 0.32 to 0.63, which is insufficient to prevent transfer.
Limitations and future work
The authors note that GAAS relies on a first-order approximation of the loss, and that the maximal orthogonal construction under other norms such as l∞ is an open problem; the SVM results differ quantitatively due to the one-vs-rest implementation; the XOR counterexample is based on handcrafted data, and whether real data has similar properties is unknown.
Experimental setuplogistic regression (LR)、support vector machine (SVM) et al. · MNIST、DREBIN · transfer rate、number of orthogonal adversarial directions et al.
Threat model
At test time, the attacker uses adversarial examples to mislead the model; the focus is on untargeted misclassification. Black-box setting: the attacker trains a local model (which can be obtained by querying the target model), constructs adversarial examples on it, and transfers them to the target model.
Evaluated models
logistic regression (LR)support vector machine (SVM)DNNCNNLinearQuadraticadversarially trained DNN
Benchmarks
MNISTDREBIN
Metrics
transfer ratenumber of orthogonal adversarial directionsMIN-DISTINTER-DISTaccuracy on shifted data
AI summaryand selection rationaleThe paper proposes the Gradient-Aligned Adversarial Subspace (GAAS) method, which directly estimates the dimensionality of the adversarial subspace by finding multiple orthogonal adversarial perturbations. In two fully connected networks on MNIST, the source model finds an average of 44.28 orthogonal perturbations, of which 24.87 transfer to the target model, and the transferred portion constitutes a continuous subspace of about 25 dimensions; randomly sampling perturbations within this subspace can cause 99% of samples for the source model and 89% of samples for the target model to be misclassified. Measurements show that the decision boundaries of different models are very close in both adversarial and normal directions, and adversarial training only increases the boundary margin in the adversarial direction from 0.32 to 0.63. The authors give sufficient conditions for the transferability of class-mean-shift perturbations, and construct an MNIST variant with XOR artifacts so that adversarial examples do not transfer between linear and quadratic models.

The paper proposes the Gradient-Aligned Adversarial Subspace (GAAS) method, which directly estimates the dimensionality of the adversarial subspace by finding multiple orthogonal adversarial perturbations. In two fully connected networks on MNIST, the source model finds an average of 44.28 orthogonal perturbations, of which 24.87 transfer to the target model, and the transferred portion constitutes a continuous subspace of about 25 dimensions; randomly sampling perturbations within this subspace can cause 99% of samples for the source model and 89% of samples for the target model to be misclassified. Measurements show that the decision boundaries of different models are very close in both adversarial and normal directions, and adversarial training only increases the boundary margin in the adversarial direction from 0.32 to 0.63. The authors give sufficient conditions for the transferability of class-mean-shift perturbations, and construct an MNIST variant with XOR artifacts so that adversarial examples do not transfer between linear and quadratic models.

Why it was selectedThe paper directly estimates the dimensionality of the adversarial subspace using the number of orthogonal perturbations, providing a reusable measurement method for understanding why black-box transfer attacks are prevalent.

Related events and news

0 events · 1 news mentions

News that mentions this paper

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Quantitatively characterize the dimensionality of the adversarial example subspace and the similarity of decision boundaries between models, and explain and delimit the transferability of adversarial examples.

    Core question: Why can adversarial examples transfer between models? How large are their adversarial subspaces, and how similar are the decision boundaries of different models? Previously there was a lack of quantitative analysis.

    • Problem setting: Adversarial examples cause models to misclassify by making small perturbations to test inputs, and they often transfer between models, enabling attackers to launch black-box attacks using local models; the authors consider this an obstacle to secure ML deployment.
    • Existing gaps: Experience shows that adversarial examples lie in large, continuous regions, but the dimensionality of these subspaces is unknown; the similarity of decision boundaries across different models also lacks quantitative study.
    • Core observation: The authors argue that the higher the subspace dimensionality, the more likely the adversarial subspaces of two models intersect, leading to transfer; and hypothesize that decision boundaries are close to each other in arbitrary directions.
    • Paper contributions: Propose a method to estimate the dimensionality of adversarial subspaces; provide the first quantitative comparison of decision boundaries across different models; formally give sufficient conditions for the transferability of model-agnostic perturbations and a counterexample where transfer fails.
    • Threat model:
      • Goal: Mislead the model at test time to produce untargeted misclassification (predicting any class other than the true class).
      • Knowledge: Black-box—the attacker trains a local model (obtainable by making prediction queries to the target model) and constructs adversarial examples on it to transfer to the target model.
      • Capability: Add norm-bounded perturbations to inputs (MNIST pixels clipped to [0,1], DREBIN features binarized).
      • Victim systems: MNIST digit classification and DREBIN malware detection models.
  2. What related research exists?

    Related work is divided into four groups: adversarial example generation, transferability and black-box attacks, defenses and gradient masking, and the geometry of adversarial subspaces.

    Adversarial example generation

    • Szegedy et al. [22] and Biggio et al. [4]—proposed test-time adversarial examples; this paper follows their threat model.
    • Goodfellow et al. [6]—proposed FGSM and argued that neural networks generalize overly linearly to out-of-distribution data; most experiments in this paper use FG(S)M, and the authors state that GAAS's results provide further evidence for this linear generalization argument.
    • JSMA of Papernot et al. [18]—an iterative attack based on the Jacobian saliency map; this paper proposes its non-iterative variant, Independent JSMA, on DREBIN.

    Transferability and black-box attacks

    • Papernot et al. [16]—studied the transfer phenomenon and used it to mount black-box attacks; this paper uses their method to find adversarial directions for SVM.
    • Papernot et al. [17]—mounts black-box attacks by querying the target model to train a local substitute model; this paper's threat model is built on this.
    • Liu et al. [13], Papernot et al. [16], etc.—empirically study the transferability of adversarial examples.

    Defenses and gradient masking

    • Distillation of Papernot et al. [20] and adversarial training of Goodfellow et al. [6]—the authors point out that adversarial examples can still transfer under such explicit defenses, and attribute this to gradient masking [19]: the defenses do not move the decision boundary but only destroy gradient information.
    • Tramèr et al. [23]—systematically studied transfer from undefended models to adversarially trained models, and on this basis proposed improved adversarial training (ensemble adversarial training).

    Geometry of the adversarial subspace and robustness analysis

    • Warde-Farley and Goodfellow [25]—proved that the space of adversarial perturbations is at least two-dimensional; Goodfellow et al. [6], Szegedy et al. [22]—adversarial regions are continuous and occupy a negligible proportion of the input space.
    • Bastani et al. [2]—for piecewise linear models, used convex optimization to find adversarial perturbations within the linear region of x; this paper extends their LP to iteratively find orthogonal directions, and finds that its transfer rate is lower than FGM.
    • Fawzi et al. [5]—proved that if the class-mean difference is small, any low-risk linear model can be fooled by small perturbations; the proof of Theorem 2 in this paper builds on their results.

    Positioning: The authors state that this paper is the first to directly measure the dimensionality of the adversarial subspace and the first to quantitatively compare the decision boundaries of models in different hypothesis classes; the formalization section gives sufficient conditions for transferability and task instances where transfer fails.

  3. How does the paper address this problem?

    Uses GAAS to find orthogonal adversarial directions to estimate the subspace dimensionality, defines distance metrics in three directions to compare decision boundaries, and then formally analyzes class-mean perturbations.

    The overall approach has three steps: propose the Gradient Aligned Adversarial Subspace (GAAS) method to directly estimate the dimensionality of the adversarial subspace; define distance metrics in three directions to quantitatively compare the decision boundaries of different models; and give sufficient conditions for transferability and a counterexample for model-agnostic perturbations.

    Adversarial example generation and notation

    • Mainly uses FG(S)M: x∗ = x + ε · ∇x J(x,y)/|∇x J(x,y)|, i.e., one step along the loss gradient direction, normalized by the lp norm used; implementation from the cleverhans 1.0 library.
    • Focuses on untargeted misclassification; MNIST pixels are clipped to [0,1], and DREBIN features are rounded to binary values.

    GAAS: estimating the dimensionality of the adversarial subspace

    • Under a first-order approximation of the loss, find the largest set of orthogonal perturbations that satisfies the norm constraint and increases the loss by at least γ; by Lemma 1, the number of orthogonal directions is proportional to the loss increment γ and inversely proportional to the loss smoothness and the perturbation magnitude ε.
    • If the γ that causes misclassification is unknown, try multiple γ values and keep the value that produces the most misclassifying perturbations.
    • The authors note that generalizing the maximal orthogonal construction to other norms such as l∞ is an open problem.

    Decision boundary distance metrics

    • Define three unit directions: the legitimate direction dleg (pointing to the nearest different-class sample in the test set), the adversarial direction dadv (pointing to an adversarial example generated for that model), and the random direction drand (pointing to a uniformly random input classified as a different class).
    • Minimum distance MIN-DISTd(f,x): the minimum l2 distance from x to the decision boundary along direction d, computed by line search with step size 0.05.
    • Distance between boundaries INTER-DISTd(f1,f2,x): the difference between the minimum distances of the two models in the same direction; in the adversarial direction, a small value means a perturbation that crosses the f1 boundary is likely to also cross the f2 boundary, i.e., transfer occurs.

    Formal analysis of transferability

    • Consider model-agnostic perturbations: shift data points along the class-mean-difference direction, with δ := 1/2(Eμ+1[x]−Eμ−1[x]) in input space, r := −ϵ· y·δ̂; for classifiers of the form f(x)=wᵀφ(x)+b (including linear, polynomial, and feedforward networks), analyze their transfer in feature space.
    • Theorem 2: if the alignment ∆ between the weights and the class-mean difference in feature space is not too small, and the mapping φ satisfies the pseudo-linear condition for the perturbation r (α large, β small in the orthogonal decomposition), then x+r fools f with nonzero probability.
    • Counterexample construction: add an XOR artifact to two center pixels on MNIST 3s vs. 7s, so that the linear model and the quadratic model learn different features, each being vulnerable but not transferring to the other.
  4. What experiments did the paper conduct?

    Measure subspace dimensionality, decision boundary distances, the effect of adversarial training, and the transfer of model-agnostic perturbations on MNIST and DREBIN.

    Experimental setup

    • Models tested: on MNIST, LR, SVM, DNN (architecture C, Table 2), two fully connected networks (source is a two-layer variant of C, target is C), two CNNs (source B, target A); for the 3s vs. 7s task, linear, quadratic, DNN, and CNN models are additionally used.
    • Datasets: MNIST digit classification; DREBIN malware detection (balanced subset, feature selection to 1,000 dimensions, model accuracy above 94%).
    • Attack methods: l2-norm FGM (ε=5) for differentiable models, the method of [16] for SVM; Independent JSMA on DREBIN (at most 10 features flipped per perturbation).
    • Metrics: number of orthogonal adversarial directions, transfer rate (prefiltered definition: original samples do not fool the source and target, adversarial samples fool the source model), MIN-DIST, INTER-DIST, post-perturbation accuracy.
    • Measurement method: distances are averaged over all points in the test set, line search step size 0.05; DREBIN features are not binarized in this section.
    • Sample size and number of repetitions: the paper does not report specific sample sizes or numbers of repetitions.

    Main results

    Number of orthogonal adversarial directions found by GAAS on MNIST and transfer (according to Section 3.2 of the main text and Table 3):

    Swipe horizontally to read the table

    Source→TargetMethodSingle-direction transfer rateNumber of source-model directionsNumber of transferred directions
    DNN (two-layer variant→C)GAAS94%44.2824.87
    DNN (two-layer variant→C)Second-Order95%1.000.95
    DNN (two-layer variant→C)Convex Opti.17%2.580.45
    CNN (B→A)GAAS69%15.182.24
    • The authors state that GAAS is the most successful method for finding a large number of orthogonal directions, and consider that this provides further evidence for neural networks generalizing overly linearly to out-of-distribution data.
    • Randomly sampling perturbations with l2 norm 5 within the DNN subspace fools the source model 99% and the target model 89%; for CNN the corresponding values are 80% and 63%.
    • The main text reports a FGM transfer rate of 68% for CNN, while Table 3 reports a GAAS single-direction transfer rate of 69%.

    Decision boundary distances

    • What was measured: MIN-DIST and INTER-DIST between pairs of LR, SVM, and DNN on MNIST in three directions (Figure 4; l1-norm and DREBIN results are in Figure 7, and the authors state the conclusions are the same).
    • Results: for most model pairs and directions, the distance between boundaries is smaller than the minimum distance from test points to the boundary; the minimum distance is smallest in the adversarial direction, and boundaries are farther apart in the random direction. Figure 4 caption example: for LR in the legitimate direction the minimum distance is about 4, and the distance between boundaries is below 1.
    • Exception: when SVM is the source model, the distance between boundaries in the adversarial direction is much larger, which the authors suggest may stem from their SVM implementation using a one-vs-rest strategy.

    Effect of adversarial training

    • What was measured: a pair of DNNs, source is an undefended model, target is an adversarially trained model, repeating the distance measurements.
    • Results: the average distance between boundaries in the adversarial direction increases from 0.32 (undefended target) to 0.63 (adversarially trained target); the total perturbation required for transfer (minimum distance 1.64 plus distance between boundaries) is still smaller than a perturbation of norm 5.
    • Authors' interpretation: the "movement" of the decision boundary by adversarial training is too small to prevent transfer; the authors consider that the reason such defenses resist white-box attacks is gradient masking.

    Model-agnostic perturbations and the XOR counterexample

    • MNIST 3s vs. 7s (Table 1, ε=4): post-perturbation accuracy is 39% for the linear model (∆=0.55), 51% for the quadratic model (∆=0.51), 43% for the DNN (∆=0.68), and 76% for the CNN (∆=0.79); the φ of the first three models satisfies the pseudo-linear condition, and the CNN does not, yet 24% of perturbed samples are still misclassified, which the authors attribute to the bound being loose.
    • Full MNIST multi-class: using l∞-norm class-mean-difference perturbations (ε=0.3), accuracy ranges from 2% for the linear model to 66% for the CNN.
    • DREBIN: on average fewer than 10 of the 1,000 binary features are modified, and accuracy drops to 65% for the linear model and 72% for the DNN.
    • XOR counterexample: on MNIST 3s vs. 7s, both the linear and quadratic models have accuracy above 98%, and the transfer rate of FGM samples (l2 norm 4) exceeds 60%; after adding the XOR artifact, the linear model is still fooled by FGM and the quadratic model is fooled by flipping the signs of two center pixels, but the two kinds of perturbation do not transfer to each other.

    Other ablations and analyses

    • Second-Order method: under the tanh source model the Hessian term is very small, and the optimal solution orthogonal to the gradient is non-adversarial; the authors consider second-order approximation of little use for finding more adversarial directions (Appendix B.2).
    • Convex Optimization: on the source model, adversarial examples can be found for 93% of test inputs, with on average about 2 orthogonal directions; transfer rate is 17% (FGM is 95%); for 70% of inputs no successful perturbation transfers to the target (Appendix B.2).
    • For architecture C under l2 norm 5, the LP is solvable for 0% of test inputs, whereas FGSM always succeeds (Appendix B footnote 4).
    • Independent JSMA (DREBIN): for 89% of inputs at least one successful perturbation is found, with on average 42 independent perturbations, of which on average 22 transfer to the target model (Appendix B.2).
  5. What are some points that can be further explored?

    The authors note that the norm generalization of GAAS is an open problem, and give an SVM exception and a handcrafted counterexample; experiments cover two task types, MNIST and DREBIN.

    Limitations and future directions noted by the authors

    • Norm generalization: finding a similar maximal orthogonal perturbation construction for other norms such as l∞ is an open problem (Section 3.1).
    • SVM exception: when SVM is the source model, the distance between boundaries in the adversarial direction differs quantitatively (much larger), which the authors suggest may stem from their SVM implementation using a one-vs-rest strategy (Section 4.2 footnote 2).
    • Tightness of the bound: the bound in Theorem 2 is expected to be loose in practice, because the orthogonal component is unlikely to align maximally with w (Appendix C.2); the CNN does not satisfy the pseudo-linear condition yet still has 24% misclassification, and the authors state the bound is "expectedly somewhat loose" (Section 5.1).
    • Realism of the counterexample: the XOR experiment is based on a manually modified dataset, and the authors suggest it is worth examining whether real datasets have similar properties that lead to low transferability (Section 5.2).
    • Model-stealing premise: the authors note that if an attacker can recover a sufficiently close approximation of the target model via model stealing, then any model that is vulnerable in the white-box setting can be attacked (Section 5 footnote 3).

    Scope of experiments

    • The datasets are MNIST (including the 3s vs. 7s binary classification and the XOR variant) and a balanced subset of DREBIN (1,000-dimensional features), two tasks in total.
    • The model classes tested include LR, SVM, DNN (architecture C and its two-layer variant), CNN (architectures A, B), and linear and quadratic models.
    • Distance measurements cover the l2 norm (main text) and the l1 norm (Appendix D), including MNIST and DREBIN.
    • The effect of adversarial training is measured only on one pair of DNNs (undefended source→adversarially trained target).
    • The paper does not report sample sizes, numbers of repetitions, or statistical significance tests for the experiments.
  6. Summarize the main content of the paper

    Estimates the dimensionality of the adversarial subspace and quantitatively compares decision boundaries, giving sufficient conditions and a counterexample for transfer; the authors consider that defending against transfer attacks may be promising.

    An analysis paper on the transferability of adversarial examples, combining empirical measurements and formal analysis to explain why transfer occurs and when it does not.

    • Problem: Cross-model transfer of adversarial examples makes black-box attacks feasible, but the dimensionality of adversarial subspaces and the similarity of decision boundaries across models were previously unknown.
    • Method: Propose GAAS to find the largest set of orthogonal adversarial directions under a first-order approximation of the loss; define MIN-DIST and INTER-DIST measures along legitimate, adversarial, and random directions; prove a sufficient condition for the transferability of model-agnostic perturbations along the class-mean-difference direction (Theorem 2).
    • Subspace dimensionality: On MNIST between two fully connected networks, GAAS finds on average 44.28 orthogonal adversarial directions, of which 24.87 transfer to the target model; randomly sampling perturbations within the subspace fools the source model 99% and the target model 89% of the time.
    • Decision boundaries: For most models and direction pairs, the distance between boundaries is smaller than the distance from data points to the boundary; adversarial training only increases the adversarial-direction boundary distance from 0.32 to 0.63, and the total perturbation required for transfer remains smaller than the attack budget of 5; the authors therefore argue that adversarial training does not substantially move the boundary and that its white-box defense effect stems from gradient masking.
    • Model-agnostic perturbations: On MNIST 3s vs. 7s, class-mean-shift perturbations reduce linear model accuracy to 39% and CNN to 76% (Table 1); on DREBIN, changing on average fewer than 10 features reduces the linear model to 65% and the DNN to 72%.
    • Conclusion: The XOR counterexample shows that linear and quadratic models can each be vulnerable yet not transfer to each other; the authors argue that transferability is not an inherent property of non-robust models, and that designing defenses against transfer attacks may be feasible even if models remain vulnerable to direct attacks.

Related papers

Show 3 more papers
Read originalarxiv.org