Skip to content

Transferability in machine learning: from phenomena to black-box attacks using adversarial samples

Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples

arXiv· arXiv:1605.07277· Nicolas Papernot, Patrick McDaniel, Ian GoodfellowThe Pennsylvania State University; OpenAIIndexed by this site · 21 days ago· Originally publishedSelected AttacksSelectedAttention47

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors study the cross-model transferability of adversarial examples, training substitute models with 800 queries and achieving misclassification rates of 96.19% and 88.94% against Amazon and Google hosted classifiers.

Threat modelThe attacker has only oracle access to the victim classifier: it can query the predicted label (without probabilities) for inputs of its choice, and the model's type, parameters and training data are all unknown; the attacker trains a substitute model locally, crafts adversarial examples and transfers them to the remote oracle.

Questions
Machine learning models are vulnerable to adversarial example attacks, and adversarial examples can often transfer to models with different architectures or training data. The authors study the extent to which this transferability supports black-box attacks on remote classifiers whose internals are unknown.
Methods
The authors systematically measure intra-technique and cross-technique transfer rates across five machine learning techniques (DNN, LR, SVM, DT, kNN) and ensemble models; improve Jacobian-based data augmentation for substitute model training, introducing periodic step size (PSS) and reservoir sampling (RS); and propose algorithms for crafting adversarial examples against SVM and decision trees.
Experiments and results
On MNIST, adversarial examples crafted with LR achieve transfer misclassification rates of 91.43% and 87.42% against SVM and DT, respectively; an LR substitute model trained with only 800 oracle queries achieves misclassification rates of 96.19% and 88.94% against the Amazon and Google hosted classifiers, respectively; adversarial training defense is ineffective.
Limitations and future work
The authors note: an SVM substitute model cannot approximate a non-SVM oracle; the label matching rate for the DT oracle is only about 48%; the technique used by Google cannot be verified, and defenses such as distillation cannot be deployed; poisoning attacks at training time remain to be studied. Experiments were conducted only on MNIST.
Experimental setupdeep neural network (DNN)、logistic regression (LR) et al. · MNIST · misclassification rate、transferability rate et al.
Evaluated models
deep neural network (DNN)logistic regression (LR)support vector machine (SVM)decision tree (DT)nearest neighbors (kNN)ensemble (Ens.)Amazon Machine Learning oracleGoogle Cloud Prediction oracle
Benchmarks
MNIST
Metrics
misclassification ratetransferability ratelabel match rate
AI summaryand selection rationaleThe paper generalizes black-box attacks based on adversarial example transferability to multiple machine learning models, and with only 800 queries it caused the commercial classifiers of Amazon and Google to misclassify 96.19% and 88.94% of adversarial examples, respectively. The authors examined DNN, logistic regression, SVM, decision tree, kNN, and ensemble models on MNIST, and found that transfer both within the same technique and across techniques generally held; for example, examples constructed for logistic regression were misclassified by SVM and decision tree at 91.43% and 87.42%, respectively. They added adversarial example construction methods for SVM and decision tree, and used DNN and logistic regression to learn substitute models; a periodic step size raised the agreement rate between the substitute model and the target labels from 78.01% to 89.28% (DNN substitute model against DNN target). After introducing reservoir sampling, the number of queries for 10 rounds of data augmentation dropped from 102,400 to 3,600, and substitute model quality decreased slightly but remained better than the original method.

The paper generalizes black-box attacks based on adversarial example transferability to multiple machine learning models, and with only 800 queries it caused the commercial classifiers of Amazon and Google to misclassify 96.19% and 88.94% of adversarial examples, respectively. The authors examined DNN, logistic regression, SVM, decision tree, kNN, and ensemble models on MNIST, and found that transfer both within the same technique and across techniques generally held; for example, examples constructed for logistic regression were misclassified by SVM and decision tree at 91.43% and 87.42%, respectively. They added adversarial example construction methods for SVM and decision tree, and used DNN and logistic regression to learn substitute models; a periodic step size raised the agreement rate between the substitute model and the target labels from 78.01% to 89.28% (DNN substitute model against DNN target). After introducing reservoir sampling, the number of queries for 10 rounds of data augmentation dropped from 102,400 to 3,600, and substitute model quality decreased slightly but remained better than the original method.

Why it was selectedThe paper provides a matrix of adversarial example transfer rates across model types, and validates black-box attacks on the commercial classifiers of Amazon and Google with 800 queries.

Related events and news

0 events · 2 news mentions

News that mentions this paper

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors study how the transferability of adversarial examples supports black-box attacks on remote machine learning classifiers whose internals are unknown.

    Core question: whether the transferability of adversarial examples is sufficient for an attacker to mount black-box attacks on various machine learning classifiers (including non-deep-learning models) with almost no knowledge of the victim model.

    • Setting and importance: many machine learning models are vulnerable to adversarial examples—carefully crafted inputs that cause the model to output errors. The authors note that adversarial examples crafted for one model can often affect another model, even if the two differ in architecture or training set, as long as the task is the same. This can be used to sabotage fraud detection, bypass content filtering, or mislead autonomous navigation systems.
    • Shortcomings of existing methods: prior work (e.g., [22, 12, 24]) mainly studies transfer where at least one side is a neural network; the substitute-model black-box attack of [20] targets only DNNs, and the substitute model itself is also a DNN, with the number of queries growing exponentially with iterations.
    • Core observations and hypotheses: the authors propose two hypotheses—(1) intra-technique and cross-technique transferability are strong phenomena across the entire machine learning technique space; (2) black-box attacks are feasible in practical settings against any unknown machine learning classifier.
    • Paper contributions: the authors propose algorithms for crafting adversarial examples against SVM and decision trees; systematically measure the cross-technique transfer matrix; improve substitute model training with periodic step size and reservoir sampling; and mount black-box attacks on Amazon's and Google's commercial classification services.
    • Threat model:
      • Goal: force a remote classifier to misclassify inputs crafted by the attacker.
      • Knowledge: the attacker does not know the model type, parameters, or training data; the technique used by Google is completely unknown (the authors state that Amazon's documentation indicates it uses multinomial logistic regression).
      • Capability: only oracle access—can query predicted labels (without confidence probabilities) for chosen inputs.
      • Victim systems: classifiers hosted on Amazon Machine Learning and Google Cloud Prediction, as well as locally trained DNN, LR, SVM, DT, kNN, and ensemble models.
  2. What related research exists?

    The paper builds on adversarial examples and black-box substitute-model attacks, extends them to non-deep-learning models, and distinguishes itself from the knowledge transfer and adversarial machine learning literature.

    Adversarial examples and transferability

    • Szegedy et al. [22] (2014)—discovered the phenomenon of adversarial examples in neural networks and proposed the optimization problem formulation (Equation 1).
    • Goodfellow et al. [12] (2015)—proposed the fast gradient sign method for crafting adversarial examples, and noted that shallow models such as logistic regression are difficult to make resistant to adversarial examples.
    • Warde-Farley and Goodfellow [24] (2016)—studied adversarial perturbations of deep neural networks.

    Black-box attacks and substitute models

    • Papernot et al. [20] (2016)—proposed a substitute model training method based on Jacobian data augmentation, using the victim DNN as an oracle to label a synthetic training set, and carried out a black-box attack on the MetaMind API; this paper directly extends that method, introducing periodic step size and reservoir sampling, and generalizing the target from DNNs to multiple machine learning techniques.
    • Papernot et al. [19] (2016)—proposed a Jacobian-based iterative method for crafting adversarial examples.

    Adversarial machine learning security evaluation

    • Barreno et al. [1] (2006), Biggio et al. [5] (2014)—a series of works on machine learning security evaluation; the authors note that these works assume the model architecture and parameters are known, which differs from this paper's threat model.
    • Biggio et al. [4] (2013)—evasion attacks against SVM; the authors state that their own crafting method for SVM is more computationally efficient and requires no optimization.
    • Biggio et al. [7] (2012), Kloft and Laskov [15] (2010)—poisoning attacks at training time; the authors note that these works assume the training data or underlying model is known, and that the poisoning threat model remains largely unexplored.

    Knowledge transfer

    • Bucila et al. [9] (2006), Chen et al. [10] (2016), Hinton et al. [13] (2014)—model compression and knowledge distillation; the authors position substitute model learning as an instance of knowledge transfer.

    Baselines and benchmarks: Experiments are conducted on the MNIST dataset; the baseline for the black-box attack is the original (vanilla) Jacobian data augmentation method of [20], compared with the improved version adding PSS and RS.

    The authors position this paper as: a generalization of the black-box attack of [20]—from DNN targets to arbitrary machine learning classifiers, from DNN substitute models to LR and SVM, and with reduced query cost.

  3. How does the paper address this problem?

    The authors first systematically measure cross-model transfer rates, then improve substitute model training (PSS+RS), and finally carry out oracle black-box attacks on commercial APIs.

    The overall approach has three steps: systematically measure within-technique and cross-technique adversarial example transferability for five machine learning techniques on MNIST; improve Jacobian-based substitute model training to reduce query cost and generalize to LR and SVM substitute models; and exploit transferability to carry out black-box attacks against remote classifiers hosted by Amazon and Google.

    Formalization of transferability and experimental design

    • Formal definition: Adversarial example transferability ΩX(f, f′) = {x̃ ∈ X : f′(x̃) ≠ f′(x̃ + δx)}, i.e., the proportion of adversarial examples crafted for model f that are misclassified by model f'; divided into within-technique (same technique, different parameters/data) and cross-technique.
    • Model set: Five techniques—DNN, LR, SVM, DT, kNN—plus an ensemble model formed by majority vote of these 5 models (Equation 3); within-technique experiments split the training set into 5 disjoint subsets (10,000 samples each) to train 25 models.
    • Parameter settings: The fast gradient sign method's input variation parameter ε=0.3 (within-technique) or 0.25 (cross-technique, matching an L1 norm of 11.5%); the SVM crafting algorithm uses ε=1.5 (within-technique) or ε=5 (cross-technique); the DT crafting perturbation has an L1 norm of only 1.05% and cannot be matched.

    Substitute model learning (Section 4)

    • Jacobian-based data augmentation: Train a substitute model on an initial small training set labeled by the oracle (the first 100 samples of the MNIST test set), then extend training points along the sign direction of the substitute model's Jacobian (Equation 4), iterating for ρ rounds.
    • Periodic step size (PSS): Every τ rounds, multiply the step size λρ by −1 (Equation 5), which the authors state can improve oracle approximation quality; experiments set τ=3.
    • Reservoir sampling (RS): After round σ, each round augments with only κ new inputs randomly selected from the existing training set, reducing the number of oracle queries from n·2^ρ to n·2^σ+κ·(ρ−σ); experiments set σ=3, κ=400, using 3,600 queries over 10 iterations instead of 102,400.
    • SVM data augmentation: For SVM, propose an augmentation heuristic that selects new points along the direction orthogonal to the decision hyperplane (Equation 7).

    Adversarial example crafting algorithms (Section 6)

    • DNN/LR/kNN: Use the fast gradient sign method (Equation 8); kNN uses a soft-min smoothed variant (Equation 11) to allow differentiation.
    • SVM: Perturb the input along the direction orthogonal to the weight vector of the binary classifier corresponding to the correct class (Equation 13); the authors state this requires no optimization and is more computationally efficient than [4].
    • Decision tree: The authors state this is the first adversarial example crafting algorithm for decision trees (Algorithm 2)—searching the neighborhood of a leaf node for leaves of a different class, and modifying the sample so that the conditions along the path evaluate to the target leaf.

    Black-box attack procedure (Section 5.1)

    1. The attacker locally trains a substitute model (DNN or LR) using the method of Section 4 to approximate the remote oracle.
    2. Apply the corresponding adversarial example crafting algorithm to the substitute model to generate adversarial examples.
    3. Rely on within-technique or cross-technique transferability so that these examples also fool the remote oracle.

    Success criterion: the proportion of 10,000 adversarial examples that are misclassified by the remote oracle (misclassification rate).

  4. What experiments did the paper conduct?

    The authors measure transferability and substitute-model fidelity on MNIST, and achieve misclassification rates of 96.19% and 88.94% against the Amazon and Google classifiers with 800 queries.

    Experimental Setup

    • Models tested: local models of five techniques—DNN, LR, SVM, DT, kNN (5 each for within-technique experiments, 25 total; 1 each plus 1 ensemble model for cross-technique experiments); remote targets were classifiers trained with Amazon Machine Learning and Google Cloud Prediction.
    • Dataset: MNIST handwritten digit dataset (50,000 training samples, 10,000 validation samples, 10,000 test samples, 28x28 grayscale images).
    • Baseline: substitute models with vanilla Jacobian data augmentation (no PSS/RS).
    • Metrics: transfer rate (percentage of adversarial examples misclassified by other models); label match rate between substitute model and oracle; misclassification rate of the remote oracle.
    • Evaluation method: no human or LLM review; model-predicted labels were compared directly.
    • Sample size: 10,000 adversarial examples crafted per model; initial training set of 100 samples for substitute models.

    Main Results

    Cross-technique transfer matrix (Figure 3, rows = source model technique that crafted the examples, columns = victim model technique) and black-box attack results (Table 3, Table 4, ε=0.3):

    Swipe horizontally to read the table

    ExperimentConditionResultSource
    Cross-technique transferSource LR → target SVM91.43%Main text
    Cross-technique transferSource LR → target DT87.42%Main text
    Cross-technique transferTarget DT (each source)47.20%–89.29%Figure 3
    Cross-technique transferTarget DNN (each source)0.82%–38.27%Figure 3
    Amazon attackLR substitute, ρ=3, 800 queries96.19%Table 3
    Amazon attackDNN substitute, ρ=6, 6,400 queries96.78%Table 3
    Google attackLR substitute, ρ=3, 800 queries88.94%Table 4
    Google attackLR substitute+PSS+RS, ρ=6, 2,000 queries97.72%Table 4
    • The authors describe cross-technique transfer as a strong but heterogeneous phenomenon: DT is the most vulnerable and DNN the most resistant; the ensemble model is not resistant (44.14% against examples crafted by LR), possibly because each expert is itself vulnerable.
    • Within-technique transfer (Figure 2b-2f): all techniques are vulnerable; transfer rates between LR models exceed 94%, and DNN at least 49%; the diagonals for SVM, DT and kNN are more pronounced, and the authors say these techniques are somewhat more robust, possibly due to the margin constraint for SVM and to non-differentiability for DT and kNN.
    • The authors say Amazon documentation shows it uses "industry-standard" multinomial logistic regression, which explains why the LR substitute model performs better; the technique used by Google is not disclosed, and the experiments remained blind.

    Substitute Model Fidelity Experiments

    • What was measured: label match rates of DNN and LR substitute models against 5 oracles (DNN, LR, SVM, DT, kNN), comparing vanilla, PSS and PSS+RS variants (Table 2, ρ=9).
    • Results: DNN substitute+PSS matched the DNN oracle at 89.28% (vanilla 78.01%); LR substitute+PSS matched the LR oracle at 84.01% (vanilla 72.00%); the DT oracle had the lowest match rate (DNN substitute+PSS+RS only 48.62%); RS reduced queries from 102,400 to 3,600, with a match rate between vanilla and PSS.
    • Authors' interpretation: PSS improves approximation quality; RS greatly reduces the number of queries at a limited quality loss; the low DT match rate may be due to its non-differentiability.

    SVM Substitute Models and Defense Evaluation

    • SVM substitute models: 18 SVM substitute models were trained; the SVM substitute matched only 79.80% of SVM oracle labels, and only 11.98% and 11.97% of DNN and LR oracle labels, with no improvement from PSS/RS. The authors believe SVM cannot transfer knowledge from non-SVM oracles, possibly due to the specificity of SVM training and decision boundaries.
    • Defense evaluation: adversarial examples were added to the MNIST training set to retrain the Google oracle (adversarial training); the new oracle's accuracy was 91.25%, but examples crafted by a new DNN substitute were still misclassified at 94.2% (ρ=3) and 100% (ρ=6). The authors call this defense ineffective, possibly because Google uses a shallow technique, but this cannot be verified; the distillation defense [21] could not be deployed because the training process could not be controlled.

    Other Ablations and Analyses

    • The LR substitute model converges faster: it reaches the asymptotic match rate at ρ=4 (1,600 queries) and has lower computational cost (Section 4.3).
    • Experiments with decreasing step magnitude λ brought no substantial improvement (Section 4.1).
    • In the Amazon attack, PSS+RS reduced the DNN substitute's queries from 6,400 to 2,000, with the misclassification rate falling only from 96.78% to 95.68% (Table 3).
    • The LR substitute model did not benefit from PSS+RS in the Amazon attack; the authors suggest this may be because LR substitute learning converges quickly (Section 5.2).
  5. What are some points that can be further explored?

    The authors note limitations such as SVM substitute learning, DT approximation, and defense deployment, and suggest future improvements to substitute learning and exploration of poisoning attacks.

    Limitations noted by the authors and future directions

    • SVM substitute model fails: An SVM substitute cannot approximate a non-SVM oracle (Section 4.4); the authors believe future work should study substitute enhancement techniques to confirm this finding.
    • Low DT oracle match rate: The label match rate of a DNN substitute against a DT oracle is only about 48%, which the authors believe may be because decision trees are non-differentiable (Section 4.2).
    • Defenses ineffective or undeployable: Adversarial training defenses are ineffective; defenses such as distillation cannot be deployed because of the lack of control over Google's training process, and the authors cannot offer advice on how to protect that service (Section 5.3).
    • Google technology unknown: The authors cannot confirm the machine learning technology used by Google Cloud Prediction, and the claim that the defense failure is "very likely" due to a shallow model cannot be verified (Section 5.3, Section 7).
    • Future directions: Continue improving substitute model learning to increase accuracy and adversarial example transferability; poisoning attacks at training time remain largely unstudied (Section 8).
    • Open problem of input validation: The authors state that validation of inputs to machine learning algorithms remains an open problem (Section 8).

    Scope of experiments

    • The local models tested are five techniques—DNN, LR, SVM, DT, kNN—plus 1 ensemble model; the remote targets are 1 classifier each on Amazon and Google, all trained on MNIST.
    • The dataset is MNIST only; within-technique experiments use 5 models per technique (25 in total), and cross-technique experiments use 1 model per technique.
    • The oracles in the substitute model experiments are the 5 local technique models and the 2 remote classifiers; the substitute model types are DNN, LR and SVM.
    • The black-box attacks target only the two MLaaS platforms Amazon and Google, with an initial substitute training set of 100 samples and a minimum of 800 queries.
    • Defense evaluation covers only adversarial training; the paper reports no human evaluation, statistical significance tests or repeated runs.
  6. Summarize the main content of the paper

    The authors systematically study the transferability of adversarial examples across machine learning techniques, and on that basis carry out black-box attacks against the Amazon and Google classifiers with 800 queries.

    An empirical study (2016) on the transferability of adversarial examples and their application in black-box attacks. The problem the authors address: whether adversarial example transferability is prevalent across various machine learning techniques and sufficient to support practical black-box attacks against remote classifiers whose internals are unknown.

    • Method highlights: On MNIST, measure within-technique and cross-technique transferability for DNN, LR, SVM, DT, kNN and an ensemble model; propose crafting algorithms for SVM (perturbing along the direction orthogonal to the decision hyperplane) and decision trees (searching for adversarial leaves and modifying path conditions); improve Jacobian-based substitute model training with periodic step size (PSS) and reservoir sampling (RS), reducing the number of queries from exponential growth to linear.
    • Transferability results: Cross-technique transfer is prevalent—examples crafted on LR achieve transfer misclassification rates of 91.43% and 87.42% against SVM and DT, respectively; DT is the most vulnerable (47.20%–89.29%), DNN the most resistant (0.82%–38.27%); the ensemble model is not resistant.
    • Substitute model results: DNN and LR substitute models achieve label match rates above 80% against most oracles (e.g., DNN substitute + PSS reaches 89.28% against the DNN oracle), but only about 48% against the DT oracle; SVM substitute models cannot approximate non-SVM oracles.
    • Black-box attack results: An LR substitute model trained with only 800 oracle queries achieves misclassification rates of 96.19% and 88.94% against the Amazon- and Google-hosted classifiers, respectively; PSS+RS further raises the Google attack's misclassification rate to 97.72% at 2,000 queries.
    • Defense results: Adversarial training defenses are ineffective—the retrained Google oracle is still misclassified at rates up to 100%.
    • Authors' conclusion: Black-box attacks are generally applicable to machine learning classifiers of all kinds, regardless of their structure; the authors call for validation of inputs to machine learning systems and state that this remains an open problem.

Related papers

Show 3 more papers
Read originalarxiv.org