Practical Black-Box Attacks against Machine Learning
Practical Black-Box Attacks against Machine Learning
Translation in progress — untranslated text is omitted.
Nicolas Papernot and other researchers state in the paper that they provide the first practical demonstration of a black-box attack against remotely hosted machine learning models, in which the attacker need not know the model architecture or training data and whose only capability is querying the target model for labels on chosen inputs. The attack has two steps: first, query the target model with synthetic inputs to obtain labels, and train a substitute model with Jacobian-based data augmentation; then, construct adversarial examples on the substitute model and use transferability to cause the target model to misclassify. On a DNN hosted by MetaMind, 84.24% of adversarial examples were misclassified; using a logistic regression substitute model to attack models hosted by Amazon and Google, the misclassification rates were 96.19% and 88.94%, respectively. The paper also shows that the attack can bypass defense strategies previously thought to increase the difficulty of constructing adversarial examples; the authors disclosed to MetaMind, Amazon, and Google, and the experiments used only models under their own accounts and caused no harm.
- Existing adversarial example attacks require knowledge of the model's internal structure or possession of training data, making them difficult to threaten real deployed remote models. The problem the authors address is: when an attacker can only query a remote DNN and observe the returned labels, can the attacker actually construct adversarial examples that it misclassifies?
- The attacker collects a small number of initial samples, uses the target model as an oracle to assign labels, iteratively trains a local substitute model, and uses Jacobian-based data augmentation to synthesize new samples along directions to which the model output is sensitive; then, on the substitute model, the attacker constructs adversarial samples using the Goodfellow or Papernot algorithm, relying on transferability to make the target model misclassify.
- The authors report: the MetaMind remote DNN misclassified 84.24% of adversarial samples (ε=0.3); the Amazon and Google hosted models misclassified 96.19% and 88.94%, respectively (LR substitute model, 800 queries); the local GTSRB oracle had a transferability of 64.24%–69.03% at ε=0.3; the attack can bypass adversarial training (small ε) and defensive distillation.
- The authors note: defending against bounded perturbations (e.g., adversarial training with ε=0.3) can prevent this black-box attack; the statistical analysis of transferability is only an intuitive explanation, and a formal treatment is left for future work; only one hypothesis-testing method, chi-square, was used, and other methods are left for future work. The experiments cover two image datasets, MNIST and GTSRB, and 5 classifier types, and do not involve other data types.
The paper shows that an attacker needs only to query output labels to train a substitute model and transfer adversarial examples; teams providing remotely hosted machine learning services can use this to assess their own exposure.
Related events and news
0 events · 2 news mentions
News that mentions this paper
- FAR.AIcites the paper's id or linkFAR.AI 提出对抗脆弱性可能让主流对齐方案失效
Papernot et al, 2017 (Table 4) found that adversarial training against examples with small perturbations produced networks that exhibit gradient masking.
- Nicholas Carlini Writingcites the paper's id or linkNicholas Carlini 给出评估对抗样本防御的建议
To ensure that the model is not causing various forms of gradient masking, it is worth attempting gradient-free attacks.
Related papers
- On the Limitation of Local Intrinsic Dimensionality for Characterizing the Subspaces of Adversarial ExamplesAnalysis / interpretabilityarXiv
Show 3 more papers
- Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial ExamplesAnalysis / interpretabilityNeurIPS