Skip to content

Practical Black-Box Attacks against Machine Learning

Practical Black-Box Attacks against Machine Learning

arXiv· arXiv:1602.02697· AsiaCCS 2017· Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, Ananthram SwamiPennsylvania State University; OpenAI; University of Wisconsin; US Army Research LaboratoryIndexed by this site · 23 days ago· Originally publishedSelected AttacksSelectedAttention45

Translation in progress — untranslated text is omitted.

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

AI summaryNicolas Papernot and other researchers state in the paper that they provide the first practical demonstration of a black-box attack against remotely hosted machine learning models, in which the attacker need not know the model architecture or training data and whose only capability is querying the target model for labels on chosen inputs. The attack has two steps: first, query the target model with synthetic inputs to obtain labels, and train a substitute model with Jacobian-based data augmentation; then, construct adversarial examples on the substitute model and use transferability to cause the target model to misclassify. On a DNN hosted by MetaMind, 84.24% of adversarial examples were misclassified; using a logistic regression substitute model to attack models hosted by Amazon and Google, the misclassification rates were 96.19% and 88.94%, respectively. The paper also shows that the attack can bypass defense strategies previously thought to increase the difficulty of constructing adversarial examples; the authors disclosed to MetaMind, Amazon, and Google, and the experiments used only models under their own accounts and caused no harm.

Questions
Existing adversarial example attacks require knowledge of the model's internal structure or possession of training data, making them difficult to threaten real deployed remote models. The problem the authors address is: when an attacker can only query a remote DNN and observe the returned labels, can the attacker actually construct adversarial examples that it misclassifies?
Methods
The attacker collects a small number of initial samples, uses the target model as an oracle to assign labels, iteratively trains a local substitute model, and uses Jacobian-based data augmentation to synthesize new samples along directions to which the model output is sensitive; then, on the substitute model, the attacker constructs adversarial samples using the Goodfellow or Papernot algorithm, relying on transferability to make the target model misclassify.
Experiments and results
The authors report: the MetaMind remote DNN misclassified 84.24% of adversarial samples (ε=0.3); the Amazon and Google hosted models misclassified 96.19% and 88.94%, respectively (LR substitute model, 800 queries); the local GTSRB oracle had a transferability of 64.24%–69.03% at ε=0.3; the attack can bypass adversarial training (small ε) and defensive distillation.
Limitations and future work
The authors note: defending against bounded perturbations (e.g., adversarial training with ε=0.3) can prevent this black-box attack; the statistical analysis of transferability is only an intuitive explanation, and a formal treatment is left for future work; only one hypothesis-testing method, chi-square, was used, and other methods are left for future work. The experiments cover two image datasets, MNIST and GTSRB, and 5 classifier types, and do not involve other data types.
Why it was selectedThe paper shows that an attacker needs only to query output labels to train a substitute model and transfer adversarial examples; teams providing remotely hosted machine learning services can use this to assess their own exposure.

The paper shows that an attacker needs only to query output labels to train a substitute model and transfer adversarial examples; teams providing remotely hosted machine learning services can use this to assess their own exposure.

Related events and news

0 events · 2 news mentions

News that mentions this paper

Related papers

Show 3 more papers
Read originalAsiaCCS 2017