Stealing machine learning models via prediction APIs

1. 01What problem does this paper attempt to solve?

   The study examines whether a confidential model can be stolen using only black-box queries to an MLaaS prediction API; the authors give an affirmative answer.

   Core question: can an attacker, using only black-box queries to a public prediction API, extract a copy that is functionally equivalent to a confidential ML model ("steal" the model).

   • Scenario and motivation: ML models need to be kept confidential because of sensitive training data, commercial value, or use in security detection, but MLaaS platforms such as Amazon, Google, Microsoft, and BigML allow model owners to open access and charge per query, creating a tension between confidentiality and openness. The authors note that stealing a model can evade query fees, leak training data privacy, and serve as a stepping stone for evading detection.
   • Shortcomings of existing work: traditional learning theory (membership query setting) assumes queries return only class labels; the authors state that actual MLaaS APIs return high-confidence scores and accept partial feature vectors, a setting more favorable to the attacker, while learning-theory algorithms have limited practical study.
   • Core observation: confidence outputs can be converted into equations about model parameters (e.g., a logit transform for logistic regression), and decision tree confidence can serve as a pseudo-identifier of the path; partial feature queries can further reduce the query count.
   • Paper contributions: propose an equation-solving attack (applicable to models with a logistic output layer) and a decision tree path search algorithm, demonstrated on the BigML and Amazon online services; and evaluate a retraining attack under the countermeasure of returning only class labels.
   • Threat model:
     • Goal: extract f̂ that is (nearly) 100% consistent with the target model f over the input space of interest, with as few queries as possible.
     • Knowledge: black-box access; may or may not know the model type, training algorithm, hyperparameters, or training data distribution; does not possess the training data itself (other than summary statistics provided by the API).
     • Capabilities: direct queries (arbitrary feature vectors) or indirect queries (queries over the input space M after feature extraction ex); can exploit incomplete queries and batch prediction.
     • Victim systems: MLaaS platforms such as BigML and Amazon; in the experiments the authors train models on their own accounts to simulate the victim.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four categories: learning theory, adversarial machine learning and evasion, model inversion, and model explanation and compression.

   Learning theory (membership queries)

   • Valiant's PAC learning [54] and variants by Angluin [3], Benedek and Itai [8], etc.—study learning an unknown concept using membership queries; the authors note that this paper's setting differs: the prediction API leaks more information than learning theory assumes.
   • Kushilevitz and Mansour [33]—extract Boolean decision trees using membership queries, with complexity polynomial in d and 2^δ; the authors state that their tree model (Boolean features and outputs) has limited practical use, whereas this paper's algorithm applies to the practical tree models of MLaaS and, under the unique-leaf-identifier assumption, has an exponential speedup (Appendix C).
   • Lowd and Meek [36]—a black-box extraction attack against linear binary classifiers (motivated by spam evasion); this paper generalizes it to model-agnostic line-search retraining and uses it as a baseline for the labels-only setting.

   Adversarial machine learning and evasion

   • Biggio et al. [9], Šrndić and Laskov [56], Nelson et al. [39], Stevens and Lowd [51]—evasion attacks and query strategies; the authors relay that both theoretical and practical experience consider extraction harder than evasion.
   • Papernot et al. [42] (parallel work)—use data augmentation to extract a "substitute" network and then construct adversarial examples for transfer; the authors note that their goal is lower extraction accuracy (Rtest ≈ 80%) to support adversarial example transfer, unlike this paper's pursuit of high fidelity.

   Model inversion and privacy

   • Fredrikson et al. [23, 24], Ateniese et al. [4]—model inversion infers training data from model access; this paper combines extraction with white-box inversion to reduce remote queries.

   Non-adversarial model extraction

   • Rule extraction and tree extraction [2, 19, 53], model compression and distillation [16, 27]—improper extraction in non-adversarial settings, used to explain and compress neural networks.

   Baselines and benchmarks: the baselines for the labels-only experiments are the Lowd-Meek attack [36] and uniform retraining; the datasets are the synthetic and public datasets in Table 3 and the BigML public model repository (Table 5). The authors position this paper as practice-oriented model extraction research that exploits the rich outputs of real APIs, in contrast to purely theoretical settings.

   Collapse this answer

3. 03How does the paper address this problem?

   Attacks are divided into two categories by API output: equation solving and path search when confidence is available, and retraining strategies when only labels are available.

   Overall approach: turn the rich outputs of a prediction API (confidence, partial query responses) into equations and identities about model parameters or structure, and design attacks for two settings, "with confidence" and "class labels only".

   Problem formalization

   • The model is a function f : X → Y; attacker A seeks f̂ with the fewest queries, measuring agreement with f using test error Rtest and uniform error Runif, with extraction accuracy 1 − Rtest, 1 − Runif; total variation distance is used when comparing class probabilities.
   • Focus on proper extraction: f̂ and f belong to the same model class; improper extraction is left as an open question (Appendix D).

   Equation-solving attacks (with confidence)

   • Binary logistic regression: f₁(x) = σ(w · x + β); each query gives a linear equation w · x + β = σ⁻¹(f₁(x)), and d + 1 linearly independent random queries suffice to solve for w and β; the queries are non-adaptive and can be submitted in a single batch.
   • Softmax / OvR multiclass regression and MLP: each query produces c nonlinear equations, solved by minimizing logistic loss (BFGS for MLR, SGD for 1,000 epochs in theano for MLP); the query budget is α times the number of parameters (α from 0.5 up to 5).
   • Kernel logistic regression (KLR): the solution of the equation system contains representers (a subset of training points), so extraction leaks training data; the experiments use an RBF kernel with 20 or all 1,257 training points as representers.

   Decision tree path search

   • Use (predicted label, confidence) as a pseudo-identity for nodes/leaves, defining a leaf-identity oracle and a node-identity oracle (the latter supports incomplete queries containing ⊥).
   • Algorithm 1 (path search): starting from a random input, use LINE SEARCH (continuous features, precision ε = 10⁻³) and CATEGORY SPLIT (categorical features) to find the predicate reaching the current leaf, and generate new queries for unvisited leaves.
   • Top-down variant: use incomplete queries to determine split features and criteria layer by layer from the root; it is somewhat robust to duplicate identities but may miss some splits.

   Online attacks and feature engineering reversal (Section 5)

   • BigML: after obtaining feature information from the public model repository, extract using the tree attack above.
   • Amazon: first reverse the feature extraction ex—one-hot-encoding can be reproduced locally, and quantile binning uses line search to find bin boundaries (granularity 10⁻³); then use the property that "missing features are set to 0 in X" to construct single-unknown equations; use an extract-and-test strategy to determine the model class (the authors state that the results indicate Amazon uses softmax regression).

   Label-only attacks (Section 6)

   • Lowd-Meek attack [36]: line search finds points near the decision boundary, applicable to linear binary classifiers; this paper gives an extension of polynomial-kernel SVM in the transformed space.
   • Three retraining strategies: uniform retraining (baseline), line-search retraining (a model-agnostic generalization of Lowd-Meek), and adaptive retraining (active learning, selecting points along the decision boundary of the current f̂ over r rounds); the budget is α times the number of parameters, 0.5 ≤ α ≤ 100.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Local benchmarks plus BigML and Amazon online cases: equation solving and path search achieve 100% equivalence across multiple model classes.

   Experimental setup

   • Model classes tested: binary/multiclass logistic regression (softmax, OvR), MLP (1 hidden layer, h = 20), kernel logistic regression, decision trees (including regression trees), RBF-kernel SVM.
   • Datasets: the 4 synthetic sets and 9 public sets in Table 3 (e.g., Adult 48,842 records, Digits 1,797 records); the 8 BigML public decision trees in Table 5; the inversion experiments used AT&T Faces (40 people, 92 × 112 pixels).
   • Training protocol: 70% training / 30% test, one-hot encoding for categorical features, numerical features scaled to [−1, 1]; Runif estimated with |U| = 10,000.
   • Metrics: 1 − Rtest, 1 − Runif, RTV, number of queries, time; experiments were run on a laptop with a 2-core Intel CPU @3.1GHz and 16GB RAM.
   • Online attacks: the authors trained models under their own accounts to simulate victims and used only public APIs; disclosure to the affected providers was made in February 2016.
   • Baselines: the Lowd-Meek attack and uniform retraining in the label-only experiments.

   Main results

   Cost of extracting 100% equivalent models on online services (Table 1):

   Swipe horizontally to read the table

   Service	Model type	Dataset	Queries	Time (s)
   Amazon	Logistic Regression	Digits	650	70
   Amazon	Logistic Regression	Adult	1,485	149
   BigML	Decision Tree	German Credit	1,150	631
   BigML	Decision Tree	Steak Survey	4,013	2,088

   • The authors state that the time is mainly determined by service latency (Amazon about 100ms/query, BigML about 500ms/query).
   • Local equation solving (Table 4, Adult 'Race', c = 5): softmax and OvR reach 1 − Rtest = 1 − Runif = 100.00% with 530 queries (equal to the number of parameters); MLP (2,225 parameters) reaches 99.96% / 99.99% with 11,125 queries.
   • Decision trees (Table 6): the 4 trees whose leaf identities are all unique reach 100.00% under both attacks; the top-down variant with incomplete queries uses fewer queries in most cases (e.g., IRS Tax Patterns drops from 101,057 to 29,609), but on Steak Survey Algorithm 1 reaches only 92.45% / 86.40% because it visits only some leaves.

   Training data leakage and model inversion

   • KLR leakage (Figure 2): solving a system of 50,000 equations / 1,490 unknowns extracts 20 representers; the authors state that the extracted representers leak information about individual training points, and that underestimating the number of representers (10) appears to leak the "average representer" of each class.
   • Extraction + white-box inversion (Section 4.1.4): black-box inversion on AT&T Faces requires about 20,600 queries per face; first extracting the softmax model with 41,216 queries (RTV about 10⁻³), then inverting all 40 faces locally in white-box fashion, the authors state that remote queries are reduced by about 20 times, total time is shorter, and the results are visually indistinguishable from using the true f.

   Label-only attacks

   • Linear models (Figure 4): Lowd-Meek is most effective when the budget is sufficient (on average about 2,050 queries, at most 5,650 queries); adaptive retraining reaches over 99% agreement with an average of 827 queries—about 50 times the query count of equation solving.
   • Softmax and RBF-kernel SVM (Figures 5, 6): adaptive retraining is best; 100 · c · (d + 1) queries (on average 26,000, at most 65,000) bring softmax extraction accuracy above 99.9%; SVM reaches over 99% at the maximum budget (on average 2,050 queries).
   • Neural networks (Section 6.3): the 100 · k budget (on average 108,200 queries per model) reaches only 99.16% in the Rtest direction and 98.24% in the Runif direction; the authors state that it is unclear whether higher budgets still yield monetary benefit.

   Other ablations and analyses

   • Confidence rounding (Figure 7): rounding softmax probabilities to 4–5 decimal places has no effect on equation solving; rounding to 2–3 places weakens the attack but it still outperforms label-only adaptive retraining.
   • Decision tree confidence rounding: IRS Tax Patterns and Email Importance are affected only when rounded to 2 decimal places; for the remaining trees the attack is severely weakened at 3–4 places (authors' wording).
   • Amazon feature extraction reversal (Table 7): Circles 278 queries, Iris 644 queries, Adult 1,485 queries; the extracted models match 100% on test inputs, at a cost of $0.03–$0.15.
   • Improper extraction (Appendix D): extracting a 530-parameter softmax with an MLP of 1,000 hidden nodes (111,005 parameters) plus 5,300 queries reaches only 99.5% label agreement and TV distance 10⁻², weaker than proper extraction.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list open problems such as ensemble methods, DP protections, and indirect queries; the experiments cover only self-trained models and a few model classes.

   Limitations noted by the authors and future directions

   • No experiments were conducted with ensemble methods (such as random forests) as the target; the authors suspect they may be harder to extract precisely (Section 7).
   • Decision tree extraction with only class labels was not evaluated; the authors state that, due to the greedy nature of the learning algorithm, they do not expect retraining to be effective for trees (Section 6 footnote).
   • Further study of indirect queries plus only class labels is left as future work (Section 6.1).
   • How to apply differential privacy to model parameters and what privacy budget would be required are left by the authors as open questions (Section 7).
   • Improper extraction (extracting a simpler representation when the model class is unknown) is left as an open question (Section 3, Appendix D).
   • Evaluating label-only attacks and more refined countermeasures on production-grade ML services is a future direction (Section 9).

   Scope of experimental coverage

   • Online attacks were validated only on two services, BigML and Amazon, and on models trained under the authors' own accounts (Section 5); Google, Microsoft, and PredictionIO appear only in the feature survey in Table 2.
   • The model classes tested were LR, softmax/OvR, MLP (single hidden layer, h = 20), KLR, decision trees, and RBF-kernel SVM; KLR was not supported by any MLaaS provider at the time (Section 4.1.3).
   • The datasets were the synthetic and public datasets in Table 3 and the BigML public model repository (Table 5); the authors state that these are substitutes for proprietary data.
   • More feature extractors such as text analysis and Cartesian products were not analyzed; the authors believe they may also be easy to reverse (Section 5.3).
   • The paper does not report the number of repetitions or variance for the label-only experiments; Figures 4–6 are average errors on each dataset.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors systematically study MLaaS model stealing: confidence outputs allow multiclass models to be extracted precisely with few queries, and label-only defenses still leave room.

   A paper on ML model extraction (stealing) attacks, targeting MLaaS scenarios that expose a query interface but require confidentiality.

   • Problem: the attacker uses only black-box queries (possibly including confidence and support for incomplete feature vectors), and the goal is to obtain, with the fewest queries, a copy that is (nearly) 100% consistent with the target model; motivations include evading query fees, leaking training data, and aiding evasion of detection.
   • Method: for models with a logistic output layer, random queries are used to set up a system of equations to solve for parameters; for decision trees, a path search algorithm is proposed that treats confidence as a path pseudo-identifier and uses incomplete queries for top-down extraction; for Amazon, the feature extraction (one-hot encoding and quantile binning) is also reverse-engineered.
   • Online results: on BigML and Amazon, 100% equivalent logistic regression and decision tree models are extracted with 650–4,013 queries (Table 1), with the lowest per-attack cost about $0.03 (Table 7).
   • Privacy consequences: extracting a kernel logistic regression leaks information about training points (Figure 2); extracting first and then performing white-box inversion of faces requires about 20 times fewer remote queries than 40 black-box inversions (Section 4.1.4).
   • Labels-only setting: adaptive retraining reaches over 99% consistency for linear models with an average of 827 queries and over 99.9% for softmax with an average of 26,000 queries, but uses up to 100 times more queries than equation solving; neural networks reach about 99% under a budget of 100 · k, and the authors state that the monetary benefit is questionable.
   • Authors' conclusion: simple countermeasures such as confidence rounding have limited effect, and new model extraction defenses are needed; ensemble methods, DP protection parameters, and tree extraction under labels only are left for future work.

   Collapse this answer

Related papers