High-accuracy and high-fidelity neural network extraction
High Accuracy and High Fidelity Extraction of Neural Networks
The authors classify model extraction attacks by accuracy and fidelity, propose more query-efficient learning-based extraction and functionally equivalent extraction for two-layer ReLU networks, reaching 100% test-set fidelity on MNIST.
Attacker goals fall into two categories: theft (stealing the model, pursuing task accuracy) and reconnaissance (pursuing fidelity, preparing for subsequent white-box attacks such as adversarial examples and membership inference). The attacker has only query access to a remote oracle, and the model access level ranges from label to logits; the functionally equivalent attack assumes 64-bit floating-point logit outputs are observable, and the learning-based attack assumes task-relevant public unlabeled data. The victim system is a remotely deployed neural network classifier.
- In model extraction attacks, an attacker steals a copy of a model using only query access to a remote model. The authors argue that prior work conflated two goals: performing well on the task (accuracy) and matching the victim model's predictions on all inputs (fidelity), which conflict with each other for imperfect victim models.
- The authors classify extraction attacks by accuracy and fidelity and design one attack for each: the learning-based attack uses the victim model as a labeling oracle and combines semi-supervised learning (rotation loss, MixMatch) to improve query efficiency; the functionally equivalent attack directly reconstructs the weights of a two-layer ReLU network through critical point search, weight recovery, sign recovery and least squares; they additionally propose a hybrid strategy that uses learning to repair the numerical errors of direct extraction.
- The learning-based attack made the extracted model exceed the ImageNet label baseline on a WSL model trained on 1 billion proprietary images; MixMatch exceeded full supervision with 4,000 queries using only 250 queries on SVHN/CIFAR10. The function-equivalence attack reached 100% test-set fidelity on a small MNIST model, the hybrid strategy raised the fidelity of a 128-neuron model from 80% to 99.75%, and the adversarial example transfer success rate was 100%.
- The authors note that the function-equivalence attack applies only to two-layer networks, and extending it to deeper networks requires addressing critical-point layering, indirect control of inputs, and numerical error; the attack assumes double-precision logit access, and defenses that restrict output information break it; the gradient-matching hybrid method requires gradient queries, and the authors leave extending it and removing this assumption to future work.
- Evaluated models
- WSL modelWideResNet-28-2one-layer fully-connected neural networks (16–512 hidden units, MNIST/CIFAR-10)Fashion-MNIST oracle
- Benchmarks
- ImageNetSVHNCIFAR10Fashion-MNISTMNIST
- Metrics
- task accuracyfidelitytop-5 accuracytop-5 fidelitytransferability ratequery count
Researchers from Google and other institutions classify model extraction attacks by two objectives, task accuracy and prediction fidelity, and implement functionally equivalent extraction that directly recovers the weights of a two-layer ReLU network using only input-output access. The learning-based attack uses the victim model to label public data, improving query efficiency by 16 times over previous methods; after about 130,000 queries to a WSL model (193 million parameters) trained on 1 billion Instagram images, the ImageNet accuracy of the extracted ResNet-v2-50 and ResNet-v2-200 models was 1.0% and 1.9% higher, respectively, than training with only ImageNet labels. Experiments show that learning-based extraction has a fidelity ceiling; even with full knowledge of the victim model's training setup, fidelity on Fashion-MNIST is only about 93.4%.
Why it was selectedThe paper classifies model extraction by task accuracy and prediction fidelity, and demonstrates a functionally equivalent attack that can precisely recover the weights of a two-layer network with only input-output access.
Paper analysis
What problem does this paper attempt to solve?
The authors classify model extraction by two goals, accuracy and fidelity, and design attack methods separately for each goal.
Core question: to what extent can an attacker steal a copy of a remotely deployed model using only query access to it—both performing well on the task (accuracy) and matching the victim model's predictions on all inputs (fidelity).
- Problem setting and importance: neural networks are often deployed as prediction services and exposed to potential attackers, while the models themselves are proprietary assets due to high training costs and security and privacy concerns. The authors state that prior work has found that attackers with query access can steal copies highly consistent with the victim model.
- Shortcomings of existing methods: the authors argue that most prior work considers only the accuracy goal, and that task accuracy and fidelity extraction usually use generic learning-based methods; the authors argue that learning-based strategies have inherent limitations and cannot achieve true high-fidelity (i.e., functionally equivalent) extraction.
- Core observation: the authors point out that accuracy and fidelity conflict for imperfect victim models—high-fidelity extraction should reproduce the victim model's errors, whereas high-accuracy extraction should make correct predictions; functionally equivalent extraction is the limiting case of fidelity.
- What the paper proposes: a taxonomy of extraction attacks around accuracy and fidelity; a learning-based attack that uses the victim model as a labeling oracle and combines semi-supervised learning; a functionally equivalent extraction attack for two-layer ReLU networks that requires only input-output access; and a hybrid strategy combining the two.
- Threat model:
- Attacker goals: divided into theft (stealing the model itself, pursuing task accuracy) and reconnaissance (pursuing fidelity, preparing for subsequent white-box attacks such as adversarial examples, membership inference and overlearning).
- Attacker knowledge: divided into three categories: domain knowledge (task-relevant knowledge, such as public unlabeled data), deployment knowledge (architecture, training process, training data), and model access (the information returned by queries).
- Attacker capabilities: can only query the oracle; model access levels range from label only, label and score, top-k scores, scores, to logits; the functionally equivalent attack assumes 64-bit floating-point logit outputs are observable.
- Victim system: a remotely deployed neural network classifier (experiments include a WSL model trained on 1 billion proprietary images and fully connected networks on MNIST/CIFAR-10).
What related research exists?
Related work is organized into four categories: extraction objectives, learning-based methods, direct recovery, and defenses; the authors place each existing attack into their classification system.
Learning-based extraction (accuracy and fidelity)
- Tramèr et al. [11]—stealing models through prediction APIs, covering linear models and neural networks, with the goals of task accuracy and fidelity; this paper's fully supervised baseline comes from this work.
- Copycat CNN [16], Knockoff Nets [12]—learning CNN functionality with random unlabeled data or black-box queries, with the goal of task accuracy.
- Papernot et al. [7], Pal et al. [15]—active learning-based extraction, with the goal of fidelity; Papernot et al.'s motivation is to use the extracted model to mount black-box adversarial example attacks.
Functionally equivalent/direct recovery extraction
- Lowd & Meek [8], Chandrasekaran et al. [13]—functionally equivalent extraction for linear models, requiring only labels.
- Milli et al. [19]—direct weight recovery for two-layer neural networks, but assumes the attacker can make gradient queries; the authors point out that this attack was not actually implemented, and this paper's functionally equivalent attack is its extension that removes the gradient query assumption.
- Batina et al. (CSI NN) [25]—extracting neural network weights through power side channels; the authors state this is the only previously actually implemented direct extraction, but its threat model is stronger, and this paper's attack does not require side channels.
Theoretical difficulty and related analyses
- Das et al. [27]—prove that deep random networks require exp(O(h)) samples to learn under the SQ learning model, and the authors use this to argue that most learning-based extraction methods inherit this inefficiency.
- Carlini et al. [44]—prototypicality metric, which this paper's Appendix B uses to analyze the correlation between fidelity and sample prototypicality.
Defenses
- Limiting the information per query: perturbing returned probabilities [11, 13, 49], returning only some class probabilities [11], returning only classes [11, 13], sampling from a distribution over model parameters [13, 50].
- Distinguishing benign from malicious users: analyzing query patterns [51, 52]; the authors point out that non-adaptive attacks (such as supervised or MixMatch extraction) can bypass query-pattern-based detection.
- Others: watermarking [54, 55], obfuscation [56], secure multi-party computation [57]; there is also work on stealing hyperparameters [53] and architecture information [14], which the authors consider orthogonal and compatible with this paper.
Baselines and benchmarks: the baselines for learning-based attacks are models trained directly with ImageNet ground-truth labels (the "ImageNet" column in Table 2) and fully supervised extraction (FS); the benchmarks are ImageNet, SVHN, CIFAR10, Fashion-MNIST, MNIST.
The authors position this paper as: a systematic taxonomy of the extraction attack space around accuracy and fidelity, with the learning-based attack improving on existing techniques in query efficiency, and the functionally equivalent attack being what the authors call the first practical functionally equivalent extraction requiring only input-output access.
How does the paper address this problem?
The authors design one attack for each of the two objectives: semi-supervised learning-based extraction improves query efficiency, and the function-equivalence attack directly reconstructs the weights of a two-layer network.
The overall approach is to divide and conquer by target: learning-based attacks use the victim model as a labeling oracle to train an extraction model, and use semi-supervised learning to improve query efficiency; functionally equivalent attacks exploit the piecewise-linear structure of ReLU networks to directly recover the weights of a two-layer network through queries; hybrid strategies use learning to repair the numerical errors of direct extraction.
Taxonomy and theoretical difficulty
- Targets are divided into three levels: task accuracy extraction (matching or exceeding the victim model's accuracy), fidelity extraction (maximizing Prx∼ DF S(Ô(x), O(x)); this paper considers only label agreement), and functionally equivalent extraction (identical predictions on all inputs).
- The authors give three difficulty results: Theorem 1 constructs a network of width 3k and depth 2 that requires Θ(p^k) queries to extract; Theorem 2 (informal) states that, given the weights, deciding whether two networks are functionally equivalent is NP-hard; Theorem 3 (cited from [27]) states that a random network requires exp(O(h)) samples under the SQ model.
- On this basis the authors argue that a practical functionally equivalent attack must make additional assumptions to avoid these pathological cases.
Learning-based extraction (Section 4)
- Fully supervised baseline: the attacker queries the oracle to obtain labels and scores for training points, and trains the extraction model with a cross-entropy loss; the experiments use a distillation temperature T=1.5 (selected by random search), with architectures ResNet-v2-50 and ResNet-v2-200.
- Rotation loss: the model adds a rotation predictor at the penultimate layer that predicts which of the {0°, 90°, 180°, 270°} rotations was applied to the input; this loss is computed on unlabeled data and trained jointly with the classification loss on labeled data.
- MixMatch: a semi-supervised technique combining guessed labels, regularization and image augmentation, validated on SVHN and CIFAR10; the oracle is trained with WideResNet-28-2, and the attacker obtains the same training set but without labels.
Functionally equivalent extraction (Section 6)
- Setting and assumptions: only for single-hidden-layer ReLU networks, with weights denoted A(0) ∈ R^{d×h}, A(1) ∈ R^{h×K}; it is assumed that the attacker can observe 64-bit floating-point logit outputs, and that the rows of A(0) are linearly independent (the authors state this holds in practice because d is at least 100 larger than h).
- Four-stage algorithm: 1) critical point search—scan along the line u+tv, and use the 2-linearity test (Algorithm 1) to locate, within O(1) queries and with 20+ digits of precision, an input at which exactly one ReLU is at a critical point; 2) weight recovery—at the critical point, measure second-order partial derivatives along each input direction to obtain the difference between adjacent linear regions, and recover the absolute-value proportions of each row's weight vector; 3) sign recovery—using a null-space vector z of A(0) and a direction v_i that activates only a single ReLU, compare O_L(z), O_L(z+v_i), O_L(z−v_i) to determine the sign of each row; 4) final-layer extraction—solve the second layer by least squares.
- Query complexity: Appendix D analyzes the total number of queries as O(dh); the authors state that, without auxiliary information, extraction itself requires Ω(dh) queries, so the algorithm is query-optimal up to a constant factor.
Hybrid strategies (Section 7)
- Gradient matching: optimize ∑i H(O(xi), f(xi)) + α|∇x O(xi) − ∇x f(xi)|22, i.e., match both predictions and input gradients; this requires gradient queries, and the authors leave extending and removing this assumption to future work.
- Learning-based error recovery: fix the first-layer weights Â(0) obtained by direct extraction, adjust the first-layer bias with only one trainable parameter, then solve the remaining weights by gradient descent, to repair the error introduced by the roughly one-in-ten-thousand failure rate of gradient estimation.
What experiments did the paper conduct?
Experiments cover learning-based extraction (WSL/ImageNet, SVHN/CIFAR10), the fidelity ceiling of learning-based extraction, functionally equivalent extraction and the hybrid strategy.
Experimental setup
- Victim models: the WSL model (193M parameters, trained on 1 billion Instagram images, ImageNet top-1 accuracy 84.2%, top-5 97.2%); oracles trained with WideResNet-28-2 on SVHN/CIFAR10 (accuracy 97.36% and 95.75%, respectively); single-hidden-layer fully connected networks with 16 to 512 hidden units on MNIST/CIFAR-10 (about 12,000 to 100,000 parameters, 5 trained per size).
- Datasets: ImageNet (attackers use its training set as unlabeled data, taking 10% ≈ 130,000 points or 100% ≈ 1.3M points), SVHN (73,257 training images), CIFAR10 (50,000 training images), Fashion-MNIST (Section 5), MNIST (Section 6).
- Baselines: models trained directly with ImageNet ground-truth labels; fully supervised extraction (FS).
- Metrics: task accuracy and fidelity (label agreement rate); Table 2 reports top-5 accuracy/top-5 fidelity, Table 3 reports accuracy/fidelity; the functionally equivalent attack reports test-set fidelity and query count.
- Repetitions: MixMatch and FS each run 5 trials, reporting the median over the last 20 checkpoints of each trial; the non-deterministic experiments in Section 5 are repeated 10 times; the functionally equivalent attack extracts each of the 5 networks per size once, and results are averaged.
Main results
Learning-based extraction of the WSL model (per Table 2, values are top-5 accuracy/top-5 fidelity, %):
Swipe horizontally to read the table
Architecture Data proportion ImageNet WSL WSL-5 ImageNet+Rot WSL+Rot WSL-5+Rot ResNet_v2_50 10% 81.86/82.95 82.71/84.18 82.97/84.52 82.27/84.14 82.76/84.73 82.84/84.59 ResNet_v2_200 10% 83.50/84.96 84.81/86.36 85.00/86.67 85.10/86.29 86.17/88.16 86.11/87.54 ResNet_v2_50 100% 92.45/93.93 93.00/94.64 93.12/94.87 N/A N/A N/A ResNet_v2_200 100% 93.70/95.11 94.26/96.24 94.21/95.85 N/A N/A N/A - The authors state that attackers able to query WSL always outperform the baseline using only ImageNet labels, even when only top 5 probabilities are returned.
- The authors state that the rotation loss gives little improvement for ResNet_v2_50 but brings a (1.36/1.80) improvement for ResNet_v2_200, comparable to the improvement from WSL labels at 10% data; in the high-data regime WSL labels bring a (0.56/1.13) improvement.
- The main text additionally reports: at 10% data the attacker improves ResNetv2-50 accuracy by 1.0% and ResNet_v2_200 by 1.9% (the main text does not specify which column metric these numbers correspond to).
MixMatch semi-supervised extraction (Table 3)
- What was measured: accuracy/fidelity of FS and MixMatch at 250/1000/4000 queries on SVHN and CIFAR10.
- Results: at 250 queries MixMatch reaches (95.82/96.38) on SVHN and (87.98/88.79) on CIFAR10, exceeding FS at 4000 queries, which reaches (94.25/94.71) and (86.51/87.37); at 4000 queries the accuracy gap between MixMatch and the oracle is 0.29% on SVHN and 2.46% on CIFAR10.
- Authors' interpretation: the authors state that MixMatch uses unlabeled data to build a prior, making it effective with very few labels; its variance is slightly higher than FS but far smaller than the performance gap.
Fidelity ceiling of learning-based extraction (Table 4)
- What was measured: on Fashion-MNIST, controlling initialization and SGD randomness, measuring the agreement rate across 10 repeated extraction models on the test set, FGSM adversarial examples (ε=25/255) and uniform random inputs.
- Results: test-set agreement rate is between 93.1% and 93.7% (93.7% with all randomness fixed, 93.4% unfixed); it drops to 65.3%–73.6% on adversarial examples and is 59.0%–65.7% on uniform inputs.
- Authors' interpretation: even if the attacker knows the victim model's exact training set, learning-based extraction cannot exceed about 93.4% fidelity on the test set; the authors believe the non-determinism in the training processes of the victim and extracted models compounds, limiting learning-based methods from reaching high fidelity.
Functionally equivalent extraction and hybrid strategy (Tables 6, 7, 8)
- Functionally equivalent attack (Table 6): on MNIST, test-set fidelity is 100% for 12,500/25,000/50,000-parameter models and 99.98% for 100,000 parameters (only 2 of 10,000 test samples disagree); query counts range from 2^17.2 to 2^20.2. CIFAR-10 results are the same as MNIST after controlling for model size: 100% agreement below 200,000 parameters, and over 99% for larger models.
- Hybrid strategy (Table 7): fidelity is 100% for 50,000/100,000 parameters, 99.95% for 200,000, and 99.31% for 400,000 (note that the model sizes in this table are 4 times larger than in Table 6); the main text additionally reports that the previously worst 128-neuron network's fidelity rose from 80% to 99.75%.
- Transferability (Table 8): running 20-step PGD (MNIST ε=0.1, CIFAR-10 ε=0.03) against models extracted by the hybrid strategy, the adversarial example transfer success rate is 100% for all sizes (50,000–400,000 parameters).
- Comparison with prior work: the authors state that the average weight error of the side-channel attack of Batina et al. [25] is 0.0025, while the average error of this paper's first-layer weight matrix is 0.0000009; the authors state that fidelity cannot be compared because [25] does not report that metric.
Other ablations and analyses
- Weight precision: the first-layer weight matrix is reconstructed with an average precision of 23 bits; Figure 6 shows the precision distribution of logit differences for a 16-neuron MNIST model (Appendix C).
- Error source: about 1 in 10,000 probability gradient estimates are wrong, causing a non-negligible error in a single weight that propagates to the bias and the second layer (Section 6.7).
- Prototypicality: Appendix B reports that all metrics find prototypicality correlated with model agreement rate, and that metrics not using a model ensemble (adversarial distance, holdout-retraining) correlate best.
- Oracle accuracy: Appendix C Table 9 lists MNIST oracle accuracy of 94.3%–98.3% and CIFAR-10 of 29.2%–45.9%.
What are some points that can be further explored?
The authors list limitations of the functionally equivalent attack, including obstacles to extension and defensive assumptions; experiments cover two-layer networks and image classification tasks.
Limitations and future directions noted by the authors
- The functionally equivalent attack applies only to two-layer networks; extending it to deeper networks requires solving three problems: producing critical points that belong to a specific layer, recovering weights without direct control over the input, and reducing numerical error (Section 6.7 "Difficulties Extending the Attack").
- The gradient-matching hybrid method requires gradient queries, an additional form of model access; the authors leave extending this technique and removing its model-access assumption to future work (Section 7.1).
- The authors state that the functionally equivalent attack is defeated by defenses that restrict output information (perturbing probabilities, returning only some classes, returning only labels, etc.), and leave defense-aware attacks to future work (Section 8).
- Fidelity extraction considers only label agreement as the similarity function; the authors leave exploring other similarity functions to future work (Section 3.2).
- The attacker may not know the model's input features or class set; the authors state that a preliminary investigation suggests these are not restrictive assumptions, but leave formal treatment to future work (Section 3.4.2).
- Learning-based extraction assumes the attacker can collect public unlabeled data; the authors believe progress in generative models may remove this assumption and leave it to future work (Section 3.4.1); the conclusion also mentions open problems such as reducing the capabilities required for the attack and extending functionally equivalent extraction (Section 9).
Scope of experiments
- The victim models for learning-based extraction are the WSL model (ImageNet, 193M parameters) and WideResNet-28-2 on SVHN/CIFAR10; the extracted model architectures are ResNet-v2-50/200 and WideResNet-28-2.
- The models tested for the functionally equivalent attack are single-hidden-layer fully connected ReLU networks with 16 to 512 hidden units on MNIST and CIFAR-10, 5 trained per size, up to about 100,000 parameters (the hybrid strategy goes up to 400,000 parameters).
- The non-deterministic experiments use fully supervised SGD on Fashion-MNIST, repeated 10 times; MixMatch and FS each have 5 trials.
- Defense evaluation is reflected only through the WSL-5 column in Table 2 (returning only top 5 probabilities); the paper does not empirically test any defense against the functionally equivalent attack.
- The paper does not report the runtime of the functionally equivalent attack; it does not report cost metrics for the learning-based attack beyond the total query budget.
Summarize the main content of the paper
The authors systematize model extraction attacks, respectively improving the query efficiency of learning-based extraction and achieving practical functionally equivalent extraction.
A systematic study of model extraction attacks, in which the authors classify attacks around two goals, accuracy (task performance) and fidelity (prediction agreement), and design one attack for each.
- Problem: remotely deployed models are proprietary assets; the authors argue that prior work conflated two extraction goals and that learning-based methods have inherent limitations and cannot reach functionally equivalent fidelity.
- Methods: the learning-based attack uses the victim model as a labeling oracle and combines semi-supervised techniques such as rotation loss and MixMatch; the functionally equivalent attack exploits the piecewise-linear structure of ReLU networks to directly reconstruct the weights of a two-layer network through four steps—critical point search, weight recovery, sign recovery and least squares—requiring only input-output access; the hybrid strategy uses learning to repair the numerical errors of direct extraction.
- Learning-based results: on the WSL model trained on 1 billion proprietary images, querying WSL always outperforms the ImageNet-label baseline; MixMatch exceeds fully supervised extraction at 4000 queries with only 250 queries on SVHN/CIFAR10, and at 4000 queries the accuracy gaps to the oracle are 0.29% and 2.46%, respectively (Table 3).
- Fidelity ceiling: on Fashion-MNIST, even with all training randomness fixed, learning-based extraction peaks at 93.7% test-set fidelity, and 93.4% when unfixed (Table 4).
- Functionally equivalent results: on MNIST, test-set fidelity is 100% for 12,500–50,000-parameter models and 99.98% for 100,000 parameters (Table 6); the hybrid strategy raises the 128-neuron network's fidelity from 80% to 99.75%, and the adversarial example transfer success rate of the extracted models is 100% (Table 8).
- Authors' conclusion: the authors state that the hybrid method unifies the two attack routes, and note that reducing the capabilities required for the attack and extending functionally equivalent extraction to deeper networks remain open problems.
Related papers
Show 3 more papers
- TranScopeAttacksarXiv
- Don't Forget to Sign the Gradients!DefensesarXiv