Membership inference attacks against machine learning models

1. 01What problem does this paper attempt to solve?

   Studies how to determine, under black-box queries, whether a record is in a model's training set (membership inference), and quantifies the degree of leakage.

   Core problem: given a data record and black-box query access to a model, determine whether that record belongs to the model's training dataset (membership inference).

   • Setting and importance: Google, Amazon, and others provide machine learning as a service, where users upload data to train a model and then query it via an API; the authors argue that if the training data is sensitive (e.g., hospital records), membership leakage can directly lead to privacy leakage, for example, a patient's record appearing in the training set of a disease-related model.
   • Shortcomings of existing methods: the authors state that model inversion cannot produce specific members of the training set, nor determine whether a given record is in the training set; membership detection based on explicit statistics (e.g., genomic mixture pools) targets public statistics rather than black-box model APIs.
   • Core observation: models often behave differently on training data and unseen data; overfitting is a common cause but not the only one; the authors turn this difference into a learnable signal.
   • Contributions: the authors propose training an attack model with shadow models, turning membership inference into a binary classification problem; they conduct empirical work on Google Prediction API, Amazon ML, and a local neural network, quantify the relationship between leakage and overfitting, and evaluate mitigation strategies.
   • Threat model:
     • Goal: determine whether a given record is in the target model's training set; the attack succeeds if the determination is correct.
     • Knowledge: knows the input and output format and value ranges; and either (1) knows the model type, architecture, and training algorithm, or (2) has only black-box access to the machine learning as a service platform that trained the model, without prior knowledge of the model architecture and hyperparameters.
     • Capability: query the model with arbitrary records and obtain a prediction vector of per-class probabilities; may have population background knowledge (independent samples or marginal distribution).
     • Victim systems: models trained on Google Prediction API, models trained on Amazon ML, and locally trained neural networks.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: statistical/model attacks, model inversion, model extraction, and privacy-preserving training and differential privacy.

   Attacks on statistical and machine learning models

   • Ateniese et al. [2] — use SVM and HMM parameters to infer population statistics of the training set (e.g., whether it contains records of a certain ethnicity); the paper notes that this paper infers the membership of specific records under a black-box setting, rather than population statistics.
   • Homer et al. [21] and follow-ups [3], [15] — compare public statistics with the population distribution to infer whether a specific genome is in the dataset; the paper notes that this paper targets trained models rather than explicit statistics.
   • Calandrino et al. [7] — infer inputs from temporal changes in the outputs of collaborative filtering recommender systems; the paper notes that they exploit temporal behavior specific to recommender systems.

   Model inversion

   • Fredrikson et al. [16], [17] — infer input features from the model's outputs on hidden inputs; the paper notes that model inversion produces average features characterizing an entire class, neither constructing specific members of the training set nor determining whether a given record participated in training, and uses a model inversion experiment on CIFAR-10 (Figure 13) to show that the reconstructed images do not correspond to any specific training image.

   Model extraction

   • Tramèr et al. [32] — extract model parameters through a prediction API; the paper notes that the leakage quantification is limited to models such as KLR that store training data (import points) directly in their parameters, and is demonstrated only by visual similarity, without extending to other model types.

   Privacy-preserving machine learning and differential privacy

   • Secure multi-party computation and encrypted computation for training (decision trees [26], linear regression [11], Naive Bayes [33], k-means [22], computation on encrypted data [4], [6], [35]) — the paper notes that these works limit leakage during training, and the resulting models are as susceptible to the inference attack in this paper as conventional models.
   • Differential privacy [12] and its applications to regression, SVM, deep learning, etc. — the paper notes that, by definition, differentially private models can limit the success probability of membership inference based solely on model outputs (including the attack in this paper), but a small ε may noticeably reduce prediction accuracy.

   Baselines and benchmarks: the baseline for the attack evaluation is random guessing (equal probability of member/non-member, baseline accuracy 0.5); the experimental datasets are CIFAR-10/100, Purchase, Locations, Texas hospital stays, MNIST, and UCI Adult.

   The authors position this paper as: quantifying training set membership leakage under a black-box setting with unknown model architecture, and state (per the authors) that this is the first membership inference attack against machine learning models.

   Collapse this answer

3. 03How does the paper address this problem?

   Uses shadow models to simulate the target model's behavior, turning membership inference into a binary classification problem over prediction vectors.

   Overall approach: train several shadow models that behave similarly to the target model; since the ground-truth membership of their training sets is known, supervised learning can be used to train an attack model to distinguish differences in model outputs between members and non-members.

   Problem formalization

   • The target model ftarget outputs a ctarget-dimensional probability vector; the attack model fattack takes as input a record with its true label (x, y) and the prediction vector y, and outputs an in/out binary classification, i.e., it estimates Pr{(x, y) ∈ D_target^train}, the probability that x is in the target model's training set.
   • One attack model is trained per target class: the authors state that the prediction vector distribution depends on the input's true class, and splitting by class can improve attack accuracy.

   Shadow models

   • The attacker trains k shadow models; their training sets have the same format and a similar distribution as the target model's training set and are disjoint from it (the authors state this is the worst case, and overlap would make the attack better).
   • Shadow models must be trained in the same way as the target model: when the algorithm and architecture are known, reproduce them directly; for ML-as-a-service, train on the same platform (e.g., Google Prediction API) (Figure 2).
   • More shadow models make the attack more accurate but cost more.

   Three methods for generating shadow training data

   • Model-based synthesis: when no real data or statistical knowledge is available, synthesize using the target model itself: randomly initialize records, use hill climbing to search for inputs that the target model classifies as class c with high confidence, then sample by probability (Algorithm 1). In the purchase dataset experiments, the parameters were kmax=128, rejmax=10, kmin=4, confmin=0.2.
   • Statistics-based synthesis: when the population marginal distribution is known, synthesize records by sampling each feature independently.
   • Noisy real data: when similar data is available, randomly flip 10% or 20% of binary features to simulate a noisy version.

   Attack model training

   • Query each shadow model with its training set (outputs labeled in) and a disjoint test set of the same size (outputs labeled out) to obtain prediction vectors, forming the attack training set (Figure 3).
   • Split the attack training set by class into ctarget parts, and train one binary classifier per class; the authors emphasize that the attack model learns not to identify high-confidence inputs, but to distinguish training from non-training inputs that are both high-confidence.
   • The attack model can be trained with any framework: the experiments used a local neural network and also the same Google Prediction API being attacked (in which case there was no control over architecture, parameters, or hyperparameters).

   Success criterion

   • The metrics are precision (the proportion of those judged as members that are true members) and recall (the proportion of the training set correctly identified); at evaluation time members and non-members are equal in number, and the baseline accuracy is 0.5.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluated the attack on 7 datasets and 3 types of platforms, and ran experiments on shadow data, number of classes, overfitting, and mitigation strategies.

   Experimental Setup

   • Models tested: black-box models trained on Google Prediction API and Amazon ML (two configurations: (10,1e-6) default and (100,1e-4)), and local Torch7 neural networks (CNN for CIFAR, a one-layer 128-hidden-unit fully connected network for the purchase dataset).
   • Datasets: CIFAR-10/100, Purchase (197,324 records, 600 binary features, {2,10,20,50,100} classes), Locations (5,010 users, 446 features, 30 classes), Texas hospital stays (67,330 records, 6,170 features, 100 classes), MNIST, UCI Adult.
   • Training scale: the target model training sets are mostly 10,000 records (1,600 for Locations; 2,500–15,000 for CIFAR-10, 4,600–29,540 for CIFAR-100).
   • Number of shadow models: 100 for CIFAR, 20 for Purchase, 10 for Texas, 60 for Locations, 50 for MNIST, 20 for Adult.
   • Attack model: a one-layer 64-hidden-unit fully connected network (ReLU) or a Google-trained black-box model.
   • Metrics and baseline: precision, recall, mostly reported per class; equal numbers of members/non-members, baseline 0.5; the paper does not report the number of repetitions or confidence intervals.

   Main Results

   According to Table II (Google-trained models, attack precision is the overall value across all classes):

   Swipe horizontally to read the table

   Dataset	Training accuracy	Test accuracy	Attack precision
   Adult	0.848	0.842	0.503
   MNIST	0.984	0.928	0.517
   Location	1.000	0.673	0.678
   Purchase (2)	0.999	0.984	0.505
   Purchase (10)	0.999	0.866	0.550
   Purchase (20)	1.000	0.781	0.590
   Purchase (50)	1.000	0.693	0.860
   Purchase (100)	0.999	0.659	0.935
   TX hospital stays	0.668	0.517	0.657

   • The authors state that more classes mean more leakage: with many classes the model needs to memorize more training data; on Adult (binary classification, almost no overfitting) the attack is equivalent to random guessing.
   • Cross-platform comparison (Figures 5, 7, 100-class purchase dataset): the 50/75/90 percentiles of precision are 0.94/0.97/1 for the Google model, 0.84/0.88/0.91 for Amazon (100,1e-4), and 0.74/0.79/0.84 for Amazon (10,1e-6); the authors state that the Google-trained model leaks the most.
   • CIFAR experiments (Figure 4): median precision for CIFAR-10 is 0.78–0.71 (decreasing as the training set grows), and 1/1/0.98/0.97 for CIFAR-100, with recall close to 1; the authors state that CIFAR-100 is especially vulnerable.

   Effect of Shadow Training Data

   • Noisy real data (Figure 8, Locations, Google model): attack precision under real data, 10% noise, and 20% noise is 0.678, 0.666, and 0.613 respectively, with recall of 0.98, 0.99, and 1.00; the authors state that the attack is robust to inaccurate distribution assumptions.
   • Synthetic data (Figure 9, Purchase, Google model): attack precision for real data, marginal-distribution synthesis, and model synthesis is 0.935, 0.795, and 0.896, with recall of 0.994, 0.991, and 0.526; the model synthesis method has precision below 0.1 on a few classes, which the authors explain as these classes accounting for less than 0.6% of the training set, some with fewer than 30 records, making them hard to synthesize.
   • Model synthesis cost: the average confidence of synthesized records is 0.24, each record requires an average of 156 queries, 30,000 records were synthesized in total, and 8 shadow models were trained.

   Effect of Overfitting and Number of Classes

   • Overfitting: the authors state that more severe overfitting means more leakage, but only for comparisons among models of the same kind—Amazon (100,1e-4) overfits more and leaks more than (10,1e-6), but both leak less than the less-overfit Google model (Tables I, II); the authors consider model structure and type to also be factors.
   • Per-class relationship (Figure 11): classes with a larger (train-test) accuracy gap have higher attack precision; classes with a larger share of training data have lower attack precision.
   • Output distribution differences (Figure 12): in models where the attack succeeds, there are observable differences in the prediction accuracy and uncertainty distributions between members and non-members.

   Evaluation of Mitigation Strategies

   According to Table III (local neural networks, 100 classes; the attack remains black-box):

   Swipe horizontally to read the table

   Mitigation	Test accuracy	Attack precision (Purchase)	Attack precision (Hospital)
   No mitigation	0.66 / 0.55	0.87	0.77
   Top k=1 label	0.66 / 0.55	0.60	0.67
   Rounding d=1	0.66 / 0.55	0.83	0.75
   Temperature t=20	0.66 / 0.55	0.83	0.76
   L2 λ=1e-2 / 5e-3	0.63 / 0.35	0.54	0.52

   • The authors state that the attack is generally robust to these mitigations: top-k, rounding, and temperature scaling cannot prevent the attack; even when only the label of the most likely class is output, the attack can still exploit the behavior that members and non-members are misclassified into different wrong classes.
   • L2 regularization can both reduce leakage and possibly improve test accuracy, but too large a λ (1e-2 on Purchase, 5e-3 on Hospital) noticeably harms test accuracy.

   Other Ablations and Analyses

   • Training set size: as the CIFAR-10 training set increases from 2,500 to 15,000, median precision drops from 0.78 to 0.71 (Figure 4).
   • With the same number of training records per class (200), the attack on CIFAR-100 is close to 1, better than on CIFAR-10 (main text VI-F).
   • Texas dataset (Figure 6): more than half of the classes have precision above 0.7, and more than 20 classes above 0.85.
   • Locations (main text VI-D): precision 0.6–0.8, recall almost constantly 1.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out the applicable boundaries of the synthetic data method and the difficulty of regularized deployment; the experiments cover 7 datasets and 3 types of platforms.

   Limitations noted by the authors and future directions

   • The model-based synthesis data method relies on efficient search of the input space; the authors state it may not be applicable to complex classification tasks such as high-resolution images (Section V-C).
   • For classes with very few samples in the training set, the model-based synthesis attack has low precision (below 0.1); the authors state it is difficult to synthesize representative samples for such classes (Section VI-E).
   • Overfitting is not the only cause of leakage; model architecture and type also affect leakage, and different models with the same degree of overfitting leak different amounts (Section VI-G, VII).
   • Regularization needs to be deployed carefully; too large a λ harms the model's performance on the test set (Section VIII-B).
   • Differential privacy can by definition defend against the attack in this paper, but a small ε may noticeably reduce the model's prediction accuracy (Section VIII).
   • The authors suggest that ML service platforms should, when selecting models, factor in leakage risk in addition to accuracy, warn customers, and use the attack in this paper as a metric for quantifying leakage (Section VIII, X).

   Scope of experiments

   • The platforms tested were Google Prediction API, Amazon ML (two configurations), and a local Torch7 neural network, three types in total; the datasets were CIFAR-10/100, Purchase, Locations, Texas hospital stays, MNIST, and Adult, 7 in total.
   • All tasks were supervised classification; the target models' training sets ranged from 1,600 to 29,540 records.
   • The mitigation evaluation was conducted on the local neural network and covered four types: top-k truncation, probability rounding, softmax temperature, and L2 regularization (Table III).
   • The paper does not report the number of experimental repetitions, confidence intervals, or statistical significance tests.
   • The paper does not report the total query budget of the attack; it only gives an average of 156 queries per record for model-based synthesis (Section VI-E).

   Collapse this answer

6. 06Summarize the main content of the paper

   Proposes a shadow-model membership inference attack, with precision up to 0.935 under black-box access on a model trained on Google, and common mitigation measures have limited effect.

   An attack paper on the training data privacy of machine learning models, studying black-box membership inference: using only the prediction vector returned by a query, determine whether a record is in the training set.

   • Problem: ML-as-a-service platforms (Google, Amazon) provide models via black-box APIs; the authors consider training data membership (e.g., hospital records) itself to be sensitive information.
   • Method: train multiple shadow models to simulate the target model's behavior, and use their member/non-member outputs to train an attack model (one binary classifier per class); shadow data can come from real data, marginal-distribution synthesis, hill-climbing synthesis with the target model, or noisy real data.
   • Main results: on a 100-class purchase model trained on Google, attack precision 0.935 and recall 0.994 (Table II, Figure 9); on the two Amazon configurations, median precision 0.74 and 0.84 (Figure 5); on Texas hospital stays data, 0.657 (Table II); with no real data, model-based synthesis still reaches 0.896 (Figure 9).
   • Negative results: on Adult and MNIST, precision is only 0.503 and 0.517, close to random guessing; the authors attribute this to a low degree of overfitting and few classes.
   • Mechanism: the authors state that leakage increases with the number of classes and the degree of overfitting, but overfitting is not the only factor, and model architecture also has an effect.
   • Mitigation: top-k, rounding, and temperature all fail to stop the attack; L2 regularization can reduce attack precision to 0.54 (Purchase, λ=1e-2) but harms test accuracy; the authors suggest that platforms use the attack in this paper as a leakage metric and warn customers of the risk.

   Collapse this answer

Related papers