ML-Leaks: Membership inference attacks and defenses on machine learning models that do not rely on the model and data

1. 01What problem does this paper attempt to solve?

   Relaxes the strong assumptions of membership inference attacks, demonstrates their low-cost broad applicability, and proposes two defenses.

   Core question: whether the membership inference risk of machine learning models (especially in the MLaaS setting) still holds under weaker assumptions, and how to defend against it.

   • Setting and importance: Under MLaaS, users upload data and the server returns a black-box API. The authors state that if an attacker can determine whether a person's data is in the training set (for example, a model trained on data from patients with a certain disease), they can infer that person's health status, with serious privacy consequences.
   • Shortcomings of existing methods: The authors point out that the first membership inference attack by Shokri et al. relies on two strong assumptions—it requires multiple shadow models with the same structure as the target model, and a dataset with the same distribution as the target training data; its synthetic data scheme can only be used for datasets with binary features, and generating a single data point requires an average of 156 queries.
   • Core observation: The authors argue that membership inference can be done with fewer assumptions and at lower cost; the shadow model need not mimic the target model's behavior, only capture the membership characteristics of data points; overfitting is the key reason the attack works.
   • Paper contributions: Proposes three types of attackers that progressively relax assumptions (ultimately reaching model and data independent), evaluated on 8 datasets; and proposes two defense mechanisms, dropout and model stacking.
   • Threat model:
     • Goal: determine whether the target data point xTarget is in the target model M's training set DTrain (binary classification).
     • Knowledge: only black-box access to the target model (such as an MLaaS API), submitting data points to obtain posterior probability outputs.
     • Capabilities: adversary 1 has same-distribution data and can train 1 shadow model; adversary 2 has no same-distribution data; adversary 3 trains no model and only queries the target model.
     • Victim system: classification models trained by MLaaS (CNN, MLP, random forests, logistic regression, etc.).

   Collapse this answer

2. 02What related research exists?

   Directly builds on the membership inference attack of Shokri et al., and also involves membership inference in other domains, attacks on ML, and privacy-preserving machine learning.

   Membership inference against machine learning models

   • Shokri et al. [38] — proposed the first membership inference attack against ML models, using multiple shadow models (one attack model per class) to imitate the target model's behavior; it is the direct object of comparison and simplification in this paper; this paper notes that its assumptions are strong, that synthetic data generation requires an average of 156 queries, and that it applies only to binary-feature data.
   • Yeom et al. [48] — proposed an attack similar to this paper's adversary 3, but the authors note that it requires the attacker to know the class label of the target data point, which is difficult to obtain in scenarios such as biomedicine.
   • Long et al. [25], Hayes et al. [17], Long et al. [26] — studied membership inference from different angles (combined into one entry).

   Membership inference in other data domains

   • Homer et al. [18] — the first membership inference attack against genomic data; Backes et al. [2] generalized it to other biomedical data.
   • Pyrgelis et al. [35] — performed membership inference on aggregated mobile trajectories, formalizing membership inference as a distinguishability game that can incorporate defenses such as differential privacy for evaluation.

   Other attacks against machine learning

   • Fredrikson et al. [13], [12] — model inversion attacks, which use model outputs to infer missing attributes and can reconstruct identifiable faces.
   • Tramèr et al. [43] — model extraction attacks, which steal model parameters through MLaaS API outputs; the authors use this to argue against regarding hiding the posterior as an effective defense.
   • A series of works on adversarial examples [33][44][8][23][42][34][46] — adding noise to cause model misclassification; other works use adversarial examples to protect social network users' privacy [31][22][49].

   Privacy-preserving machine learning

   • Mohassel and Zhang [28] (SecureML), Bonawitz et al. [5] — privacy-preserving training and secure aggregation protocols based on multi-party computation.
   • Bost et al. [6], Backes et al. [1] — privacy-preserving classification and privacy-preserving random forests based on homomorphic encryption.

   Baselines and benchmarks: The attack baseline is the attack of Shokri et al. (10 shadow models trained according to its original configuration, one attack model per class); evaluation uses 8 datasets, 6 of which are the same as in Shokri et al. (MNIST, CIFAR-10, CIFAR-100, Location, Purchase, Adult), with News and Face added.

   The authors position this paper as: greatly relaxing the attack assumptions of Shokri et al., extending to model and data independent attackers, and providing defenses under this broader attack category.

   Collapse this answer

3. 03How does the paper address this problem?

   Three types of attackers progressively relax assumptions: single shadow model, cross-distribution data transfer, and training-free threshold attack; defenses use dropout and model stacking.

   Overall approach: formulate membership inference as a binary classification problem A : x_Target, M, K → {0, 1}, i.e., given the target data point, the target model, and the attacker's external knowledge K, determine whether the point is in the training set; then progressively reduce the assumptions about the number of shadow models, the target model structure, and the data distribution, and finally use anti-overfitting techniques for defense.

   Adversary 1: single shadow model and structure independence

   • Three-stage process: shadow model training (split DShadow in half, train shadow model S with DShadow^Train) → attack model training (use S to predict all points in DShadow, take each point's largest 3 posteriors (2 for binary classification datasets) as the feature vector, label as member/non-member according to whether it is in DShadow^Train, and train attack model A) → membership inference (query the target model, take the 3 largest posteriors as input to A).
   • Key design: use only 1 shadow model and 1 attack model (Shokri et al. use multiple shadow models and one attack model per class); the authors state that because shadow models are built through MLaaS with pay-per-query, this greatly reduces cost.
   • Structure independence: still effective when the shadow model's parameter count is halved or doubled; further proposes a combining attack—using multiple sub-shadow models of different algorithms (multilayer perceptron, random forests (1,000 trees), logistic regression) to stack features and train the attack model, thereby requiring no knowledge of the target model's classifier type.

   Adversary 2: data transferring attack

   • Design: the shadow model is trained on an existing dataset with a different distribution from the target model's training data; the shadow model no longer mimics the target's behavior and is only used to summarize the membership characteristics of "a data point is in the training set."
   • Handling the number of classes: because the attack model only uses the largest 3 (2 for binary classification) posteriors, the authors consider the effect of differing numbers of dataset classes negligible.
   • Advantages (per the authors): no need to make synthetic-data-generation queries to the target model, lower cost, harder for the MLaaS provider to detect, and applicable to any type of data.

   Adversary 3: training-free threshold attack

   • Design: train no model; query the target model to obtain posteriors, take the maximum value and compare it with a threshold; if it exceeds the threshold, judge it as a member. The rationale is that the model is more confident about points it has been trained on.
   • Threshold selection: generate random points in the target data point feature space (for image datasets, generate pixels according to a uniform distribution; for binary features, generate by unbiased coin flips), query the target model, and use the top t percentile of the maximum posterior of the random points as the threshold; the authors assume random points are approximately non-members. For Adult and News, because the feature boundaries are unclear, this method is not applicable, and the authors leave it for future work.
   • Alternative statistics: also tested standard deviation and entropy (−∑_{pi ∈ Y} p_i log p_i) as features.

   Defenses: dropout and model stacking

   • Dropout: randomly drop edges with a dropout ratio (default 0.5) at both the input layer and hidden layers of the target model to reduce overfitting; only applicable to neural networks (the authors note that other classifiers can use regularization such as L2-norm instead).
   • Model stacking: the target model consists of 3 models in two layers—the first layer has two models (multilayer perceptron or CNN, random forests in the evaluation) that directly receive the raw data, and the second layer has one model (logistic regression) that takes the concatenation Y1||Y2 of the first layer's two models' posteriors as input; the three models are trained on mutually disjoint data subsets (data is split into 12 parts, 6 parts for training/testing the target model, 6 parts for the shadow model).
   • Defense evaluation setup: assume the attacker knows the defense technique and builds the shadow model with the same defense; because adversary 1 and 2 use the same method, the defense is evaluated only against adversary 1 and 3.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On 8 datasets, all three types of attacks are effective, and the single-shadow-model attack is comparable to the original attack; dropout and model stacking can reduce attack performance while largely maintaining utility.

   Experimental Setup

   • Models under test: Image datasets (MNIST, CIFAR-10, CIFAR-100, Face) use a CNN (two convolutional layers, two pooling layers, one 128-unit hidden layer); the remaining datasets use a single-hidden-layer (128-unit) multilayer perceptron; the combining attack and model stacking additionally use random forests (1,000 trees) and logistic regression; the MLaaS evaluation uses the Google Cloud Prediction API (users cannot choose the classifier or architecture).
   • Datasets: 8—MNIST, CIFAR-10, CIFAR-100, Location, Purchase (expanded via K-means clustering into 5 datasets with 2/10/20/50/100 classes), Adult, News (20 Newsgroups, 20,000 documents, 20 classes), Face (LFW, taking 19 people with more than 40 images each, 19 classes, imbalanced).
   • Data split: Each dataset is split in half into DShadow and DTarget, then each is split in half again; half of DTarget trains the target model (members), the other half serves as non-members.
   • Attack model: multilayer perceptron with a 64-unit hidden layer and softmax output; implemented in Python + Lasagne.
   • Baseline: Shokri et al. attack, trained with 10 shadow models and one attack model per class according to its original code configuration.
   • Metrics: precision, recall (membership inference binary classification); accuracy (target model prediction performance); AUC additionally for adversary 3. The paper does not report repetition counts or confidence intervals.

   Main Results

   Figure 1 gives the precision and recall comparison between adversary 1 and Shokri et al. on all datasets (bar chart; representative numbers given in the text):

   Swipe horizontally to read the table

   Attack setting	Dataset	Precision	Recall	Source
   adversary 1 (1 shadow model)	CIFAR-100	0.95	0.95	Text/Figure 1
   Shokri et al. (10 shadow models)	CIFAR-100	0.95	0.94	Text/Figure 1
   adversary 1	Adult	about 0.5	about 0.5	Text
   adversary 1 (Google MLaaS)	Purchase-100	0.90	0.89	Text
   adversary 1 (local)	Purchase-100	0.89	0.86	Text
   adversary 1 (Google MLaaS)	Location	0.89	0.86	Text
   adversary 1 (local)	Location	0.88	0.86	Text

   • The authors state that adversary 1 performs very close to the original attack and is effective on both balanced (CIFAR-10) and imbalanced (Face) datasets.
   • The authors attribute the differences across datasets to overfitting level: the Adult target model's training/test accuracy differs by only 2%, and the attack is about 0.5; CIFAR-100 differs by 78%, and the attack is 0.95 (Figure 2).
   • Figure 12 compares three types of adversaries: the authors state the first two perform very close on most datasets, with adversary 3 only slightly worse (especially precision).

   Data Transfer Attack (adversary 2)

   • What was tested: The shadow model and target model are trained on different datasets; Figure 7 gives the precision/recall matrix for all pairwise dataset combinations.
   • Results: Attacking CIFAR-100 with Face reaches 0.95 precision, 0.95 recall, the same as adversary 1; attacking News with Purchase-10 reaches 0.93/0.92, higher than adversary 1's 0.88/0.86; attacking CIFAR-100 with a News shadow model reaches 0.94 precision, 0.93 recall (introduction). On Google MLaaS, attacking Purchase-100 with a Location shadow model gives 0.8/0.78, and the reverse gives 0.87/0.82.
   • Authors' interpretation: In the t-SNE visualization (Figure 8), for dataset pairs where transfer is effective (CIFAR-100 and News), member/non-member points each form clusters with a shared decision boundary; for combinations where transfer is ineffective (MNIST and Purchase-10), no clear clusters are visible.

   Training-Free Attack (adversary 3)

   • What was tested: Without a threshold, AUC is used to evaluate three statistics—maximum posterior, standard deviation, entropy; then specific inference is evaluated using a threshold selection method (1,000 random points generated per dataset).
   • Results: Maximum posterior exceeds AUC 0.8 on multiple datasets, and the three statistics have almost identical AUC (Figure 9); at t = 10, most datasets (e.g., CIFAR-100) have good precision and recall (Figure 10); a fixed threshold (e.g., 50%) cannot work well on all datasets.
   • Exceptions: Performance is poor on datasets such as Purchase-10 (Figure 11b); Adult and News were not evaluated due to unclear feature boundaries.

   Defense Evaluation

   • Dropout (ratio 0.5): For adversary 1 on Purchase-100, precision drops from 0.89 to 0.64 and recall from 0.86 to 0.63; on CIFAR-100, precision and recall drop by more than 30% (the introduction gives drops to 0.61/0.60); on the News dataset it is almost ineffective. For adversary 3 on CIFAR-100, precision drops by more than 25% and recall by more than 40%, but on MNIST recall instead rises, and the authors state adversary 3 is more resistant to dropout.
   • Utility: The CIFAR-100 target model's accuracy drops only from 0.22 to 0.21; on more than half the datasets dropout instead improves accuracy, e.g., Purchase-50 rises from 0.72 to 0.83 (Figure 15).
   • Model stacking: Attack performance drops in all tested cases, with precision and recall dropping by more than 30% on CIFAR-10; by 28% on News (where dropout is ineffective); on Location, adversary 3's precision drops by more than 20% and recall by more than 30%, with exceptions such as MNIST still present; the impact on target model accuracy is greater than dropout on multiple datasets (e.g., the Purchase series).

   Other Ablations and Analysis

   • Number of shadow models 1–10: no clear difference in precision and recall on Purchase-100/50, Adult, Location (Figure 5).
   • Training epochs 10–100: attack performance rises as epochs increase (Location, Purchase-100, Figure 3).
   • Number of posteriors used as features: little effect on most datasets, with 3 generally best (Figure 4).
   • Halving shadow model parameter count: 0.86/0.83 on Purchase-100; doubling: 0.82/0.80.
   • Combining attack (Table II): 0.94/0.93 when the target is random forests, lower than 1.0/1.0 with a same-architecture shadow model; comparable to the original attack when the target is an MLP or logistic regression.
   • Dropout ratio: at 0.75 the attack drops to 0.53/0.53, but both too-large and too-small ratios make the target model's accuracy low, and the authors consider 0.5 appropriate (Figure 17, Purchase-100).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that threshold selection is not applicable to some datasets and leave it for future work; the defense evaluation and its scope of applicability have clear boundaries.

   Limitations noted by the authors and future directions

   • Adversary 3's threshold selection method does not apply to Adult and News (feature boundaries are unclear); the authors propose collecting similar data from the internet as "random" points, "We leave this for future work" (Section V-B).
   • Dropout only applies to neural network target models, which is the direct motivation for the authors' proposed model stacking (Section VI-B); the authors note that other classifiers can use regularization such as L2-norm instead (Section VI footnote).
   • Model stacking has a greater impact on the target model's accuracy than dropout on multiple datasets (such as the Purchase series) (Section VI-B).
   • Dropout is almost ineffective against attacks on the News dataset; both model stacking and dropout have exceptions such as MNIST for adversary 3 (Section VI-A, VI-B).
   • Because model extraction attacks are effective, the authors do not regard hiding the posterior as an effective defense mechanism (Section VII).

   Experimental coverage

   • The models tested are CNN, multilayer perceptron, random forests, logistic regression and models trained with the Google Cloud Prediction API; there are 8 datasets in total (Purchase is expanded into 5 variants of the number of classes).
   • Defenses are evaluated only against adversary 1 and adversary 3; adversary 2 is not evaluated separately (the authors explain this is because adversary 1 and 2 use the same method); model stacking is not evaluated on the Face dataset (insufficient data).
   • Defense evaluation assumes the attacker knows the defense technique and builds shadow models with the same defense (Section VI).
   • MLaaS measurements cover only the Google Cloud Prediction API and two datasets, Purchase-100 and Location.
   • The paper does not report the number of experimental repetitions, statistical significance tests or confidence intervals; the membership inference metrics are precision/recall/AUC, and no other privacy metrics are used.

   Collapse this answer

6. 06Summarize the main content of the paper

   Membership inference attacks can hold under weak assumptions with no shadow model and no same-distribution data, and dropout and model stacking can mitigate them.

   A paper on membership inference attacks and defenses for machine learning (NDSS 2019), studying the risk of determining whether a data point belongs to the training set in the MLaaS black-box setting.

   • Problem: The authors state that the first membership inference attack by Shokri et al. relies on strong assumptions such as multiple same-structure shadow models and same-distribution data, and underestimates the risk.
   • Method: Proposes three types of attackers—adversary 1 uses only 1 shadow model and 1 attack model (taking the largest 3 posteriors as features), and uses a combining attack to eliminate the need for knowledge of the target model's algorithm type; adversary 2 trains the shadow model on different-distribution data (data transferring attack); adversary 3 does no training and sets a threshold on the target model's maximum posterior for judgment. The defenses are dropout and model stacking.
   • Main results: adversary 1 achieves precision/recall of 0.95 on CIFAR-100, comparable to the original attack with 10 shadow models (0.95/0.94); cross-domain transfer attacks are effective, e.g., attacking CIFAR-100 with a News shadow model reaches 0.94/0.93; adversary 3 achieves AUC above 0.8 on multiple datasets; on Google MLaaS, Purchase-100 is attacked at 0.90/0.89.
   • Defense effects: dropout (0.5) reduces the attack on CIFAR-100 to 0.61/0.60, while the target model's accuracy only drops from 0.22 to 0.21, and accuracy on some datasets instead improves; model stacking reduces the attack's precision and recall on CIFAR-100 by more than 30%, and is applicable to non-neural-network models.
   • Authors' conclusion: membership inference is simpler and more generally applicable than previously thought, and the risk is more serious; overfitting is the key to the attack's effectiveness, and anti-overfitting techniques can mitigate this risk while maintaining model utility.

   Collapse this answer

Related papers