Membership-private machine learning based on adversarial regularization
Machine Learning with Membership Privacy using Adversarial Regularization
The authors propose min-max adversarial training to defend against black-box membership inference attacks, reducing attack accuracy on Purchase100 from 67.6% to 51.6%, with a 3.6% drop in test accuracy.
Attacker goal: determine whether a target data record is in the model's training set D (membership inference / tracing attack). Knowledge: black-box, can only observe model outputs (prediction vectors), cannot see parameters; background knowledge includes a known member set D_A (a subset of the training set) and a known non-member set D'_A. Capability: train an inference model h that takes (x, y, f(x)) as input and outputs a membership probability. Victim system: a classification model exposed through a prediction API, such as MLaaS.
- Machine learning models leak training set membership information: a black-box attacker can infer whether a record is in the training set solely from the outputs of a prediction API (membership inference attack). Existing simple mitigation methods cannot guarantee privacy, and differential privacy mechanisms are costly for large models at small ϵ.
- The authors model the defense as a min-max privacy game: the inner layer trains the strongest membership inference model h to maximize inference gain, and the outer layer trains the classifier f to minimize the classification loss plus λ times the attack gain (as an adversarial regularization term), solved by GAN-like alternating stochastic gradient descent, making the model's prediction distributions on the training set and on non-members indistinguishable.
- On the four model/datasets CIFAR100 (Alexnet, DenseNet), Purchase100, and Texas100, attack accuracy drops to 50.7%–51.6% (close to random guessing), and test accuracy decreases by 1.1%–4.4% relative to the undefended model (Table 4); at the same privacy level, L2 regularization's test accuracy is only 32.1% (Table 6).
- The paper does not specifically discuss limitations. Experimental coverage: 4 model/dataset combinations, all 100-class classification tasks; attack evaluation targets the strong attacker with knowledge of part of the training set as set up in the paper; the paper does not report multiple repeated experiments or variance.
- Evaluated models
- AlexnetDenseNet4-layer fully connected neural network (Purchase100)5-layer fully connected neural network (Texas100)
- Benchmarks
- CIFAR100Purchase100Texas100
- Metrics
- attack accuracy (membership inference accuracy)training accuracytesting accuracygeneralization error
Milad Nasr et al. propose an adversarially regularized training method that makes the model's predictions for training set members hard to distinguish from its predictions for other data points from the same distribution, in order to defend against black-box membership inference attacks. The method formulates training as a min-max optimization problem that simultaneously minimizes the classification loss and the maximum gain of a membership inference attack. The authors evaluate on deep neural networks and multiple benchmark datasets and state that this strategy can reduce membership inference risk to near random guessing at little cost in classification error.
Paper analysis
What problem does this paper attempt to solve?
How to train a model that provably protects the privacy of training set members under black-box membership inference attacks while minimizing the loss in classification accuracy.
Core problem: A machine learning model's predictions leak information about its training set members; how to train a model that is provably robust to black-box membership inference / tracing attacks while minimizing the loss in classification accuracy.
- Scenario and importance: MLaaS platforms (Google, Amazon, Microsoft, etc.) let users train models and expose them to others through prediction APIs; training data often contains sensitive information (user profiles, location traces, medical records, financial data), and even if the training platform is trusted, the model's prediction computation itself may jeopardize training data privacy (according to the authors).
- Shortcomings of existing methods: The authors state that existing defenses fall into two categories—simple mitigation techniques (restricting top-k outputs, reducing prediction accuracy, L2 regularization) cannot guarantee any strict privacy notion; differential privacy mechanisms, although they can guarantee ϵ-level (membership) privacy, may cause significant classification accuracy loss for large models on high-dimensional data at small ϵ, because their design goal is to guarantee input indistinguishability for all possible input datasets and all model parameters, without explicitly incorporating utility into the design objective.
- Core observation/assumption: The authors propose a more targeted privacy notion, membership privacy: no attacker can distinguish the model's predictions on the training set from its predictions on other data from the same distribution (up to a privacy parameter); the defense should anticipate the strongest attack against itself, rather than being designed against a fixed attack.
- Method proposed in the paper: The defense is formalized as a min-max privacy game between the defense mechanism and an inference attack, and an adversarial training algorithm is designed that treats the attack gain as a regularization term for the classifier, making the strongest attack degenerate to random guessing under minimal classification loss.
- Threat model:
- Attacker's goal: Determine whether a target data record belongs to the target model's training set D.
- Knowledge: black-box—can only observe model outputs (prediction vectors), cannot see parameters; background knowledge is a known member set D_A (a subset of the training set) and a known non-member set D'_A (samples from the same distribution that are not in D).
- Capability: Train an inference model h that takes (x, y, f(x)) as input and outputs the probability that (x, y) belongs to D; the attacker adapts the attack to the target model to maximize gain.
- Victim system: A classification model exposed through a prediction API (MLaaS scenario).
What related research exists?
Related work covers four groups: membership inference attacks, differential privacy defenses, cryptographic/trusted hardware approaches, and game-theoretic privacy frameworks.
Membership inference and reconstruction attacks
- Homer et al. (2008), Sankararaman et al. (2009), Dwork et al. (2015, 2017), Backes et al. (2016), Pyrgelis et al. (2017), etc.—membership/tracing attacks against simple statistical linear functions: comparing how close a published statistic is to a random sample or a target record, which can be formalized as hypothesis testing.
- Shokri et al. (2017)—proposed a black-box membership inference attack against machine learning models, training an inference model to distinguish predictions of members from non-members, which can be trained with shadow models; this paper's Purchase100 and Texas100 datasets and attack setting come from that work.
- Dinur and Nissim (2003), Wang et al. (2009)—reconstruction attacks: inferring sensitive attributes of multiple individuals in the training set; the authors state that membership inference and reconstruction attacks are the two major categories of attacks under the setting where "the attacker observes computation results."
Differential privacy defenses
- Dwork et al. (2006, 2014)—foundations of differential privacy; Abadi et al. (2016), Bassily et al. (2014), Chaudhuri et al. (2011), Papernot et al. (2016, 2018)—applying differential privacy to machine learning training.
- The authors point out that differential privacy mechanisms are difficult to achieve with negligible utility loss, because indistinguishability must be guaranteed for similar states of all possible input datasets, and the function sensitivity (which determines noise magnitude) is difficult to bound tightly. Li et al. (2013) and Yeom et al. (2018) analyzed the relationship between membership privacy and differential privacy definitions.
Cryptographic and trusted hardware approaches
- Homomorphic encryption, garbled circuits, secure multi-party computation (Lindell and Pinkas 2000; Gilad-Bachrach et al. 2016; Mohassel and Zhang 2017; Bonawitz et al. 2017) and trusted hardware such as Intel SGX (Ohrimenko et al. 2016; Hunt et al. 2018)—enabling blind training and blind inference; the authors point out that these methods prevent attackers from directly seeing sensitive data, but do not limit information leaked through the computation itself.
Game-theoretic privacy frameworks and adversarial training
- Shokri et al. (2012), Hsu et al. (2013), Manshaei et al. (2013), Shokri (2015), Alvim et al. (2017)—formalizing and optimizing privacy with game theory: minimizing privacy loss against the strongest corresponding attack, with solutions provably robust to any attack that threatens that "loss" function, and utility can be explicitly incorporated into min-max optimization.
- Goodfellow et al. (2014), Arjovsky et al. (2017), Daskalakis et al. (2017)—GANs provide algorithms for solving min-max games when training complex models; Miyato et al. (2015, 2017), Odena (2016), Dai et al. (2017), Koziński et al. (2017), etc. use adversarial training for regularization/generalization.
Baselines and benchmarks: The comparison baseline is L2-norm regularization (Table 6, Purchase100); the benchmark datasets are CIFAR100, Purchase100, and Texas100 (the latter two from Shokri et al. 2017).
The authors position this paper as: within a game-theoretic privacy framework, using GAN-style adversarial training to solve the min-max optimization of "minimal classification loss + strongest attack gain," explicitly incorporating utility into the privacy mechanism's objective, targeting a more targeted prediction indistinguishability guarantee than differential privacy.
How does the paper address this problem?
Modeling membership privacy as a min-max game: the classifier minimizes "classification loss + λ×strongest attack gain," solved by GAN-style alternating gradient training.
Overall approach: add the membership inference attack's gain as an adversarial regularization term to the classifier's training objective, forming a min-max privacy game, trained with GAN-like alternating stochastic gradient descent, making the model's prediction distributions on the training set and on non-members indistinguishable.
Problem setting and formalization
- Classifier: f : X → Y, outputting a k-class probability vector; the empirical objective is min_f L_D(f) + λ R(f) (Equation 3), where L_D is the empirical loss on training set D and R is the regularization term.
- Inference model: h : X × Y² → [0,1], taking a data point (x, y) and the model prediction f(x) as input and outputting the probability that (x, y) belongs to training set D.
- Attack gain: Gf(h) = 1/2EPrD[log h(x,y,f(x))] + 1/2EPr∖ D[log(1−h(x,y,f(x)))], i.e., the attack's average log-correctness on members and non-members; the attacker's goal is to maximize the empirical gain (computed on the known member set D_A and known non-member set D'_A).
Min-max privacy game
- Joint objective: minf {LD(f) + λ maxh Gf,D,D′(h)} (Equation 7)—the inner layer finds the strongest attack h for a given f, and the outer layer finds the optimal defense classifier f for a given h; λ controls the trade-off between classification accuracy and membership privacy, and the attack term corresponds to the regularization term R(f) in Equation 3.
- Theoretical analysis: Drawing on the analysis of GANs (Goodfellow et al. 2014): given f, the optimal attack h*_f compares whether the prediction comes from the training set output distribution p_f or the non-member output distribution p_f′; according to Theorem 1 of that work, the global optimum is reached if and only if p_f* = p_f′*, at which point the optimal attack converges to 0.5 (random guessing); according to its Proposition 2, the authors state that the stochastic gradient descent of Algorithm 1 eventually converges to the game equilibrium.
Adversarial training algorithm (Algorithm 1)
- Alternating optimization: In each epoch, first perform k inner maximization steps—sample a mini-batch of m samples each from training set D and reference set D′, and update the inference model h's parameters ω along the gradient ascent direction; then perform 1 outer minimization step—sample a new mini-batch from D and update the classifier f's parameters θ along the gradient descent direction, with objective l(f(x),y) + λ log(h(x,y,f(x))).
- Two datasets: Training requires training set D and a reference set D′ that is disjoint from it and from the same distribution; the inference gain is computed on both, and the classification loss is computed only on D.
Inference attack model architecture (Figure 3)
- Single attack model: Unlike Shokri et al. (2017), which trains one attack model per class (k in total), this paper uses only one attack model; three fully connected subnetworks process the prediction vector f(x) (layer sizes [100,1024,512,64]) and the one-hot label ([100,512,64]) respectively, and then output the membership probability in the concatenated common network ([256,64,1]).
- Training details: ReLU activation, weights initialized from a normal distribution with mean 0 and standard deviation 0.01, Adam optimizer, learning rate 0.001; each training batch has the same number of member and non-member samples, preventing the attack model from favoring one side.
Success criterion and evaluation method
- Attack accuracy: On D\D_A and another set D″ from the same distribution that overlaps with neither D nor D'_A, compute the average probability that the attack correctly judges member/non-member; at maximal membership privacy, attack accuracy is 50% (random guessing).
- Utility metrics: Training accuracy, test accuracy, generalization error (the difference between test and training accuracy, computed per class as defined by Hardt et al. 2015); and the distribution gaps of prediction accuracy and prediction uncertainty (normalized entropy) between members/non-members are used to show the degree of information leakage.
What experiments did the paper conduct?
On 4 model/datasets, attack accuracy drops to 50.7%–51.6% and test accuracy drops by 1.1%–4.4%, outperforming L2 regularization at the same privacy level.
Experimental setup
- Models tested: On CIFAR100, Alexnet (Adam, learning rate 0.0001, 100 epochs, Dropout 0.2) and DenseNet (SGD, 300 epochs, learning rate stepped down from 0.001 to 0.00001, L2 regularization 5e-4); on Purchase100, a 4-layer fully connected network ([1024,512,256,100], Tanh, 50 epochs); on Texas100, a 5-layer fully connected network ([2048,1024,512,256,100], Tanh, 50 epochs).
- Datasets: CIFAR100 (60,000 32×32 color images, 100 classes); Purchase100 (197,324 records, each with 600 binary features, 100 classes); Texas100 (67,330 discharge records, 6,170 binary features, 100 classes).
- Experimental scale (Table 3): Purchase100 uses |D|=20,000, |D′|=20,000, λ=3, |D_A|=5,000, |D′_A|=20,000; Texas100 uses |D|=10,000, |D′|=5,000, λ=2, |D_A|=5,000, |D′_A|=10,000; CIFAR100 uses |D|=50,000, |D′|=5,000, λ=6, |D_A|=25,000, |D′_A|=5,000.
- Attacker setting: When evaluating the defense, a strong attacker is assumed—one who knows a substantial portion of the training set (D_A) and attempts to infer the membership of the rest; the attack model architecture is shown in Figure 3.
- Baselines: A conventionally trained model without defense; L2-norm regularization (Table 6).
- Metrics: Training/test accuracy, attack accuracy (membership inference accuracy, 50% is random guessing), generalization error; implementation based on Pytorch. The paper does not report the number of repetitions or variance.
Main results
According to Table 4 ("With defense" is this paper's min-max mechanism):
Dataset/Model No defense: train/test/attack accuracy With defense: train/test/attack accuracy Purchase100 100% / 80.1% / 67.6% 92.2% / 76.5% / 51.6% Texas100 81.6% / 51.9% / 63% 55% / 47.5% / 51.0% CIFAR100-Alexnet 99% / 44.7% / 53.2% 66.3% / 43.6% / 50.7% CIFAR100-DenseNET 100% / 70.6% / 54.5% 80.3% / 67.6% / 51.0% - The authors state that the experimental results are consistent with the theory: with defense, attack accuracy is close to random guessing, while test accuracy decreases by only 3.5% (Purchase100), 4.4% (Texas100), 1.1% (CIFAR100-Alexnet), and 3% (CIFAR100-Densenet).
- The authors state that the mechanism is also a strong regularizer: generalization error shrinks to at most 1/4 of the original, for example Texas100 from 29.7% to 7.5%, CIFAR100-Alexnet from 54.3% to 22.7%, CIFAR100-Densenet from 29.4% to 12.7%, and Purchase100 is almost unchanged.
- Figure 4 (Purchase100) shows that during training the classification loss and attack gain converge to an equilibrium point; the authors state that the undefended model's loss continues to decrease and may overfit.
Comparison with L2 regularization
- What was tested: On Purchase100, L2-norm regularization is used as a mitigation measure, comparing utility at the same privacy level (Table 6).
- Results: With L2 factor 0.01, attack accuracy is 50.6% and test accuracy is only 32.1%; with factor 0.001, attack accuracy is 60% and test accuracy is 81.3%. The authors state that at near-maximal membership privacy, this paper's mechanism achieves more than twice the test accuracy of the L2-regularized model (76.5% vs. 32.1%).
- Authors' interpretation: Such regularizers do not guarantee privacy, nor do they minimize the cost of achieving privacy, which is consistent with the objective of this paper's min-max optimization.
Indistinguishability of prediction distributions
- Figure 6: On class 50 training samples of Purchase100, the undefended model's predicted probability for the correct class is close to 1; the defended model spreads the predicted probability across multiple classes, and its output distributions on training samples (middle panel) and test samples (bottom panel) are indistinguishable. The authors state that this additional uncertainty is precisely the reason for the provable mitigation of information leakage.
- Figure 7 (prediction accuracy distribution): Comparing defended and undefended, the maximum gap between member/non-member curves is Purchase100 (0.02 vs. 0.34), Texas100 (0.05 vs. 0.25), CIFAR100-Densenet (0.06 vs. 0.56); the average gaps are (0.007 vs. 0.013), (0.004 vs. 0.016), (0.005 vs. 0.021).
- Figure 8 (prediction uncertainty, normalized entropy): The maximum gaps are Purchase100 (0.03 vs. 0.30), Texas100 (0.02 vs. 0.15), CIFAR100-Densenet (0.04 vs. 0.49); the average gaps are (0.004 vs. 0.012), (0.002 vs. 0.04), (0.002 vs. 0.01). The authors state that these figures do not prove privacy; they only show that the statistical gaps an attacker can exploit are reduced by one to two orders of magnitude.
Other ablations and analyses
- Effect of λ (Table 5, Purchase100): With λ=0, attack accuracy is 67.6% and test 80.1%; with λ=1, 57.0%/78.3%; with λ=2, 55.0%/77.4%; with λ=3, 51.8%/76.5%; with λ=10, 50.6%/70.1%.
- Effect of reference set size (Table 7, Purchase100, |D|=20,000, λ=3): With |D′|=1,000, attack accuracy is 59.2% and test 80.0%; with 5,000, 52.8%/77.4%; with 10,000, 52.4%/76.8%; with 20,000, 51.6%/76.5%; with 30,000, 50.6%/76.4%. The authors state that the larger the reference set, the better it represents the underlying distribution, and attack accuracy converges to 50%.
- Generalization error CDF (Figure 5): On the three datasets, the cumulative distribution curve of generalization error for the defended model lies overall to the left of the undefended model (covering more classes at the same error); the authors state that the privacy mechanism yields lower generalization error across all classes.
- Exception: Purchase100's generalization error is almost unchanged with defense (explicitly stated in the main text), unlike the other three models/datasets.
What are some points that can be further explored?
The paper does not specifically discuss limitations; the experiments are limited to 4 100-class classification models/datasets and a strong attacker setting with knowledge of part of the training set.
Limitations and future directions noted by the authors
- The paper does not specifically discuss limitations (there is no Limitations section, and no list of limitations or future work items in the Conclusion or Discussion).
- In the Conclusion, the authors limit the conclusion to guaranteeing minimal accuracy loss "given the available training/reference data and model capacity," which can be regarded as a statement of the conditions under which the results apply.
Scope of experimental coverage
- Models tested: 4 model/dataset combinations in total—Alexnet and DenseNet on CIFAR100, a 4-layer fully connected network on Purchase100, and a 5-layer fully connected network on Texas100, all 100-class classification tasks (Section 5.1–5.2).
- Attacker setting: Evaluation targets a strong attacker who knows part of the training set members (D_A up to 25,000 records) and part of the non-members; the attack model is the single neural network in Figure 3 of the paper; the paper does not evaluate the defense using Shokri et al. (2017)'s one attack model per class (k in total).
- Baselines: Compared only with the undefended model and L2-norm regularization (Purchase100, Table 6); the paper discusses differential privacy defenses and explains their cost problem, but does not conduct an experimental comparison.
- Hyperparameter coverage: λ is swept over {0,1,2,3,10} (Table 5), and reference set size is swept over 1,000–30,000 (Table 7), both only on Purchase100.
- Not reported in the paper: multiple repeated experiments, variance, or confidence intervals; evaluation under white-box attacks (where the attacker can see model parameters); training time and computational overhead.
Summarize the main content of the paper
Using min-max adversarial training with the membership inference attack gain as a regularization term, attack accuracy is pushed close to random guessing on 4 model/datasets at an accuracy cost of 1.1%–4.4%.
A defense paper on machine learning membership privacy: against black-box membership inference attacks (the attacker can only see prediction API outputs and determine whether a record is in the training set), the authors propose adversarial regularization—modeling the defense as a min-max game between a classifier and the strongest inference attack, solved with GAN-style alternating gradient training.
- Problem to solve: simple mitigation techniques (top-k restriction, L2 regularization) do not provide strict privacy guarantees; differential privacy at small ϵ has a high utility cost for large models, and does not incorporate utility into the design objective (according to the authors).
- Method highlights: the classifier minimizes "classification loss + λ×attack empirical gain", and the attack term acts as a regularizer; theoretically, equilibrium is reached when the model's prediction output distributions p_f and p_f′ on the training set and non-members are indistinguishable, and the optimal attack degenerates to random guessing; the attack model is a single fully connected network with input (x, y, f(x)).
- Main results (Table 4): Purchase100 attack accuracy drops from 67.6% to 51.6% (test accuracy 80.1%→76.5%); Texas100 from 63% to 51.0% (51.9%→47.5%); CIFAR100-Alexnet from 53.2% to 50.7% (44.7%→43.6%); CIFAR100-Densenet from 54.5% to 51.0% (70.6%→67.6%).
- Comparison baseline (Table 6): at a similar privacy level, L2 regularization (factor 0.01) achieves only 32.1% test accuracy on Purchase100, whereas the mechanism in this paper achieves 76.5%.
- Additional effect: the authors state that the mechanism is also a strong regularizer, shrinking the generalization error to at most 1/4 of the original (e.g., Texas100 from 29.7% to 7.5%), but it is almost unchanged on Purchase100.
- Authors' conclusion: given available training/reference data and model capacity, this mechanism guarantees membership privacy against the strongest inference attacks at a negligible accuracy cost, and the model generalizes well.
Related papers
Show 3 more papers
- The Privacy Onion Effect: Memorization is RelativeAnalysis / interpretabilityarXiv