A Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

1. 01What problem does this paper attempt to solve?

   Using white-box membership inference attacks to quantify privacy leakage of deep models about training samples, covering centralized and federated learning.

   Core question: how much information a deep neural network leaks about individual data records in its training set, and how to quantify this leakage with white-box membership inference attacks.

   • Problem setting: deep models are used on sensitive data such as speech, medical and financial data; the authors define privacy leakage as the information an attacker learns from the model that cannot be inferred from other models of the same distribution about a training sample, and measure it with the decision problem of membership inference.
   • Shortcomings of existing methods: existing membership inference attacks are limited to black-box (only seeing model prediction outputs); the authors argue such attacks may not be effective enough against well-generalizing deep models; the authors' experiments show that directly extending black-box attacks to the outputs of all activation layers does not exceed the accuracy of black-box attacks, because earlier layers generalize faster and do not carry training-set-specific information.
   • Core observation: SGD training pushes the gradient of the loss on training samples toward zero, and each training record leaves a distinguishable trace in the gradients of all model parameters; the gradient vector has enormous dimensionality and is hard to generalize, so the gradient distributions of members and non-members may be separable even if the model generalizes well at the prediction level.
   • What the paper proposes: a white-box membership inference framework with gradients as the core feature, covering stand-alone (including fine-tuning) and federated learning, passive and active attackers, and supervised and unsupervised prior knowledge, evaluated on pretrained CIFAR100 models and in federated learning settings.
   • Threat model:
     • Goal: determine whether a target data record is in the target model's training set (membership inference).
     • Knowledge: white-box—the attacker can access the model architecture, parameters W, intermediate-layer computations and the required hyperparameters; prior knowledge is divided into supervised (a known subset of D plus same-distribution non-members) and unsupervised (only same-distribution data, without knowing member labels).
     • Capability: a passive attacker only observes the model or parameter updates; an active attacker (a participant in federated learning or the central aggregator) can tamper with the parameter updates it uploads, and the aggregator can also control the global parameters that participants see (isolating the target participant).
     • Victim system: a centrally trained and fully trained model, a fine-tuned model, and federated learning with averaging aggregation (N participants plus a central parameter server).

   Collapse this answer

2. 02What related research exists?

   Related work is divided into four groups: black-box membership inference, collaborative learning inference, other inference attacks, and differential privacy defenses.

   Black-box membership inference attacks

   • Shokri et al. [6]—propose a black-box membership inference attack against machine learning models, using shadow models to imitate the target model's behavior to train the attack model; this paper uses it as the comparison baseline for the unsupervised attack and extends its setting.
   • Salem et al. [17]—extend [6]; the authors state that the attack can be done with a single shadow model, and that when target training data is unavailable, statistical properties of the output (such as entropy) can be used for membership inference.
   • Yeom et al. [7]—analyze the relationship between overfitting and membership inference; Hayes et al. [18]—use GANs to perform membership inference on generative models; Homer et al. [4]—early membership inference work on genomic data.

   Inference in collaborative/federated learning

   • Melis et al. [19]—membership inference attack against collaborative learning; the paper points out that it assumes participants update the central server after each mini-batch (rather than each epoch), and that the attack applies only to models with explicit word embeddings and very small mini-batches. The authors state that this paper evaluates standard learning mechanisms and standard target models, and that the attack also holds for pretrained models.

   Other inference attacks

   • Input inference [31], property inference [32], parameter inference [33], [34], side-channel attacks [35]: attackers use additional information about the training data distribution to make various inferences.
   • Ateniese et al. [36]—an attacker with access to SVM and HMM parameters can extract information about training data (e.g., a speaker's accent in speech recognition).

   Defense directions

   • Differential privacy [22]–[27] is used as a defense against inference attacks; adversarial training [28]–[30] can achieve a better trade-off between privacy and accuracy. The paper points out that this line of work focuses on membership inference in the black-box setting.

   Baselines and benchmarks: the comparison for the unsupervised attack is the shadow models method of Shokri et al. [6] (Table VII); the experimental benchmarks are CIFAR100, Purchase100 and Texas100. The authors position this paper as: extending privacy analysis from black-box, fully trained models to a complete framework covering white-box, the training process (fine-tuning and federated learning), and passive/active attackers.

   Collapse this answer

3. 03How does the paper address this problem?

   A layered attack model is trained with the gradients of the loss with respect to each layer's parameters as the core feature, covering four combinations of attack dimensions.

   Overall approach: exploiting the property that SGD pushes the gradients on training samples toward zero, the gradients on the target data point, the activations of each layer, the output, the loss and the label are used as features to train a layered attack model that outputs a membership probability; the framework can be summarized as a white-box membership inference attack (architecture in Figure 1, Table XIV).

   Attack dimensions and settings

   • The paper divides attacks along four dimensions (Table I): observation (black-box / white-box), target (stand-alone including fine-tuning / federated), mode (passive / active), prior knowledge (supervised / unsupervised).
   • In the white-box setting the attacker can compute any function of W and x; the authors point out that simply extending black-box attacks to the outputs of all activation layers has limited effect, because the activations of each layer generalize faster than the output layer.

   Attack features and attack model architecture

   • Input features: the gradients of each layer ∂L/∂W₁, ∂L/∂W₂, …, the activations of each layer h₁(x), h₂(x), …, the model output f(x), the one-hot encoding of the label y, and the loss L(f(x; W), y), each fed into a separate component.
   • Gradients are processed by a CNN submodule (for gradients of fully connected layers, the convolution kernel size is set to that layer's input dimension, to capture the correlation of gradients within each activation function); the output, label and loss use a single-hidden-layer FCN submodule; the outputs of the components are concatenated and passed through a multi-hidden-layer FCN encoder to output a single score.
   • Attack model details (Table XIV): each component is 2 fully connected layers (128, 64) or convolutional layers (1000 kernels) plus 2 fully connected layers, and the encoder is 4 layers (256, 128, 64, 1), all using ReLU and 0.2 dropout; in the unsupervised case a 2-layer decoder (64, 4) is attached.

   Federated learning and fine-tuning settings

   • In federated learning the attacker can be the central parameter server (seeing each participant's individual updates) or a participant (seeing only the global parameters); the attacker observes T versions of the model, and the attack components stack the T observations and process them at once (e.g., the loss feature is a 1×T-dimensional input), to capture the dependence of parameters over time.
   • The fine-tuning setting is a special case of the federated attack model: the attacker sees two model snapshots, before and after fine-tuning, and can distinguish D, D∆ and non-members.

   Passive and active attacks

   • Passive attacks only observe the training process. Active attack (Equation 2): the attacker performs gradient ascent on the target record x, W \leftarrow W + \gamma \frac{\partial L_x}{\partial W}}, i.e., updates the local parameters in the direction that increases the loss on x and then uploads them; if x is a member, SGD will quickly push its gradient down, and this abrupt change can be detected by the attack model.
   • An active global attacker can also isolate the target participant: not sending it the aggregated parameters of the other parties, so that its model only memorizes its own training data, thereby increasing leakage.

   Supervised and unsupervised training

   • Supervised: the attacker knows D′ (containing some members and same-distribution non-members), and minimizes the mean squared loss (Equation 3) so that the attack output h(d) approximates the membership probability h(d) = Pr(d ∈ D; f) (Equation 4); when no members are known, shadow models can be used for supervised training.
   • Unsupervised: an encoder-decoder structure is used so that the attack output (the encoded value) can reconstruct key features such as the loss, whether the prediction is correct, the confidence of the correct label, the prediction entropy and the gradient norm; after training the decoder is discarded, spectral clustering is performed on the encoded values to split them into two clusters, and the cluster with the smaller average gradient norm is judged to be members.

   Success criterion and metrics

   • The main metric is attack accuracy: the proportion of correct member/non-member predictions on unseen data, with equal numbers of member and non-member samples at evaluation (random guessing baseline 50%); true/false positive rate and prediction uncertainty (normalized entropy, Equation 5) are also reported.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluated on 3 datasets and 4 architectures, white-box gradient attacks exceed black-box, and federated active attacks reach up to 92.1% accuracy.

   Experimental setup

   • Models tested: Alexnet, ResNet, DenseNet on CIFAR100 (including publicly available pretrained versions, with DenseNet test accuracy 82%, ResNet 73%, Alexnet 44%); fully connected models on Texas100 and Purchase100 (Table VIII).
   • Datasets: CIFAR100 (60,000 32×32 images, 100 classes); Purchase100 (197,324 purchase records, 600-dimensional binary features, 100 classes); Texas100 (67,330 discharge records, 6,170-dimensional binary features).
   • Data splits: see Table II for the stand-alone setting (e.g., CIFAR100: 50,000 target training, 25,000 attack training members and 5,000 non-members, 5,000 test members/non-members each); see Table XI for the federated setting.
   • Baselines: black-box attack (using only the prediction vector); the comparison for the unsupervised attack is the shadow models of Shokri et al. [6].
   • Metrics and criterion: attack accuracy (proportion of correct predictions on a test set with half members and half non-members), TP/FP rate, prediction uncertainty (normalized entropy); the unsupervised attack uses spectral clustering into two classes.
   • Training details: target models are trained for 100 epochs with SGD (CIFAR100) or Adam (fully connected models); attack models use Adam, learning rate 0.0001, batch size 64, trained for 100 epochs with the highest test accuracy selected; the federated experiments use 4 participants with averaging aggregation, and the global attacker observes only 5 non-consecutive epochs due to GPU resource constraints.

   Main results

   According to Table VIII (stand-alone, CIFAR100 models are publicly available pretrained versions):

   Swipe horizontally to read the table

   Dataset	Architecture	Train/Test accuracy	Black-box	White-box (output)	White-box (gradient)
   CIFAR100	Alexnet	99% / 44%	74.2%	74.6%	75.1%
   CIFAR100	ResNet	89% / 73%	62.2%	62.2%	64.3%
   CIFAR100	DenseNet	100% / 82%	67.7%	67.7%	74.3%
   Texas100	Fully Connected	81.6% / 52%	63.0%	63.3%	68.3%
   Purchase100	Fully Connected	100% / 80%	67.6%	67.6%	73.4%

   • The authors state that gradients leak more membership information than layer outputs; using only the layer outputs (white-box output column) yields almost no improvement over black-box.
   • The authors state that ResNet and DenseNet have similar generalization error but different degrees of membership leakage (64.3% vs. 74.3%), argue that generalization error is not a suitable metric for white-box privacy leakage, and note that DenseNet has 25.62M parameters and ResNet 1.7M.
   • The authors state that even a well-generalizing model (DenseNet test accuracy 82%) has high leakage under white-box membership inference attacks.

   Federated learning attacks (Table X, 4 participants)

   • Passive attack: the global attacker (aggregator) achieves 85.1% on CIFAR100-Alexnet and 79.2% on DenseNet; the local attacker (participant) achieves 73.1% and 72.2% respectively; on Texas100 and Purchase100 the accuracy is lower than in the stand-alone setting, which the authors explain as averaging aggregation weakening the influence of a single participant.
   • Active gradient ascent: the global attacker achieves 88.2% on Alexnet and 82.1% on DenseNet; the local attacker achieves 76.3% and 76.7% respectively; active attacks are evaluated on 100 random members plus 100 non-member samples.
   • Isolation attack: after the global attacker isolates the target participant, 84.3% on DenseNet, and 87.3% when combined with gradient ascent; on Purchase100 isolation plus gradient ascent reaches 82.5%, and on Alexnet 92.1%.

   Unsupervised attacks and fine-tuning attacks

   • Unsupervised attack (Table VII): CIFAR100-Alexnet 75.0%, DenseNet 71.2%, ResNet 63.1%, Texas100 66.3%, Purchase100 71.0%, all higher than the shadow models comparison (70.5%, 64.2%, 60.9%, 65.3%, 68.2%); the authors state that the encoded values show different distributions for members/non-members (Figure 5).
   • Fine-tuning attack (Table IX): given the two snapshots before and after fine-tuning, the attacker can distinguish D from D∆ (CIFAR100-Alexnet 62.1%, DenseNet 63.3%), D from non-members (75.4%, 74.6%), and D∆ from non-members (71.3%, 71.5%).

   Other ablations and analyses

   • Layer outputs: last layer 74.6%, second-to-last 74.1%, third-to-last 72.9%, and combining all three layers is still 74.6% (Table III, pretrained Alexnet).
   • Layer gradients: last layer 75.1%, second-to-last 74.6%, third-to-last 73.5%, fourth-to-last 72.6%, and 75.15% after combining (Table IV); combining all gradients plus outputs gives 75.18% (Table V).
   • Attacker training data size: 73.2% with 10,000 members/2,000 non-members, rising to 75.1% with 25,000/5,000 (Table VI).
   • Choice of observed epochs: 57.4% using epochs 5–25, 85.1% using 100–300 (Table XII, federated passive global attack, CIFAR100-Alexnet).
   • Number of participants: 89.0% with 2 parties, dropping to 67.2% with 5 parties (Table XIII, federated passive local attack, CIFAR100-Alexnet).
   • Gradient norm and uncertainty: member gradient norms decrease with training while non-member ones do not (Figure 3); attacks are more accurate for classes with larger member/non-member gradient norm differences (Figure 2); classes with larger prediction uncertainty differences leak more (Figure 6).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors only note that theoretical bounds on white-box privacy leakage were not studied; the experiments cover 3 datasets and a 4-participant federated setting.

   Limitations and future directions noted by the authors

   • The paper states in the Conclusions (Section 6): it did not study theoretical bounds on deep learning privacy leakage in the white-box setting, leaving this as a future research direction. Beyond this, the paper does not specifically discuss other limitations.

   Scope of experiments

   • The models tested are Alexnet, ResNet, DenseNet on CIFAR100 and fully connected models on Texas100 and Purchase100, for a total of 4 architectures and 3 datasets (Table VIII); the CIFAR100 experiments used publicly available pretrained models.
   • The federated learning experiments are fixed at 4 participants, averaging aggregation, and equal-sized training data per participant; the effect of the number of participants was tested only on the passive local attack of CIFAR100-Alexnet with 2–5 parties (Table XIII).
   • The federated global attacker observes only 5 non-consecutive training epochs due to GPU resource constraints (Section 4 D); active attacks are evaluated on 100 random members plus 100 non-members (Section 4 E).
   • The unsupervised attack's comparison is a single shadow model trained per dataset; the CIFAR100 shadow model's training data overlaps with the target model's training set (Section 4 B).
   • The paper does not report the runtime or computational cost of the attacks, nor the variance across multiple repetitions; on defenses, it discusses differential privacy and adversarial training only in related work, and the experimental section does not evaluate any defense.

   Collapse this answer

6. 06Summarize the main content of the paper

   White-box membership inference framework: attacks centralized and federated learning using SGD gradient features, reaching 74.3% on pretrained DenseNet.

   An attack paper on deep learning privacy analysis, using white-box membership inference to quantify model leakage about training samples.

   • Problem: Existing black-box membership inference attacks may fail against well-generalized deep models; the authors find that directly extending black-box attacks to the outputs of all activation layers cannot exceed black-box accuracy, because earlier layers generalize faster.
   • Method: Exploiting the property that SGD pushes training sample gradients toward zero, using the gradient of the loss with respect to each layer's parameters as the core feature, together with activations, outputs, loss, and labels, to train layered CNN/FCN attack models; supports both supervised and unsupervised (autoencoder plus spectral clustering) priors, covers stand-alone, fine-tuning, and federated learning, and proposes active attacks with gradient ascent and participant isolation.
   • stand-alone results: On publicly pretrained CIFAR100-DenseNet (test accuracy 82%), black-box attack 67.7%, white-box gradient attack 74.3% (Table VIII); the authors accordingly state that well-generalized models also have high membership leakage, and that generalization error cannot measure white-box privacy leakage (ResNet has similar generalization but only 64.3%).
   • Federated learning results: In a 4-participant setting, on DenseNet passive local attack 72.2%, passive global (aggregator) 79.2%; active gradient ascent rises to 76.7% and 82.1% respectively, and reaches 87.3% after the aggregator isolates the target participant, and on Alexnet isolation plus gradient ascent reaches 92.1% (Table X).
   • Unsupervised results: The unsupervised attack, which does not require known membership labels, reaches 75.0% on CIFAR100-Alexnet, higher than the shadow models baseline of 70.5% (Table VII).
   • Authors' conclusions: Even well-generalized models are vulnerable to white-box membership inference attacks; repeated parameter updates in federated learning increase attack accuracy, and both a curious aggregator or participant can launch accurate attacks against other participants; theoretical bounds on white-box privacy leakage are left for future work.

   Collapse this answer

Related papers