How to combine membership inference attacks against multiple updated models

1. 01What problem does this paper attempt to solve?

   This work studies whether repeated model updates exacerbate membership inference risk, and proposes a black-box MI attack that exploits the models before and after an update.

   Core question: When a machine learning model is repeatedly updated with new data and an attacker can observe the models before and after the update, how much does membership inference (MI) risk increase, and how can this additional information be exploited.

   • Problem setting: The authors point out that production platforms (Amazon SageMaker, Azure ML, Tensorflow-Extended) commonly support automatically updating models daily or weekly with new batches of data, and the updated models are repeatedly released for inference. Most MI research targets only a single independent model and ignores the additional information that updates give an attacker.
   • Core observation: Using a mean estimation example, the authors show that combining the means before and after an update can recover the mean of the update batch itself, enabling accurate membership inference in a parameter regime that would otherwise be impossible (n ≫ d ≫ t); the authors believe the same effect occurs in more complex models.
   • Threat model:
     • Attacker goal: Infer whether a target sample belongs to a certain update set D_i; with multiple updates, also infer which update the sample belongs to (specific accuracy).
     • Knowledge: black-box—can query the output of each model f_i, but does not know the model architecture or parameters.
     • Capability: Observe the outputs of the initial model f0 and all updated models f1…fk at multiple query points; when setting thresholds, three strategies—Batch/Transfer/Rank—each assume different side information.
     • Victim system: A supervised classification model updated with SGD-New (only new data) or SGD-Full (all accumulated data).
   • Proposed in the paper: The authors claim to propose the first black-box MI attack algorithm for the repeated model update setting, improving attack effectiveness by combining single-model MI attack scores run separately on the models before and after the update, and provide analytical and experimental support; they also measure for the first time the effect of distribution shift on MI attacks with updates (the authors phrase this as "the first measurements").

   Collapse this answer

2. 02What related research exists?

   Related work falls into four categories: single-model MI, attacks in multi-model/update scenarios, memorization attacks, and differential privacy defenses.

   Single-model membership inference

   • Early work such as Homer et al. [16], Sankararaman et al. [33], and Dwork et al. [13] proposed MI attacks for simple statistical tasks; Yeom et al. [40] linked MI to overfitting, and Shokri et al. [36] proposed the shadow-model black-box attack.
   • Carlini et al. [5] proposed the LiRA attack (used in this paper as one of the score functions); Choquette Choo et al. [10] proposed the gap attack in the label-only setting (used in this paper as a baseline).

   Privacy attacks in multi-model and model update scenarios

   • Federated learning: Wang et al. [39] infer label composition, Melis et al. [26] infer properties, and Nasr et al. [29] perform MI attacks.
   • Salem et al. [32] (Updates-Leak) use the models before and after an update to perform a reconstruction attack, recovering the labels and features of the update points. The paper notes the difference: it is limited to small updates in online learning, whereas this paper focuses on MI, supports multiple updates, various update sizes, and distribution shift, and does not rely on a large number of shadow models.
   • Chen et al. [9] construct an MI attack against machine unlearning. The paper notes that unlearning can be viewed as the inverse operation of an update, and compares experimentally in Section 5.8, stating that the attack in this paper is more effective under their setting.

   Memorization attacks

   • Carlini et al. [6, 7] extract training data from generative language models, and [7] uses MI to verify whether generated samples belong to the training set.
   • Zanella-Béguelin et al. [42] state that updates to language models improve the effectiveness of memorization attacks; the paper notes that their experiments cannot distinguish whether this is privacy leakage or the model adapting to a new distribution, whereas the method in this paper can distinguish.

   Differential privacy and defenses

   • Nasr et al. [28] and Jia et al. [20] propose defenses specifically targeting MI; Abadi et al. [1], Song et al. [38], and others propose differentially private training algorithms (DP-SGD).
   • Jagielski et al. [18] and Nasr et al. [30] use MI attacks to audit differential privacy deployments; this paper follows this approach to audit update scenarios.

   Baselines and benchmarks: The experimental baselines are the Loss attack of Yeom et al., the Gap attack of Choquette Choo et al., and LiRA [5] (except IMDb); the multi-update setting additionally has Random and Generic baselines, and Section 5.8 reproduces the SortedDiff attack of Chen et al.; the datasets are FMNIST, CIFAR-10, Purchase100, and IMDb, and distribution shift uses the BREEDS framework [34].

   Authors' positioning: This paper states it is the first work to study aspects specific to repeated model updates (large-batch updates, multiple updates, distribution shift); the attack is implemented by combining the scores of existing single-model MI attacks, and is more efficient than attacks designed for unlearning.

   Collapse this answer

3. 03How does the paper address this problem?

   Apply the single-model MI score to the models before and after the update respectively, combine them with ScoreDiff/ScoreRatio, and set a threshold for the decision.

   Overall approach: For a target sample, compute the confidence score of a single-model MI attack on the initial model f0 and the updated model f1 respectively, then use a fixed combination function to combine the two scores into a membership decision score for the update set, and output a binary decision with a threshold. See Table 1 for the method overview.

   Problem formalization

   • The learner sequentially obtains datasets D0…Dk, trains f0 with A_train, then uses the update algorithm A_up to sequentially produce f1…fk (Algorithm 1); each update set has a fixed size n_up.
   • MI experiment (Algorithm 2): The attacker obtains all models f0…fk and must distinguish whether a sample comes from the update set or an i.i.d. test set, and (with multiple updates) indicate which update it belongs to; metrics are accuracy, precision, recall (single update) and generic/specific accuracy (multiple updates).
   • Two update strategies: SGD-New updates only with the new dataset D_i (small learning rate, few epochs to prevent forgetting); SGD-Full updates with all accumulated data [D0…D_i].

   Single-update attacks: ScoreDiff and ScoreRatio

   • Core combination functions: ScoreDiff = ℓ(x,y,f₁) − ℓ(x,y,f₀), ScoreRatio = (ℓ(x,y,f₁)+c)/(ℓ(x,y,f₀)+c), where ℓ is the single-model MI score and c>0 is a damping constant to prevent the denominator from approaching 0.
   • Two choices of score function: standard cross-entropy loss [40] and LiRA score [5]; both require only class probabilities, and LiRA additionally requires training shadow models.
   • The authors prove that combining binary decisions alone yields no benefit (Theorem 4.1): under 0/1 loss, if f0 performs the same on update points and test points and f1 performs better on update points, then an attack using only f1 is no worse than an attack using both f0 and f1, so continuous scores must be exploited.
   • Analytical basis: For ℓ2² loss (mean), a single-step gradient update reduces the loss by a fixed ratio, and ScoreRatio is a perfect membership test; for ℓ2 loss (geometric median), the loss decreases by a fixed constant, and ScoreDiff is a perfect test. The authors therefore argue that no universally optimal combination function exists.

   Threshold setting (three side-information assumptions)

   • Batch: The attacker holds unlabeled data containing update points and test points, and uses the median score (maximizing accuracy) or the 10th percentile (maximizing precision) as the threshold.
   • Transfer: The attacker trains a shadow model with test points and simulates updates, sets the threshold on the shadow model using the Batch method, then transfers it to the target model; only one shadow model is needed.
   • Rank: Uses the q-quantile of test-point scores as the threshold, with false negative rate q (following [23]).

   Multi-update attacks (Algorithm 3)

   • Back-Front: Ignores intermediate models and compares only the score difference/ratio between f0 and fk, used for generic accuracy.
   • Delta: Compares adjacent models f_{i-1}, f_i pairwise; if the score change exceeds threshold T_i, the sample is judged to belong to the i-th update, used for specific accuracy and also convertible to a generic decision.
   • Thresholds use the Batch strategy: Back-Front takes the median score over the full dataset; Delta sets each T_i so that each update is assigned n_up points.

   Distribution shift evaluation method

   • Uses the BREEDS framework [34] to convert CIFAR-10 into an animal vs. vehicle binary classification, with update distribution D1 = (1−α)D_S + αD_T, α ∈ {0.2, 0.6, 1.0} controlling shift strength; two class-transfer groups are set: Hard (Airplane→Automobile, Cat→Bird) and Easy (Automobile→Truck, Cat→Dog). Test points and update points are sampled from the same distribution, to isolate the effect of shift on MI itself.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On four datasets, the update attack consistently outperforms the no-update baseline; the larger the shift, the higher the risk; DP with a small ε provides good protection; it outperforms the unlearning attack.

   Experimental setup

   • Models evaluated: logistic regression on FMNIST (initial training set of 1000 points); ImageNet-pretrained VGG-16 fine-tuned on CIFAR-10 (25000 points); a single layer neural network on Purchase100 (25000 points); BERT base (uncased) fine-tuned on IMDb (25000 points).
   • Datasets: FMNIST, CIFAR-10, Purchase100, IMDb; the distribution shift experiments use BREEDS-adapted CIFAR-10 binary classification.
   • Baselines: the no-update baselines are Gap [10], Loss [40], and LiRA [5] (LiRA is not used for IMDb); the multi-update baselines are Random (1/2k) and Generic (p/k); Section 5.8 reproduces the SortedDiff attack of Chen et al. [9] (up to 30 shadow models).
   • Metrics: accuracy, precision, and recall for single updates; generic accuracy and specific accuracy for multiple updates.
   • Update size: in the single-update experiments the update set is 1%–32% of the initial training set (IMDb is fixed at 10–320 points); in the multi-update experiments n_up is fixed at 250 (CIFAR-10), 100 (Purchase100), and 10 (FMNIST, IMDb), with 1–10 updates.
   • Number of repetitions: the paper does not state the number of repetitions for the main experiments; the DP auditing experiments have 400 trials per ε (20 models × 20 points each), with 98% confidence.

   Main results

   Single-update attack accuracy (Table 2, update set at 1% of the initial training set, loss score):

   Swipe horizontally to read the table

   Dataset	Training strategy	No-update best (Gap/Loss)	ScoreDiff Batch	ScoreRatio Batch	ScoreRatio Transfer
   FMNIST	SGD-New	0.51/0.53	0.68	0.71	0.72
   FMNIST	SGD-Full	0.52/0.51	0.63	0.57	0.58
   CIFAR-10	SGD-New	0.50/0.53	0.61	0.65	0.68
   CIFAR-10	SGD-Full	0.50/0.59	0.58	0.61	0.63
   Purchase100	SGD-New	0.51/0.50	0.56	0.57	0.58
   IMDb	SGD-New	0.50/0.45	0.63	0.63	0.60

   (According to Table 2; the SGD-Full rows for Purchase100 and IMDb and some Transfer columns are omitted, with consistent trends.)

   • The authors report that across all datasets, update sizes, and training strategies, attacks with updates outperform no-update attacks (Figure 2); the advantage shrinks as the update set grows, with the Batch attack at 79% at 10 update points on FMNIST and 70% at 320 points, and the gap versus no update narrowing from 27% to 18%.
   • The authors state that ScoreRatio is usually better than ScoreDiff, but with large update sets ScoreDiff can be comparable or better; Batch and Transfer thresholds perform comparably; LiRA scores are generally better than loss, but perform poorly on CIFAR-10 because there is insufficient data to train enough shadow models.

   Multiple updates and training strategy (Q2, Q3)

   • Multiple updates: Figure 3 shows that the multiple of specific accuracy relative to random guessing rises with the number of updates: on CIFAR-10 with SGD-New, 2.03× at 2 updates and 6.10× at 8 updates; but absolute specific accuracy decreases with k (on CIFAR-10 from 50.8% at 2 updates to 38.1% at 8 updates, with baselines of 25% and 6.25% respectively).
   • Training strategy: the authors report that with small update sets SGD-New is more vulnerable, and with large update sets this reverses—on CIFAR-10, from 250 to 8000 update points, the accuracy drop is 10% for SGD-New and only 2% for SGD-Full; Purchase100 already reverses at the smallest update set. The empirical rule the authors give is: for small update sets, training on the full data is safer, and for large update sets the opposite, with the reversal point varying by dataset.

   Distribution shift (Q5, Table 4)

   • On BREEDS-adapted CIFAR-10 (n_up=50, n0=5000, Batch threshold), under Hard shift as α rises from 0.2 to 1.0, ScoreRatio accuracy rises from 0.559 to 0.687 for SGD-New and from 0.597 to 0.666 for SGD-Full; under Easy shift accuracy instead decreases as α increases.
   • Precision often exceeds 90% under severe shift, with a maximum of 98% (Hard, α=1.0, SGD-New, ScoreDiff); the authors state that severe shift carries higher MI risk than gradual shift.

   Differential privacy (Q6, Figure 5)

   • Using DP-SGD on FMNIST (update set 100, clipping norm 0.5, δ=10⁻⁴): at ε=0.26 the best attack precision is only 0.59, at ε=1.1 it is 0.63, and then it levels off; for ε<1 there is little difference between attacks with and without updates.
   • Auditing: converting attack precision into an empirical lower bound on ε (Clopper-Pearson, 98% confidence), the lower bounds for SGD-Full are all 0; for SGD-New, when the ε upper bound is 0.12–1.09 the lower and upper bounds differ by 2.0-3.6x. The authors state that in independent-model DP-SGD auditing this gap is 5-10x [18, 30], and argue that the update scenario carries higher risk.

   Comparison with Chen et al. [9] (Q7, Figure 6)

   • On FMNIST, ScoreDiff and ScoreRatio (each using 1 shadow model to set the threshold) outperform the SortedDiff attack of the other work with up to 30 shadow models across all update sizes, with the gap varying with update size; the conclusion is the same on Purchase100.
   • After giving the other work's attack model two additional features from this paper (12 dimensions in total, 30 shadow models, logistic regression), the average weight of this paper's features is 7.5x that of the SortedDiff features. The authors state that this reflects the value of hand-crafted test statistics relative to learning from a high-dimensional space.

   Other ablations and analyses

   • Delta vs. Back-Front (Figure 4, SGD-Full): Delta is best on CIFAR-10, Back-Front is best on Purchase100, and they are comparable on FMNIST.
   • precision/recall (Table 3, Table 5): precision for SGD-New is higher than for SGD-Full (on FMNIST at n_up=0.01n0, Batch is 90% and 86% respectively); precision is higher with fewer update points (FMNIST Transfer drops from 86% to 78%).
   • The Rank threshold has higher recall, but Batch/Transfer usually have higher precision.
   • Mean estimation experiment (Figure 7, Appendix C): n0=200, d=250, σ=0.1, 60 trials; the attack using updates is no worse than the no-update Neyman-Pearson optimal attack for all n1.
   • Appendix Figure 8: on FMNIST with n_up=100, the attack on SGD-Full can outperform SGD-New, consistent in direction with the opposite of the main text's small-update-set conclusion.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that there is no universally optimal attack and that defense heuristics need dataset-specific tuning; the experiments are limited to four datasets and specific update settings.

   Limitations noted by the authors and future directions

   • No universally optimal method (Section 4.2, 5.5): The authors state that no combination function or threshold strategy is optimal in all situations, and attackers need to experimentally choose ScoreDiff/ScoreRatio and the score function for the specific scenario (e.g., using a self-built shadow model).
   • Defense heuristics require tuning (Section 5.4): The crossover point at which SGD-New versus SGD-Full is more vulnerable depends on the dataset; the authors suggest learners tune this heuristic by running the attacks in this paper, and note that DP is a more principled defense but may still require tuning or auditing in practice.
   • Limited DP audit evidence for SGD-Full (Section 5.7): The attack could not rule out the possibility that SGD-Full satisfies ε=0, but the authors emphasize that this is not proof that SGD-Full retraining has no privacy issues.
   • Score functions are extensible (Section 4.1): The authors state that better single-model MI score functions in the future can be directly incorporated into their combination strategy.
   • Distribution shift methodology (Section 5.6): The authors believe their shift evaluation method can become a platform for future measurement of MI under distribution shift.

   Experimental coverage

   • The models tested are logistic regression, VGG-16, single layer neural network, and BERT base (uncased), 4 in total, corresponding to the four datasets FMNIST, CIFAR-10, Purchase100, and IMDb respectively.
   • Update strategies cover both SGD-New and SGD-Full; multi-update experiments use at most 10 updates, with n_up fixed at 1%, 0.4%, 1%, 0.04% of the initial dataset (CIFAR-10, Purchase100, FMNIST, IMDb).
   • Distribution shift experiments are conducted only on the BREEDS binary classification task converted from CIFAR-10, with α ∈ {0.2, 0.6, 1.0}, one class-transfer group each for Hard/Easy.
   • DP experiments are conducted only on FMNIST in the single-update setting, with δ=10⁻⁴ and 400 trials per ε.
   • The paper does not report the number of repetitions or confidence intervals for the main experiments (Figure 2–4, Table 2–4); the comparison with Chen et al. uses the loss score and Transfer threshold, and the authors state that for simplicity they did not compare the LiRA score.

   Collapse this answer

6. 06Summarize the main content of the paper

   Combining the MI scores of the models before and after an update improves membership inference; the smaller the update and the larger the shift, the higher the risk; DP is effective at small ε.

   A membership inference (MI) attack paper: it studies how much additional privacy information a black-box attacker who can observe the models before and after an update can obtain when models in production scenarios are repeatedly updated with new data.

   • Problem: Most MI research targets only a single model, whereas real systems update models in batches; the authors aim to quantify the additional MI risk brought by updates, covering factors such as multiple updates, update size, distribution shift, and training strategy.
   • Method: For target samples, single-model MI scores (loss or LiRA) are computed on f0 and f1 respectively, combined using ScoreDiff or ScoreRatio, with Batch/Transfer/Rank threshold decisions; in multi-update scenarios, Back-Front (comparing the first and last models) and Delta (comparing adjacent models) are used. The authors prove that binary decisions alone cannot benefit from updates (Theorem 4.1), and use mean/median examples to analytically show that the two combination functions each have applicable scenarios.
   • Main results:
     • On four datasets, attacks with updates consistently outperform the no-update baseline; on FMNIST at 10 update points the accuracy is 79%, versus 52% with no update (Figure 2 and abstract).
     • The smaller the update set, the more effective the attack; with small update sets SGD-New is more vulnerable, and with large update sets this reverses (on CIFAR-10, from 250 to 8000 points, SGD-New accuracy drops 10%, SGD-Full only 2%).
     • With multiple updates, the multiple of specific accuracy relative to the baseline rises with the number of updates (CIFAR-10, 8 updates: 6.10×), but absolute accuracy decreases with k.
     • In the distribution shift experiments, under Hard shift the larger α is, the more accurate the attack (SGD-New ScoreRatio rises from 0.559 to 0.687), with precision up to 98%; the authors state that severe shift carries higher risk than gradual shift.
     • DP-SGD protects well at small ε (precision 0.59 at ε=0.26), and protection is limited at large ε; the empirical lower bound of ε for SGD-New differs from the theoretical upper bound by 2.0-3.6x.
     • Compared with the unlearning attack of Chen et al. [9], the attack in this paper with 1 shadow model outperforms their 30 shadow models across all update sizes.
   • Authors' conclusion: Model updates increase MI risk, and update set size, training algorithm, and distribution shift are key factors; no single attack configuration is universally best-performing, and learners can use the attack in this paper to tune defense heuristics or audit differential privacy deployments.

   Collapse this answer

Related papers