Skip to content

Why Train More? Using Memorization for Effective and Efficient Membership Inference Attacks

Why Train More? Effective and Efficient Membership Inference via Memorization

arXiv· arXiv:2310.08015· Jihye Choi, Shruti Tople, Varun Chandrasekaran, Somesh JhaUniversity of Wisconsin-Madison; Microsoft; University of Illinois Urbana-ChampaignIndexed by this site · 23 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors state that by explaining differences in MIA success rates via label memorization, using high-memorization samples can reduce shadow models from 2000 to 5 (400×).

Threat modelThe adversary knows the data distribution D and the learning algorithm L, and assumes the stronger M I^s game: the adversary also controls the training dataset S (e.g., an auditor verifying data deletion). It is assumed that a memorization oracle O_mem exists that returns an approximate memorization score for a sample. Attack success is measured by high TPR at low FPR (0.1%).

Questions
Membership inference attacks (MIA) have widely varying success rates across samples, and practical black-box MIA often requires training hundreds of shadow models, incurring high computational cost. The authors aim to identify a data property that explains this variation and use it to improve both attack effectiveness and efficiency.
Methods
The authors propose using label memorization to explain sample susceptibility to MIA: theoretically, they give a lower bound of MI advantage in terms of the memorization score, with the required number of shadow models inversely proportional to the memorization score; experimentally, they have the adversary select high-memorization samples (singletons) to launch attacks.
Experiments and results
Evaluated on 7 dataset combinations, 4 models, and 5 MIAs: on highly memorized samples, the Carlini et al. attack reaches AUROC 1.0 and [email protected]%FPR up to 100%; the number of shadow models needed to reach the same AUROC drops from 2000 to 5 (the authors state a 400× reduction).
Limitations and future work
The authors state that instantiating the memorization oracle O_mem (Feldman and Zhang algorithm) is computationally expensive, and cheaply estimating memorization scores remains an open problem; the inverse-flow-based attack attempt failed; experiments are limited to vision datasets and 4 model architectures.
Experimental setupCNN32、CNN64 et al. · MNIST、SVHN et al. · AUROC、TPR @ 0.1% FPR et al.
Evaluated models
CNN32CNN64ResNet-9ResNet-18
Benchmarks
MNISTSVHNCIFAR-10CIFAR-100augMNISTaugCIFAR-10
Metrics
AUROCTPR @ 0.1% FPRprivacy scorememorization score
AI summaryThe researchers propose a membership inference attack (MIA) optimization method based on selecting samples by memorization; by using samples with high memorization scores, attackers can significantly improve attack effectiveness regardless of which MIA they use, and reduce the number of shadow models required by nearly two orders of magnitude compared with existing state-of-the-art methods. Membership inference attacks aim to identify specific samples in a machine learning model's private training set and pose serious privacy risks; many practical black-box MIAs require training hundreds or more shadow models to extract attack signals, incurring high computational cost. The paper's motivation experiments show that memorization is a key property explaining differences in samples' vulnerability to MIA; based on this, the authors provide a theoretical bound linking membership inference advantage with memorization, as well as a sample complexity bound connecting the number of shadow models required with memorization, and verify these experimentally.

The researchers propose a membership inference attack (MIA) optimization method based on selecting samples by memorization; by using samples with high memorization scores, attackers can significantly improve attack effectiveness regardless of which MIA they use, and reduce the number of shadow models required by nearly two orders of magnitude compared with existing state-of-the-art methods. Membership inference attacks aim to identify specific samples in a machine learning model's private training set and pose serious privacy risks; many practical black-box MIAs require training hundreds or more shadow models to extract attack signals, incurring high computational cost. The paper's motivation experiments show that memorization is a key property explaining differences in samples' vulnerability to MIA; based on this, the authors provide a theoretical bound linking membership inference advantage with memorization, as well as a sample complexity bound connecting the number of shadow models required with memorization, and verify these experimentally.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors state that label memorization is the key property explaining the variation in MIA success rates across samples, and that it can simultaneously improve attack effectiveness and efficiency.

    Core question: Which data property best explains the variation in MIA success rates across samples, and can an adversary exploit it to launch more effective and more computationally efficient attacks?

    • Problem setting: MIA aims to infer whether a record is in a model's private training set; the authors argue this leads to privacy leakage and is a foundation for understanding more complex privacy threats.
    • Shortcomings of existing methods: The authors state that many practical black-box MIAs require query access to the data distribution to train shadow models, often needing hundreds, which constitutes a computational bottleneck; and existing explanations based on overfitting, outliers, and under-represented subpopulations do not always hold (§4 shows that not all OOD samples are vulnerable).
    • Core observation: A motivational experiment (§4) shows that OOD scores correlate weakly with MIA susceptibility (absolute value below 0.4 or at most 0.551), whereas memorization scores correlate better with privacy score; the authors therefore argue that memorization is a more suitable explanation.
    • Paper contributions: The authors give a lower bound on MI advantage in terms of label memorization (Theorem 2), a result that shadow-model sample complexity is inversely proportional to memorization score (Corollary 1), and validate them on multiple vision datasets and models.
    • Threat model:
      • Goal: Infer whether sample z is in training set S, with success measured by high TPR at low FPR (0.1%) (following the setting of Carlini et al.).
      • Knowledge: The adversary knows the data distribution D and the learning algorithm L (but not the randomness of L).
      • Capability: Assuming the stronger M I^s game, the adversary also controls the training dataset S (the authors cite an auditor verifying data deletion as an example showing this assumption is not far-fetched); assuming a memorization oracle O_mem exists that returns an approximate memorization score.
      • Victim system: A machine learning model trained on S (vision models in the experiments).
  2. What related research exists?

    The paper organizes related work along three lines: MIA attacks, explanations of MIA differences, and memorization theory, and uses five MIAs as baselines.

    MIA attack methods (baselines in this paper)

    • Shokri et al. [46]—train an attack model with shadow models to determine membership; Sablayrolles et al. [44]—an attack based on scaled loss and per-sample thresholds; Song et al. [47]—use shadow models to approximate the entropy distribution for hypothesis testing.
    • Yeom et al. [53]—a simple attack that thresholds the loss, requiring no access to training data or distribution; Carlini et al. [10] (LiRA)—an attack from first principles, which the authors state is the best-performing attack at low FPR, and argue in §8 is equivalent to estimating label memorization plus hypothesis testing.
    • Baselines and benchmarks: all five attacks above are used as baselines and compared on mixed datasets of MNIST/SVHN/CIFAR-10/CIFAR-100 and their augmented variants.

    Explanations for differences in MIA success rates

    • Kulynych et al. [30]—propose disparate vulnerability, noting that differences in generalization ability across subgroups lead to differences in susceptibility; Yeom et al. [53]—study feature effects under black-box settings; Leino et al. [33]—exploit differences in model behavior on specific features under white-box settings.
    • Carlini et al. [11]—note that outlier samples are more vulnerable and demonstrate differences in attack performance on ID versus OOD samples; the authors consider these explanations relatively coarse, and this paper proposes memorization as the "key" property and gives the first formal characterization (per the authors).

    Memorization and hypothesis testing theory

    • Feldman [16], Feldman and Zhang [17], Brown et al. [8]—formalize label memorization, noting that overparameterized models need to memorize labels of atypical samples in order to generalize; this paper instantiates the memorization oracle using the algorithm of [17].
    • Ye et al. [52], Carlini et al. [10]—treat MIA as hypothesis testing; this paper follows this framework and maps the number of shadow models to the sample complexity of hypothesis testing.

    Positioning: The authors state that, unlike the above work, this paper treats memorization as the core property explaining differences in sample susceptibility to MIA, and gives the first formal lower bound of MI advantage in terms of memorization (per the authors).

  3. How does the paper address this problem?

    Formulates MIA as hypothesis testing, proving that MI advantage is lower-bounded by the memorization score and that the number of shadow models is inversely proportional to the memorization score.

    Overall approach: formalize MIA as a hypothesis testing problem distinguishing the two model distributions "trained with z" and "trained without z", then use label memorization to connect attack advantage with the number of shadow models.

    Problem formalization

    • Adopt the M I game: challenger C selects dataset S and trains a model, adversary A gives a challenge sample z and guesses its membership; advantage is defined as the difference between TPR and FPR (Equation 5). The paper assumes the stronger M I^s game, i.e., A also controls S.
    • Label memorization is defined as mem(L, S, zi) := Prθ ∼ L(S(i))[θ(xi) = yi] − Prθ ∼ L(S)[θ(xi) = yi], i.e., the difference in the probability that the model predicts its label before and after adding z_i (Equation 1).

    Theory: memorization lower bound on MI advantage

    • When the challenger controls the dataset (§5.1): A uses shadow models to estimate the parameters of P0 (z not in training set) and P1 (z in) (assuming multivariate normal), then uses the sLLR test; from Lemma 1, Theorem 1 follows: Adv(L, A) ≥ H²(P0, P1).
    • When the adversary controls the dataset (§5.2): taking attack A(θ, z) = 1{θ(x)=y}, the advantage simplifies exactly to mem(L, S, z), giving Theorem 2: Adv(L, A) ≥ mem(L, S, z). The authors note that this result depends on the existence of the memorization oracle O_mem, and list a realizable O_mem as future work.

    Theory: memorization and attack efficiency

    • The number of shadow models corresponds to the sample complexity of hypothesis testing; from Feldman's observation (memorization equals the TV distance between the distributions of the label indicator variable) and the TV–Hellinger inequality, H²(P0, P1) ≤ mem(L, S, z).
    • Substituting into Lemma 2 gives Corollary 1: MαP0,P1(T⋆) = Θ̃(1/mem(L,S,z)), i.e., the higher the memorization score, the fewer shadow models are needed to reach a given advantage.

    Experimental design

    • Data construction: D_tr is a mixture of an over-represented group D_Otr (MNIST 60,000 or CIFAR-10 50,000) and an under-represented group D_Utr at mixing ratio α; D_Utr has three choices—(a) random OOD samples, (b) subpopulations identified using the method of Jagielski et al. [24] (Efficientnet features + PCA down to 10 dimensions + KMeans into 5 clusters, taking the smallest cluster), (c) singletons with memorization score above a threshold (MNIST 0.8, CIFAR-10 0.5).
    • Memorization scores are estimated using the algorithm of Feldman and Zhang [17]; privacy score follows the definition of Carlini et al. [10] (Algorithm 1).
    • Evaluation: five MIAs, 2000 shadow models trained per setting; metrics are AUROC and TPR @ 0.1% FPR (threshold τ chosen to maximize TPR at that FPR).
  4. What experiments did the paper conduct?

    7 dataset combinations, 4 models, 5 MIAs: AUROC reaches 1.0 and TPR 100% on singletons, shadow models 2000→5.

    Experimental setup

    • Evaluated models: CNN32, CNN64 (taken from Carlini et al. [10]), ResNet-9, ResNet-18, 4 architectures in total.
    • Datasets: MNIST (60,000), SVHN (cropped version, 1,000), CIFAR-10 (6,000), CIFAR-100 (1,000), augMNIST and augCIFAR-10 (1,000 each, augmented using the method of Hendrycks et al. [22]); mixed settings in Table 1.
    • Baseline attacks: Yeom et al. [53], Shokri et al. [46], Sablayrolles et al. [44], Song et al. [47], Carlini et al. [10], five in total.
    • Metrics and judging: AUROC and TPR @ 0.1% FPR; privacy score and memorization score estimated per Algorithm 1, with no LLM judge.
    • Scale: 2000 shadow models trained per setting, 38,000 in total; accuracy reported with 95% confidence intervals over 2000 bootstrap iterations (Table 1).

    Main results

    According to Table 2 and Table 3 (Under-represented column, TPR is TPR @ 0.1% FPR):

    Swipe horizontally to read the table

    AttackDataset (D_tr)AUROC (random→singletons)TPR (random→singletons)
    Shokri et al.MNIST+SVHN0.76 → 0.990.30% → 72.86%
    Sablayrolles et al.MNIST+SVHN0.81 → 1.02.86% → 97.55%
    Carlini et al.MNIST+SVHN0.95 → 1.051.5% → 100%
    Carlini et al.MNIST+augMNIST0.87 → 1.027.55% → 100%
    Shokri et al.CIFAR-10+CIFAR-1000.85 → 1.00.28% → 100%
    Carlini et al.CIFAR-10+CIFAR-1000.96 → 1.054.42% → 100.0%
    Carlini et al.CIFAR-10+augCIFAR-100.93 → 1.036.73% → 98.01%
    • The authors state that every attack performs better on under-represented group samples than on random samples, with the largest improvement when using OOD singletons.
    • Exception: Yeom et al. on CIFAR-10+CIFAR-100, where subpopulation (AUROC 0.85, TPR 17.32%) outperforms singletons (0.76, 0.0%); the authors' explanation is that in this mixed dataset all samples have memorization scores below 0.6, so samples selected by the 0.5 threshold are not truly high-memorization (Figure 6).
    • The authors state that the attack of Carlini et al. outperforms other attacks in all scenarios, and attribute this to the attack's intrinsic connection to memorization estimation (§8).

    Computational cost experiment (§7.2)

    • What was measured: AUROC of the Carlini et al. attack as a function of the number of shadow models (Figure 3, Figure 4), with challenge samples drawn from the under-represented group.
    • Results: With high-memorization OOD data, 5 shadow models suffice to reach the performance of 2000 (the authors state a 400× reduction); in Figure 3d, 1 shadow model each for in/out reaches AUROC 1.0. With random OOD data, fewer than 50 shadow models reach the AUROC of 2000 (the authors state a 40× reduction).
    • Authors' interpretation: Even without a reliable memorization oracle, random OOD data can reduce cost; the authors acknowledge that the latter observation is purely empirical with no theoretical support.

    DP defenses and failed attempts (§8)

    • DP-SGD: Trained on MNIST+SVHN (same 1,000 SVHN images, 70% subsampling) with δ=10⁻⁵, noise multiplier 1.3, ℓ2 clipping norm 1.0, 100 epochs, ε=3.2; Figure 4 shows that under DP-SGD the memorization scores of under-represented samples also become lower, which the authors state is consistent with Feldman [16] that "DP impedes memorization"; the authors also note low model utility at this privacy budget.
    • Failed inverse-flow experiment: The authors trained a flow model on CIFAR-10, attempting to generate x′ by perturbing the latent vector and measure its accuracy to determine whether z was memorized; the authors report this attempt failed because the true label of the modified input x′ could not be determined.

    Other ablations and analyses

    • The absolute Spearman correlation between OOD scores and privacy score is below 0.4 or at most 0.551 (Figure 1b, p<0.001); the authors state the correlation is not strong.
    • The correlation between memorization score and privacy score is better (Figure 2a; specific values not given in the main text).
    • Memorization score has no correlation with OOD detector scores (Hendrycks et al. [21], Liu et al. [35]) (Figure 9, Figure 10); the authors state OOD scores cannot serve as a proxy for memorization.
    • In single-dataset settings (e.g., pure MNIST/CIFAR-10), the "Under-represented" column reports samples with memorization scores above 0.4 (MNIST)/0.8 (CIFAR-10), and attacks are likewise more effective on these samples (first rows of Table 2, Table 3).
    • Full results for the MNIST+CIFAR-10 mix are in Table 5: Carlini et al. achieves AUROC 1.0 and TPR 96.51% on singletons.
  5. What are some points that can be further explored?

    The authors list open directions such as instantiating the memorization oracle and inverse-flow; experiments are limited to vision datasets and 4 architectures.

    Limitations and future directions noted by the authors

    • A realizable and effective memorization oracle O_mem is important future work (footnote to Theorem 2 in §5.2); instantiating it with the Feldman and Zhang [17] algorithm is computationally expensive, and designing a cheap mechanism for estimating memorization scores remains an open problem (§8).
    • The MIA attempt based on an inverse-flow model failed; the authors consider this direction promising but requiring more experiments before it can be combined with memorization for MIA (§8).
    • Defenders removing high-memorization samples face two problems: model error increases on the corresponding low-density subset, and new high-memorization samples appear (privacy onion effect, §8).
    • More analysis is needed to convert the mutual-information results of Brown et al. [8] into probability bounds, and to understand MIA's effectiveness on samples that are not easily memorized (§9).
    • The observation that random OOD data can also reduce the number of shadow models is purely empirical with no theoretical support (§7.2).

    Scope of experimental coverage

    • The evaluated models are CNN32, CNN64, ResNet-9, ResNet-18, 4 architectures in total, all vision models; the authors state the focus on vision is because label memorization is more maturely studied in this field (§6).
    • The datasets are mixtures of MNIST, SVHN, CIFAR-10, CIFAR-100 and two augmented variants, 7 combinations in total; the baselines are 5 MIA methods (§6).
    • 2000 shadow models are trained per setting, 38,000 in total; accuracy is reported with 95% confidence intervals over 2000 bootstrap iterations (§6, Table 1).
    • The defense evaluation includes only one setting, DP-SGD (ε=3.2), and the authors note low model utility at this budget (§8).
    • The paper does not report specific numbers for attack runtime or per-query cost.
  6. Summarize the main content of the paper

    The authors state that memorization explains the variation in MIA success rates: AUROC 1.0 and TPR 100% on singletons, shadow models 2000→5.

    An attack-class paper on membership inference attacks (MIA) that proposes using label memorization to explain and amplify susceptibility differences across samples.

    • Problem: MIA success rates vary greatly across samples, and existing explanations (overfitting, OOD, under-represented subpopulations) do not always hold; moreover, black-box MIAs often require hundreds of shadow models, incurring high computational cost.
    • Method: Formalize MIA as hypothesis testing, prove that MI advantage is lower-bounded by the memorization score (Theorem 2) and that the number of required shadow models is inversely proportional to the memorization score (Corollary 1); the attacker uses a memorization oracle to select highly memorized samples (singletons) to launch attacks.
    • Results: On 7 dataset combinations, 4 models, and 5 MIAs, the Carlini et al. attack achieves AUROC 1.0 on singletons, with TPR @ 0.1% FPR reaching 100% on MNIST+SVHN and MNIST+augMNIST (Table 2); Sablayrolles et al. on MNIST+SVHN singletons raises TPR from 2.86% (random) to 97.55%.
    • Efficiency: Under highly memorized data, 5 shadow models reach the AUROC of 2000 (the authors state a 400× reduction); under random OOD data, fewer than 50 reach the same level (the authors state 40×).
    • Exception: Yeom et al. on CIFAR-10+CIFAR-100 performs better on subpopulation than on singletons, which the authors attribute to sample memorization scores in that mixture generally being below 0.6.
    • Authors' conclusion: Memorization characterizes MIA susceptibility more accurately than OOD; sample memorization scores decrease under DP-SGD; cheaply estimating memorization scores, inverse-flow attacks, and the defensive implications of the privacy onion effect are left as open questions.

Related papers

Show 3 more papers
Read originalarxiv.org