Are membership inference attacks effective against large language models?

1. 01What problem does this paper attempt to solve?

   The authors systematically evaluate whether five membership inference attacks are effective on LLM pretraining data, finding that they are close to random in most settings.

   Core question: Can existing membership inference attacks (MIA) effectively infer whether a given data point belongs to an LLM's pretraining data?

   • Setting and importance: MIA is the de-facto threat model for discussing machine learning privacy, and can be used for privacy auditing, memorization research, copyright infringement, and test set contamination detection (§1).
   • Existing research gap: The authors state that previous high attack performance of MIA was mostly limited to classifiers or LM fine-tuning settings, and MIA performance on LLMs and their pretraining data is largely unexplored (§1).
   • Core observation: The authors believe that the characteristics of LLM pretraining (massive data, near single-epoch training) and the naturally high n-gram overlap between members and non-members may blur membership boundaries, thereby reducing MIA performance (§3.2).
   • Paper output: Release of MIMIR—a unified MIA evaluation repository implementing five attacks from the literature; large-scale evaluation on the Pythia suite (160M–12B, trained on the Pile), and analysis of the reasons for poor attack performance and the impact of candidate set selection (§1).
   • Threat model:
     • Goal: Infer whether a given data point x is in the training set D of the target model M, by thresholding the membership score f(x; M) (§2).
     • Knowledge: blackbox access (next-token probabilities from model output); reference-based attacks additionally assume a reference model; the authors explicitly do not adopt settings that require training a meta-classifier on known member/non-member subsets (Appendix A.4).
     • Capability: Can query target model probabilities; the Neighborhood attack requires a masking model (BERT).
     • Victim system: Autoregressive LMs pretrained on the Pile (Pythia, Pythia-Dedup, etc.).

   Collapse this answer

2. 02What related research exists?

   Related work covers MIA attacks and defenses, memorization and data extraction, and LLM pretraining data detection work that reports high AUC.

   MIA Attacks and Defenses

   • Shokri et al. (2017)—proposed membership inference attacks against machine learning models and the shadow model method; Yeom et al. (2018) proposed a loss-based attack and linked privacy risk to overfitting.
   • Carlini et al. (2022) proposed LiRA—the authors state that such methods require training multiple shadow models, which is often infeasible for LLMs; Sablayrolles et al. (2019) and Watson et al. (2022) proposed reference-based calibration.
   • On defenses, the paper mentions DP-SGD (Abadi et al., 2016), self-distillation ensembles (Tang et al., 2022), and RelaxLoss (Chen et al., 2022), listed only as background.

   MIA Against Language Models

   • Mireshghallah et al. (2022a; 2022b), Fu et al. (2023)—MIA against masked LM pretraining or fine-tuning; the authors note that these settings train for more than 10 epochs, unlike the near single-epoch pretraining of LLMs (§3.2.1).
   • Mattern et al. (2023) proposed the Neighborhood attack; Shi et al. (2023) proposed Min-k% Prob and reported over .7 AUC on pretrained LLMs—the authors analyze in §4 specifically how its non-member selection (split by knowledge cutoff) may introduce temporal distribution shift.
   • Meeus et al. (2023) reported document-level MIA over .7 AUC on pretrained LLMs, which the authors likewise take as an object of time-shift analysis.

   Memorization and Data Extraction

   • Carlini et al. (2021; 2023), Biderman et al. (2023a), Tirumala et al. (2022)—measure memorization by extractability; the authors note that MIA and extraction rely on different assumptions and reveal different types of leakage (§2).
   • Kandpal et al. (2022) studied the effect of training data deduplication on extractability; Kandpal et al. (2023) studied user-level leakage in the fine-tuning setting.

   Baselines and Benchmarks: the five attacks evaluated in this paper are the baselines—LOSS (Yeom et al., 2018), Reference-based (Carlini et al., 2021; implementation follows Watson et al., 2022), Zlib Entropy (Carlini et al., 2021), Neighborhood (Mattern et al., 2023), Min-k% Prob (Shi et al., 2023); the evaluation benchmarks are seven Pile subdomains and the full Pile, plus C4 (Datablations) and Dolma (OLMo).

   The authors position this paper as: a large-scale evaluation of the performance of existing MIAs on LLM pretraining data under a unified blackbox setting, noting that the high performance reported in prior work may stem from temporal distribution shift in the candidate set (§4).

   Collapse this answer

3. 03How does the paper address this problem?

   The authors build the unified MIMIR benchmark, run five MIAs on models such as Pythia across Pile domains, and analyze along two lines: training characteristics and membership fuzziness.

   Overall approach: using a unified implementation and a unified benchmark, evaluate five MIAs under controlled member/non-member sources, then locate the factors affecting attack performance by varying training data amount, number of epochs, non-member overlap and time range. The framework is named MIMIR.

   Problem Setting and Attack Implementation

   • The target model M is an autoregressive LM; the attack computes a membership score f(x; M) and thresholds it to make a decision (§2).
   • Five attacks (Appendix A.4): LOSS directly takes the sample loss; Reference-based calibrates with a reference model, f(x; M) = L(x; M) − L(x; M_ref), i.e., the difference between the target model and reference model losses; Zlib normalizes the loss by the sample's zlib compression length; Neighborhood uses BERT to generate neighbor samples with a 5% masking ratio and compares losses; Min-k% Prob takes the average negative log-likelihood of the k% tokens with the lowest likelihood, with k = 20 in the experiments.
   • The reference model is uniformly StableLM-Base-Alpha-3B-v2 (selected based on the ablation in Appendix A.5).

   Benchmark Construction

   • Sample 1,000 members (train set) and 1,000 non-members (test set) per Pile subdomain; for the full Pile experiments, 10,000 each (Appendix A.3).
   • Samples are documents longer than 100 words, truncated from the beginning to 200 words; on top of the existing document-level decontamination in Pile, additional deduplication is done with the bloom filter of Groeneveld et al. (2023) (n = 13, threshold ≤ 80% overlap).
   • Metrics: mainly report AUC ROC, and also TPR@low%FPR; averaged over 1,000 bootstraps, with shaded 95% confidence intervals in the figures (§3, Appendix A.4).

   Training Characteristic Analysis

   • Training data amount: use intermediate checkpoints of Pythia-Dedup (one every 5000 steps, plus step1000 and step99000), with members taken from documents seen within the most recent 100 steps of each checkpoint and non-members fixed (Appendix A.3.1).
   • Number of training epochs: attack the Datablations suite (2.8B, 55B tokens, 1–14 epochs) and SILO-PDSW, and observe the relationship between effective epoch count and AUC (§3.2.1, Appendix A.2).

   Member Ambiguity Analysis

   • n-gram overlap: the n-gram overlap of a non-member x with the training set D is defined as 1/(m−n+1)∑_i=1^m−n+1 1{∃ y ∈ D : x_i...x_i+n−1 ∈ y}, i.e., the proportion of n-grams in the non-member that appear in any member document (§3.2.2).
   • Overlap threshold experiment: resample non-members with 7-gram overlap ≤ 20% and compare the change in attack performance (Table 2).
   • Time shift experiment: non-members are replaced with Wikipedia (RealTimeData WikiText, from August 2023) and ArXiv (monthly from 2020-08 to 2023-06) samples after the knowledge cutoff (§4, Appendix A.3.2).

   Revisiting the Member Definition

   • Lexical distance: randomly replace n ∈ {1, 10, 25} tokens in members (20 trials per n), and observe the MIA score and FPR (§5).
   • Semantic distance: use semantic neighbors generated by the Neighborhood attack, as well as GPT-4 rewritten paraphrases (5 trials per member), as modified members, and test whether they are judged as non-members (§5, Appendix D).

   Collapse this answer

4. 04What experiments did the paper conduct?

   The five MIAs have AUC below 0.6 on most Pile domains; more training data and deduplication lower performance, while more epochs and time shift raise it.

   Experimental setup

   • Evaluated models: Pythia-Dedup (70M/160M/1.4B/2.8B/6.9B/12B), Pythia (same specifications, except 70M), GPT-Neo (125M/1.3B/2.7B), OLMo (1B/7B), Datablations (2.8B, 1–14 epochs), SILO-PDSW (1.3B).
   • Data: 1,000 members + 1,000 non-members for each of seven Pile subdomains, 10,000 each for the full Pile; samples truncated to 200 words (Appendix A.3).
   • Baselines/attacks: LOSS, Reference-based, Zlib, Neighborhood, Min-k% Prob (k = 20).
   • Metrics and evaluation: AUC ROC and TPR@1%FPR, averaged over 1,000 bootstrap iterations; no LLM judge, decisions are score thresholding.
   • Reference model: StableLM-Base-Alpha-3B-v2; the masking model for Neighborhood is BERT.

   Main results

   According to Table 1 (Pythia-Dedup, AUC ROC, excerpting the 12B and 160M rows; the full table includes 70M/1.4B/2.8B/6.9B):

   Swipe horizontally to read the table

   Domain	Model	LOSS	Ref	min-k	zlib	Ne
   Wikipedia	12B	.516	.579	.517	.524	.520
   Github	12B	.678	.559	.683	.690	.660
   Pile CC	12B	.516	.582	.521	.517	.514
   PubMed Central	12B	.506	.559	.512	.506	.497
   ArXiv	12B	.527	.555	.530	.521	.519
   HackerNews	12B	.518	.565	.533	.512	.515
   Wikipedia	160M	.504	.515	.488	.514	.513
   Github	160M	.638	.591	.634	.656	.638

   • The authors state that except for the GitHub domain, no MIA or target model has an AUC above 0.6, and most domains are close to random; the GitHub exception is discussed in Appendix B.3.
   • The authors state that the reference-based attack is generally the best, but under individual settings other attacks are better (e.g., Min-k% Prob on Pile CC at 160M), making it difficult to select an overall best attack.
   • The authors state that MIA performance increases with model size and decreases with training data deduplication (Figure 1); the trend is the same on non-deduplicated Pythia with slightly higher values (Table 12, e.g., 12B Github zlib is .748).

   Training data size and number of epochs

   • What was tested: Reference-based attack on Pythia-Dedup intermediate checkpoints (training data size) and Datablations (number of epochs).
   • Results: Figure 2 left—AUC rises within the first few thousand steps, then gradually declines as data size increases; the authors state that the 2.8B trajectory remains close to random throughout; Figure 2 right—on Datablations, AUC rises linearly with the effective number of epochs (1–14), with the vertical axis ranging approximately 0.5–0.9.
   • Authors' interpretation: The authors speculate that the early rise is due to a small data/parameter ratio and the model tending to overfit, followed by improved generalization; multi-epoch training may increase training data leakage.

   n-gram overlap and temporal shift

   • Overlap threshold: After restricting non-members to 7-gram overlap ≤ 20% (Table 2, Pythia-Dedup-12B), zlib on Github rises from .690 to .908, LOSS on PubMed Central rises from .506 to .780, and on Wikipedia rises from .516 to .666; the authors note that this introduces member/non-member distribution drift and is not recommended for benchmark construction.
   • Temporal shift: Table 3—on time-shifted Wikipedia, the Ref attack AUC for 12B reaches .796 (LOSS .680, min-k .719, zlib .607, Ne .626); Figure 6 shows that for ArXiv non-members, the later the time, the higher the AUC (Appendix B.4 gives .723 for 2020-08 to .795 for 2023-06). Natural Wikipedia non-members have an average 7-gram overlap of 39.3%, which is 13.9% after temporal shift (§4).
   • Authors' interpretation: The authors believe that the high AUC reported in prior work (Shi et al., 2023; Meeus et al., 2023) may stem from this temporal distribution shift, but because their setups cannot be reproduced, it cannot be concluded that their differences are entirely caused by temporal shift (§4, Appendix A.3.2).

   Modified-member experiments

   • Lexical modification: Table 4 left—at edit distance n = 1, the Ref attack's FPR on modified members at 1%/5%/10% FPR thresholds is 0.1%/0.3%/0.7% (ArXiv) and 0.0%/0.2%/0.6% (Wikipedia); the LOSS attack has an FPR of 0 under all settings.
   • Semantic neighbors: Table 4 right—at the 10% FPR threshold, LOSS's FPR on semantic neighbors is 2.5% (ArXiv)/2.3% (Wikipedia), and Ref is 4.0%/8.2%.
   • GPT-4 rewriting: Table 10—at the 1% FPR threshold, the FPR of LOSS and Ref on paraphrase members is no higher than 0.2% (three domains). The authors believe these samples are actually members from an information leakage perspective but are classified as non-members, requiring rethinking of the membership definition.

   Other ablations and analyses

   • Reference model choice (Table 6): StableLM is generally the best, and aggregating all reference models still performs poorly.
   • GPT-Neo (Table 7): The trend is the same as Pythia, with AUC < .55 in most domains.
   • OLMo (Table 8): Both 1B and 7B are close to random, with some settings below .5 (e.g., 1B Ref on s2ag is .376).
   • TPR@1%FPR (Table 11): Most settings are below 3%, with GitHub as an exception (up to 23.4%).
   • Member recency (Figure 9): Recently seen members have slightly higher AUC, which drops to a plateau as training progresses.
   • SILO-PDSW (Figure 10): On HackerNews, AUC first rises then plateaus or declines; on DM Math, it declines with the number of epochs; the authors state that further research is needed.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that phenomena such as the inability to reproduce prior work, GitHub outliers, and OLMo falling below random remain to be resolved, and suggest studying fuzzy membership definitions.

   Limitations noted by the authors and future directions

   • Unable to reproduce the experimental setups of Shi et al. (2023) and Meeus et al. (2023) for distribution analysis (their non-member candidates or reproduction procedures are unavailable, and LLaMA does not disclose its training data), so it cannot be concluded that their result differences are entirely caused by temporal shift (§4, Appendix A.3.2).
   • On OLMo, multiple settings have AUC below .5 (the attack is more likely to classify members as non-members); the authors state that more research is needed to understand this behavior on domains such as peS2o and s2ag (Appendix A.7).
   • On SILO-PDSW, the AUC plateau/decline for HackerNews and the decline for DM Math; the authors state that further research on the target domains and attack settings is needed (Appendix C.2).
   • The authors encourage future work to study membership definitions that account for fuzzy members (lexically/semantically close to members), to better understand privacy leakage (§1, §5).
   • The authors believe the membership inference game needs to be extended to generative models, aligning with audit goals such as user-level leakage and PII; when constructing MIA benchmarks, unconscious distribution shift of non-members should be avoided (§6).
   • The authors suggest that baselines for new MIAs should include both calibration-based and non-calibration-based attacks (Appendix B.2).

   Experimental coverage

   • The evaluated models are six suites: Pythia, Pythia-Dedup, GPT-Neo, OLMo, Datablations, and SILO-PDSW, with a maximum of 12B parameters (§3, Appendix A.2).
   • The five evaluated attacks are all blackbox score-threshold-based; attacks that require training a meta-classifier on a known subset of members are not included (Appendix A.4).
   • The sample size per domain is 1,000 members + 1,000 non-members (10,000 each for the full Pile), with 1,000 bootstrap iterations (Appendix A.3, A.4).
   • The paper does not report the number of queries or computational cost for each experiment; the GPU is only described as a mix from RTX6k to A100 (Appendix A.1).
   • Sample length is limited to 100–200 words; the authors note that longer samples may improve MIA performance, but that experiment is orthogonal to the question in this paper (Appendix A.3).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors' systematic evaluation finds that existing MIAs on LLM pretraining data are close to random, and attributes the cause to training characteristics and fuzzy membership boundaries.

   A large-scale evaluation study on the effectiveness of membership inference attacks (MIA) on LLM pretraining data, which also releases a unified benchmark package MIMIR.

   • Problem: MIA performs well on classifiers and in fine-tuning settings, but there is a lack of systematic research on whether it is effective on LLM pretraining data.
   • Method: uniformly implement five attacks—LOSS, Reference-based, Zlib, Neighborhood, Min-k% Prob—and attack models such as Pythia/Pythia-Dedup (160M–12B) on seven Pile subdomains and the full Pile, with 1,000 members vs 1,000 non-members per domain, reporting AUC ROC.
   • Main results: the authors report that except for the GitHub domain, AUC is below 0.6 in all settings, mostly close to random (Table 1); GitHub is the exception (12B zlib is .690), which the authors attribute to outlier non-members selected by the decontamination threshold (Appendix B.3).
   • Cause analysis: more training data and deduplication lower attack performance, while more epochs raise it linearly (Figure 2); the natural 7-gram overlap between non-members and members is high (Wikipedia 32.5%, ArXiv 39.3%, GitHub 76.9%), and after limiting overlap to ≤ 20%, AUC rises (e.g., Github zlib to .908, Table 2).
   • Candidate set effect: time-shifted non-members split by knowledge cutoff raise the Ref attack AUC on 12B to .796 (Table 3), with an average 7-gram overlap of only 13.9%; the authors believe the high AUC reported in prior work may stem from such distribution shift.
   • Member definition: modified members with only 1 token replaced are almost all judged as non-members at the 1% FPR threshold (FPR ≤ 0.1%, Table 4); the authors believe the exact-match member definition may miss leaks that auditors care about.
   • Authors' conclusion: the poor performance of MIA on LLMs may stem from both pretraining characteristics (data leaves no imprint) and the blurry member/non-member boundary; the authors recommend avoiding distribution shift when constructing benchmarks and extending the membership inference game to fuzzy member definitions.

   Collapse this answer

Related papers