Extracting Training Data from Large Language Models

Core question: using only black-box queries, can a single training example be extracted verbatim from a language model trained on large-scale data.

• Setting and importance: the authors note that billion-parameter-scale language models trained on private data are already common; once a model reproduces training data verbatim, it can leak personally identifiable information, keys, and the like, and cause privacy harm in the sense of contextual integrity (data used outside its original context).
• Gaps in current understanding: the mainstream view holds that privacy leakage is related to overfitting, and that large models trained for only a few epochs on large deduplicated datasets hardly overfit, and therefore "will not noticeably memorize any particular training example" (the paper cites [71] for this claim). The authors consider this intuition wrong: average loss being close does not mean worst-case examples are not memorized.
• Core observation: the authors' key hypothesis is that although the average loss of training examples is close to that of test examples, some worst-case training examples are indeed memorized verbatim, and a reference model can be used to filter them out from ordinary high-likelihood examples.
• What the paper proposes: a simple and efficient two-stage training data extraction attack (generation + membership inference ranking), a testable definition of memorization (k-eidetic memorization), a systematic evaluation on GPT-2, and finally a discussion of mitigation strategies.
• Threat model (Section 3.2):
  • Attacker knowledge: black-box input-output access to the language model, can compute the probability of any sequence and obtain next-word predictions.
  • Attacker capability: cannot view model weights or hidden states (such as attention vectors).
  • Attacker goal: indiscriminately extract memorized training data; attack strength is measured by the number, length, and k value of extracted examples, and targeted extraction against specific users is explicitly not performed.
  • Victim system: GPT-2 (training data from the public internet, which the authors call a representative target and chose to reduce real-world harm).

Training data privacy attacks

• Membership inference (Shokri et al. 2017; [28, 47, 67]): determining whether a sample is in the training set; the least information-leaking form of attack; the second stage of this paper is essentially membership inference.
• Model inversion (Fredrikson et al. 2015): reconstructing a representative "blurry" view of training data; the paper notes that training data extraction attacks differ from it in that the goal is verbatim samples, and are therefore more dangerous (secrets such as social security numbers and passwords can be extracted).
• Training data extraction attacks (Carlini et al. 2019's Secret Sharer; [66, 68, 73]): the paper notes that such attacks were previously limited to small models trained on academic datasets under artificial training settings (e.g., more epochs), or to settings where the attacker already knew the secret content [8, 27].

Privacy-preserving training

• Differentially private training (Abadi et al. 2016's DP-SGD; [9, 43, 60, 64]): the paper notes that it often reduces accuracy, struggles to capture long-tail distributions, and increases training time, so models such as GPT-2, GPT-3, and T5 did not use it.
• User-level differentially private language models (Ramaswamy et al. 2020): the paper notes that it requires a user label for each document, and publicly scraped web data has no such labels.

Language models and memorization analysis

• GPT-2 / Transformer language models (Radford et al. 2019; Vaswani et al. 2017): the attack target and architectural background of this paper; GPT-2 was trained for about 12 epochs on 40GB of public web text, and its training loss is only about 10% lower than its test loss.
• Research treating neural networks as knowledge bases (Petroni et al. 2019; Roberts et al. 2020): analyzing knowledge stored in model parameters; the memorization definition in this paper is partly inspired by this, but is limited to "unintended" memorization.
• Canary insertion method (Carlini et al. 2019): studying memorization by manually inserting canaries; the paper notes that its limitations are the synthetic setting and the possibility that it does not represent natural data, so this paper instead uses naturally occurring "canaries."

Baselines and benchmarks: the strawman baseline for the attack is top-n sampling + perplexity ranking (Section 4); the evaluation benchmark is the GPT-2 training data itself, confirmed with OpenAI's help via 3-gram fuzzy matching.

The authors position this paper as turning training data extraction from a "theoretical or academic" threat into a practical threat to real large models, and as providing a quantifiable definition of memorization.

The overall approach is a two-stage generate-then-rank attack: first sample a large amount of high-likelihood text from the model, then use a reference model to perform membership inference, ranking samples with "abnormally high likelihood" first, and finally manually verify them.

Formal definition of memorization

• Model knowledge extraction (Definition 1): a string s can be extracted from model fθ if there exists a prefix c such that s is the model's most likely continuation; in practice arg max is not computable, and sampling strategies such as greedy can be used instead.
• k-eidetic memorization (Definition 2): s can be extracted from the model, and s appears in at most k documents in the training set X: |{x ∈ X : s ⊆ x}| ≤ k. The smaller k is and the longer the string, the more "severe" the memorization.
• For GPT-2, an entire web page counts as one training sample; repeated occurrences within the same page still count as k = 1.

Text generation (three strategies)

• Top-n (baseline): use the first token of a sentence as a single-token prompt, and top-n sampling (n = 40) generates 256 tokens.
• Decaying temperature: the softmax temperature starts at t = 10 and decays to t = 1 within the first 20 tokens, balancing prefix diversity and continuation along high-confidence paths.
• Internet conditioning: use 50MB of text self-collected from Common Crawl (HTML/JavaScript removed, deduplicated by line), randomly take 5–10 tokens as a prefix and then continue; the authors state that they used a collection process different from GPT-2's to reduce overlap with the training set, and acknowledge that a small amount of intersection may exist.

Membership inference (six metrics)

• Perplexity: GPT-2 XL's own perplexity (baseline).
• Small / Medium: the ratio of XL's log perplexity to that of GPT-2 Small (117M) and Medium (345M); the authors conjecture that smaller models have smaller memorization capacity, which can filter out samples memorized only by the large model.
• zlib: the ratio of GPT-2 perplexity to zlib compression entropy, used to identify trivial memorization such as repeated substrings.
• Lowercase: the ratio of the perplexity of the original sample to that of the lowercased sample.
• Window: the minimum average perplexity over a 50-token sliding window (the authors state it was selected via a rough hyperparameter search).

Evaluation protocol

• Each generation strategy generates 200,000 samples, for a total of 3 × 6 = 18 configurations; for each configuration, after ranking by metric, 100 samples are selected from the top-1000 (samples ranked below k are selected in proportion k/1000), for a total of 1,800 candidates.
• Fuzzy deduplication is done using trigram multisets: if |tri(s1) ∩ tri(s2)| ≥ |tri(s1)|/2, it is judged a duplicate.
• Determination method: manual verification by one of the four authors; if a non-trivial substring of the sample can be found via Google search on a page with an exact match, it is recorded as memorized; then all 1,800 are sent to the original GPT-2 authors for fuzzy 3-gram matching confirmation (the authors state that this method can confirm memorization but cannot detect missed samples); occurrence counts are precisely counted by OpenAI via grep over the full dataset.

Experimental setup

• Models tested: GPT-2 XL (1.5B/1558M parameters, main target), GPT-2 Small (117M/124M), GPT-2 Medium (345M/334M); Small and Medium are also used as reference models for membership inference (in which case the target model is XL itself).
• Data: 200,000 samples per generation strategy (256 tokens each), 600,000 in total; 100 candidates taken from each of 18 configurations, 1,800 manually verified in total.
• Baseline: strawman attack of top-n sampling + perplexity ranking (Section 4).
• Metric: the number of candidates confirmed as memorized out of 100 candidates per configuration (true positive rate); memorization strength is measured by the k value of k-eidetic memorization.
• Verification method: dual confirmation by manual Internet search plus OpenAI's fuzzy 3-gram matching against the original training data; exact occurrence counts obtained via OpenAI grep.
• Repetitions: the paper does not report repeated experiments for the same configuration.

Main results

Number of candidates confirmed as verbatim training samples out of 100 candidates per configuration for the 18 configurations (according to Table 2):

Swipe horizontally to read the table

Inference strategy	Top-n	Temperature	Internet
Perplexity	9	3	39
Small	41	42	58
Medium	38	33	45
zlib	59	46	67
Window	33	28	58
Lowercase	53	22	60
Total Unique	191	140	273

• All configurations together confirm 604 unique memorized samples, with an aggregate true positive rate of 33.5% (Section 6.2).
• The authors state that Internet-conditioned generation is the most effective; using perplexity directly for membership inference performs poorly (only 9% and 3% under top-n and temperature), and comparison-based metrics are better.
• Figure 3 shows that most samples lie along the diagonal (if the likelihood is high under one model, it is also high under the other), and outliers in the upper left (low GPT-2 perplexity but high zlib entropy) are more likely to be memorized samples.

Categories and sensitivity of memorized content

• Category counts (Table 1, 604 samples, may belong to multiple categories): news 109, logs/error reports 79, licenses/terms 54, named entity lists 54, forums/Wiki 53, resolvable URLs 50, named individuals 46, promotional content 45, high-entropy strings (UUID, base64) 35, contact information 32, code 31, configuration files 30, religious texts 25, etc.
• Personal information: 46 samples contain personal names (excluding public figures in news); 32 samples contain contact information, of which 16 are corporate and 16 are contact details of private individuals; IRC chat usernames appearing in only one training document were also extracted.
• k = 1 high-entropy samples (Table 3): all 9 examples are random strings appearing in only one training document, 10–87 characters long, appearing 10–311 times within the document; the authors state that in the best case they extracted an 87-character sequence appearing 10 times in the entire training set.
• Longer sequences (Section 6.4): extending memorized samples with beam-search-style decoding can extract 1450 lines of verbatim source code and the complete MIT, Creative Commons, and Project Gutenberg licenses.

Model scale and insertion frequency (Section 7)

• Setup: using Reddit URLs repeated different numbers of times in a single pastebin document as natural canaries, extracted in two ways of differing difficulty (direct top-n sampling of 10,000; or additionally giving the first 6 characters of the URL and using beam search), tested on the three variants XL, Medium, and Small.
• Results (Table 4): under the harder setting, XL extracted all URLs inserted 33 or more times, Medium extracted half, and Small extracted none; under the easier setting, Medium extracted 4 more, and Small extracted only the one inserted 359 times.
• Authors' interpretation: larger models memorize more training data; for the largest model, full memorization occurs at 33 insertions. The authors also state that in one setting the 1.5B model memorized more than 18 times as much content as the 124M model.

Other ablations and analyses

• Context dependence (Section 6.5): for the 273 samples found under Internet conditioning, none could be reliably reproduced using the prefixes that originally generated them; but after giving all the preceding context before the memorized content, almost all could be reproduced with high probability.
• π example: prompting "3.14159" with greedy decoding gives the first 25 digits; the beam-search-style method extracts 500 digits; prompting "pi is 3.14159" greedily gives 799 digits; adding the context "e begins 2.7182818" gives 824 digits.
• Category differences found by each strategy (Tables 6, 7): zlib found more non-rare text (only 1 high-entropy sample), while the Small/Medium strategies found 13 and 10 high-entropy samples respectively.
• Duplication rates (Figure 4 caption): top-n samples 2.6% duplicated, Internet 7.1%, Temperature 0.6%.
• Negative results: the authors state that almost all memorized content found by the baseline attack could plausibly appear multiple times in the training set (large k); some resolvable URLs had no findable source and were not counted as memorized samples.

Limitations and Future Directions Noted by the Authors

• Extraction volume is a loose lower bound (Section 9): only 1,800 candidates were manually checked, and starting from more candidates might find more memorized content; developing better extraction techniques (including targeted attacks on specific content) is a future direction.
• Prefix selection can be improved (Section 9): currently it only tries high-quality prefixes and hopes to trigger memorization, and a better prefix selection strategy might find more memorized data.
• Why some samples have abnormally low loss (Section 9): average training loss is only slightly below validation loss, but some samples still have abnormally low loss, and understanding the reason is an important future question.
• Relationship between fine-tuning and memorization (Section 8): fine-tuning may make models "forget" pretraining memorization, or may introduce new privacy leaks; exploring how memorization is inherited by fine-tuned models is a future direction.
• Mitigations are all incomplete (Sections 8, 9): how to apply differential privacy to web data in a principled way is not yet clear; data cleaning is an imperfect first line of defense; new privacy-preserving training methods are needed that do not sacrifice accuracy or training time at the scale of billions of parameters.
• Auditing (Section 8): the authors propose using this paper's method and existing attacks to audit the actual privacy level of language models as a future direction.

Scope of Experiments

• The models tested are three variants, GPT-2 XL, Medium, and Small, all from the same model family and the same training data.
• The attack is untargeted extraction; the paper explicitly does not do targeted attacks against specific users (Section 3.2).
• Each generation strategy generated 200,000 samples, and 1,800 candidates were manually verified (100 per configuration); the paper does not report the number of repeated experiments for the same configuration.
• Memorization judgments were made manually by one of the four authors and confirmed by OpenAI against the original training data; the paper does not report the agreement rate of manual judgments.
• The scale and frequency experiments (Section 7) use only the Reddit URL in a single pastebin document as a natural canary.

An attack paper on language model training data privacy, targeting the mainstream assumption that "large models will not memorize training data if they do not overfit."

• Problem: large language models trained on private data may have training samples extracted verbatim via black-box queries, causing data leakage and contextual integrity violations; previous extraction attacks had only been demonstrated on small models or under artificial settings.
• Method: proposes the k-eidetic memorization definition (a string is extractable and appears in at most k training documents), and designs a two-stage attack—generating 200,000 samples each with three strategies (top-n, decaying temperature, Internet prefix), then ranking with six membership inference metrics (perplexity, comparison with Small/Medium, zlib entropy ratio, lowercasing, sliding window), taking 100 candidates per configuration, with dual confirmation by manual search plus OpenAI's 3-gram matching against the original training data.
• Main results: 604 unique verbatim training samples confirmed out of 1,800 candidates, with an aggregate true positive rate of 33.5%; the best configuration (Internet generation + zlib inference) is 67% (Table 2).
• Content sensitivity: the extracted content includes 46 named individuals, 32 pieces of contact information (16 of which belong to private individuals), 50 resolvable URLs, 31 code snippets, and 9 k = 1 high-entropy random strings that appear in only one training document (Table 3).
• Scale effect: the authors report that larger models memorize more—in the natural canary experiment, XL extracted all URLs inserted 33 or more times, while Small extracted none under the harder setting (Table 4); the authors also state that in one setting the 1.5B model memorized more than 18 times as much as the 124M model.
• Authors' conclusion: training data extraction is a practical threat, and memorization does not require overfitting; mitigation directions such as differential privacy, data cleaning, deduplication, downstream filtering, and auditing each have weaknesses and are all incomplete, and the authors believe new privacy-preserving training methods need to be developed without sacrificing accuracy and training time.