Quantifying Memorization in Neural Language Models

Core question: How much training data do large language models actually memorize, and how does memorization vary with model scale, data repetition count, and prompt context length?

• Why it matters: The authors state that memorizing training data harms privacy (exposing user data), reduces utility (easily memorized repeated text is often low quality), and harms fairness (some text is more easily memorized than other text). Training-data extraction attacks have been shown to be a real threat.
• Shortcomings of existing work: The authors state that existing attacks give only loose lower bounds—Carlini et al. (2020) manually found 600 memorized samples in GPT-2, proving only that at least 0.00000015% of a 40GB dataset is memorized; and prior studies mostly fix the model or dataset scale, lacking quantitative understanding across scales and model families.
• Core observation: The authors find that memorization grows log-linearly with three factors—model scale (larger models within the same family memorize 2-5× more), data repetition count, and prompt context length; but generalization across model families is more complex.
• Contributions of the paper: Using access to the original training sets of three model families (GPT-Neo/J, T5, OPT), prompting the models with training-set prefixes and checking verbatim reproduction, the paper gives memorization bounds orders of magnitude more precise than prior work, estimating that GPT-J 6B memorizes at least 1% of The Pile.

Training-data extraction attacks

• Carlini et al. (2020), Ziegler (2021)—demonstrate extraction of memorized data such as URLs and phone numbers from language models; the authors note that these works are qualitative, only proving the existence of extractable data without precisely quantifying memorization, e.g., Carlini et al. (2020) found only 600 samples in GPT-2.
• Carlini et al. (2019), Henderson et al. (2018), Thakkar et al. (2020), Thomas et al. (2020)—study memorization by injecting "canaries".

Privacy attacks and defenses

• Shokri et al. (2017), Yeom et al. (2018)—membership inference attacks; Ganju et al. (2018), Fredrikson et al. (2015)—attribute/model inversion attacks. The authors state that this paper focuses on extraction attacks because they are most relevant to language models and grow with data repetition (Lee et al., 2021).
• Differential privacy of Dwork et al. (2006): the authors note that it protects individual user data but cannot prevent memorization of highly repeated data, and differentially private training algorithms are computationally expensive, slow to converge, and have poor utility (Abadi et al., 2016; Anil et al., 2021).

Definitions of memorization

• Counterfactual memorization of Feldman and Zhang (2020), Zhang et al. (2021), exposure of Carlini et al. (2019), k-eidetic memorization of Carlini et al. (2020): the authors state that these definitions require training hundreds or thousands of models or thousands of generations per sample, which is impractical for large models, so this paper adopts the more operational "extractable" definition.

Concurrent work and scaling hypotheses

• Kandpal et al. (2022) (concurrent work)—studies the relationship between memorization and data repetition, but focuses on explaining why attacks succeed; the authors state that this paper explicitly prompts with training-data prefixes to measure worst-case memorization.
• Sources of scaling evidence: model scale (Zhang et al., 2017; Carlini et al., 2020), data repetition (Lee et al., 2021), context length (Carlini et al., 2020; Ziegler, 2021).

Baselines and data: The GPT-2 family serves as a control baseline not trained on The Pile; the benchmarks are The Pile (825GB) and C4 (806GB). The authors describe this paper as "the first comprehensive quantitative analysis" of large-model memorization, with the goal of tightening memorization bounds to order-of-magnitude precision.

Overall approach: using access to the model's original training set, prompt the model with prefixes from the training data and check whether the model can verbatim complete the suffix, thereby systematically measuring memorization across three model families. The core definition is extractability (extractable).

Definition of memorization

• Definition 3.1: A string s is extractable for model f under a k-token context when there exists a prefix p of length k such that the concatenation [p || s] appears in f's training data and f outputs s for p under greedy decoding.
• Rationale for the choice: The authors state that this definition is more operational—differential privacy lower bounds or counterfactual memorization require training hundreds or thousands of models, exposure requires thousands of generations per sample, and k-eidetic memorization is not suitable for tightening bounds using training-data prompts.
• Greedy decoding is used throughout, and Section 4.1 verifies that the decoding strategy has little effect on the results.

Construction of evaluation data

• Cost of the ideal approach: Testing every sequence in the training set is too costly—the authors estimate that a 6B model on a V100 generates about one 100-token sequence per second, so testing all 800GB of data would take over 30 GPU-years.
• Uniform random subset: About 50,000 sequences, used to estimate absolute memorization; however, because repetition frequency decays quickly, a uniform sample contains almost no highly repeated samples and cannot measure the tail of the repetition distribution.
• Repetition-normalized subset: For each length l ∈ {50, 100, …, 500} and integer n, select 1,000 sequences each that appear in the training set between 2^(n/4) and 2^((n+1)/4) times, until fewer than 1,000 can be found; this covers buckets from 6–8 repetitions to 724–861 repetitions.
• Scale: 50,000 sequences collected per length, about 500,000 sequences in total; for a sequence of length l, the first l−50 tokens are used as the prompt, and if the model verbatim generates the last 50 tokens it is judged extractable; 50 tokens is about 127 characters or 25 words.
• Tool: Lee et al. (2021)'s suffix array is used to efficiently enumerate sequences in a specified repetition-count range over 800GB of data (Appendix A).

Adaptation across model families

• T5 (masked language model): Definition 3.1 does not directly apply; instead, a sequence is judged memorized if the model can perfectly predict the randomly masked 15% of tokens using 85% of the visible tokens; only one masking is tested per sequence.
• OPT: The training set overlaps with The Pile but is not identical and has been deduplicated; the authors cannot access its original training set.

Success criterion and alternative settings

• Primary criterion: The generated content exactly matches the sample's true suffix (greedy decoding).
• Alternative settings (Section 4.4, Appendix C): uniformly randomly sample 100,000 sequences; beam search (100 beams); lenient criterion—the generated content counts as memorized if it appears anywhere in the training set (computationally expensive, used only as a comparison).

Experimental setup

• Models tested: GPT-Neo family (125M, 1.3B, 2.7B, 6B, where 6B is GPT-J); T5 v1.1 (77M to 11B, with Base/Large/XL etc. reported in the figures); Lee et al. (2021)'s 1.5B causal language model (three variants: original C4, NearDup deduplication, ExactSubstr deduplication); OPT (125M to 66B; the authors could not access 175B).
• Control baseline: GPT-2 family (not trained on The Pile), used to distinguish "memorization" from "generalization".
• Datasets: The Pile (825GB), C4 (806GB) and their deduplicated versions.
• Sample sizes: The main experiment uses the repetition-normalized subset of about 500,000 sequences (50,000 per length); the uniform random subset has 100,000 sequences; 1,000 sequences per repetition bucket.
• Metric: fraction extractable—the fraction of sequences whose 50-token suffix is verbatim reproduced under greedy decoding; for T5, the fraction that perfectly recovers the 15% masked tokens.
• Criterion: exact string matching, no LLM judging; repetition counts are computed with a suffix array.

Main results

Main results are in Figure 1 (repetition-normalized subset, GPT-Neo/J on The Pile); key numbers given in the text:

Condition	Value	Source
10× increase in model scale, increase in memorization	19 percentage points (log-linear fit R2=99.8%)	Section 4.1 / Figure 1a
Extractable fraction for GPT-Neo 1.3B	40%	Section 4.1
Extractable fraction for GPT-2 (similar-scale control)	about 6%	Section 4.1
GPT-J 6B, 50-token context	33%	Section 4.3 / Figure 1c
GPT-J 6B, 450-token context	65%	Section 4.3 / Figure 1c

• The authors state that all three factors show log-linear relationships: larger models memorize more, more repetitions are more extractable, and longer contexts are more extractable.
• The authors state that the GPT-2 control group does not grow with scale, indicating that GPT-Neo's growth comes from true memorization rather than generalization ability; qualitative analysis shows that what GPT-2 "memorizes" is mostly digit sequences, repeated tokens, etc. (Appendix Figure 15).
• The authors call the context effect the discoverability phenomenon: some memorization only becomes apparent with a sufficiently long context.

Uniform random sampling and alternative settings

• Uniform sampling (Figure 2a/2b): Trends are consistent with the main experiment; extracting the last 50 tokens of 1000-length sequences, GPT-J 6B is 7%, GPT-Neo 125M is 4%, and GPT-2 XL 1.5B is 2%; based on this the authors estimate that GPT-J 6B memorizes at least 1% of The Pile (and the non-extractable portion of GPT-2 XL).
• Beam search (Figure 2c left): 100 beams extract on average less than 2 percentage points more than greedy decoding, at most 5.6%; the two produce the same output 45% of the time.
• Lenient criterion (Figure 2c right): Among samples repeated 100 times, 32.6% of generations appear anywhere in the training set, while only 15.8% match the true suffix; the maximum difference between the two criteria is 28.4% (at 2,200 repetitions); it was not used in the main experiment due to high computational cost.

Replication experiments: T5, deduplicated data, OPT

• T5 (Figure 4a/4b): The scale trend holds, but absolute memorization is an order of magnitude lower than same-scale causal models—T5-XL 3B memorizes 3.5% of sequences repeated 100 times, whereas GPT-Neo 2.7B memorizes 53.6% (150-token context). The relationship between repetition count and memorization is non-monotonic: sequences repeated 138–158 times have a memorization rate of at least 6.2%, higher than the under-5.1% for sequences repeated 159–196 times (both at 99.7% confidence); the authors qualitatively find that the 138–158 bucket contains mostly whitespace sequences, which are easier to predict.
• Deduplicated C4 (Figure 4c): For sequences repeated fewer than 35 times, the exact-deduplication model memorizes 1.2% versus 3.6% without deduplication, a 3× decrease (p < 10^-15); however, deduplication is ineffective for sequences repeated more than about 100 times, and sequences repeated at least 408 times are statistically more extractable. The authors conjecture that deduplication is necessarily imperfect on large-scale data.
• OPT (Figure 7): The scale trend is almost identical to GPT-Neo, but the effect is orders of magnitude smaller—66B OPT memorizes a lower fraction of The Pile samples than 125M GPT-Neo. The authors offer two possible explanations (data curation reduced memorization, or a slight shift in data distribution changed what is memorized), and state that without the original training set they cannot distinguish between them.

Other ablations and analyses

• Sequence length (Appendix B, Figure 5): Under the same repetition count and the same 6B model, the extractable-fraction curves across prompt lengths for sequences of 100–500 tokens almost completely overlap; the authors state that longer sequences are not more easily extracted.
• Memorization unique to each model (Appendix D, Table 1): The 6B model memorizes 14,792 sequences, of which 10,273 are not memorized by 125M; 125M memorizes 4,812 sequences, of which 293 are not memorized by 6B—each model has unique memorization.
• Qualitative examples (Figure 3, Appendix F): Among samples memorized only by 6B, the small model outputs are topically related but semantically wrong; samples memorized by all models are mostly code, open-source licenses, etc.; there are 35 samples memorized only by 125M and 2,860 memorized only by 6B.
• Negative/exceptions: Samples repeated thousands of times but not memorized by 6B (Figure 14) are mostly text differing only in semantically irrelevant ways (comment markers, template values, error messages, etc.); the repetition-memorization relationship is non-monotonic on T5 (see above).

Limitations and future directions noted by the authors

• OPT conclusions cannot be determined (Section 5.3): without OPT's original training set, it is impossible to distinguish between the two explanations "careful data curation and training can reduce memorization" and "a slight shift in data distribution changes what is memorized"; the authors hope future work will resolve this.
• Auditing depends on training data prompts (Section 4.3): the authors state that properly auditing large models may require training data prefix prompts; there is currently no technique for identifying memorized tails without providing long contexts, and improving this is future work.
• Deduplication cannot fully prevent leakage (Sections 4.2, 5.2): memorization still occurs with few repetitions; deduplication is ineffective for sequences repeated more than about 100 times, and the authors conjecture that any deduplication strategy scalable to hundreds of GB must be imperfect.
• Transferability of the definition (Section 3.1): the authors state that future work could explore whether the three scaling laws apply to other definitions of memorization.
• Inference about larger models (Section 6): existing frontier models have more than 200 times the parameters of the 6B model tested; the authors believe larger models may memorize sequences repeated only a few times, and actual extraction would require entirely new attack strategies.
• 20B GPT-Neo variant not included (Section 4 footnote): its training setup and tokenizer differ, making it difficult to apply this paper's method.

Scope of experiments

• Four model families were tested: GPT-Neo/J (125M–6B, 4 sizes), T5 v1.1 (77M–11B), the 1.5B model of Lee et al. (2021) (3 deduplicated variants), and OPT (125M–66B; 175B not tested).
• The datasets are The Pile (825GB) and C4 (806GB) and their two deduplicated versions; the main experiment uses about 500,000 samples (repetition-normalized), a uniformly random subset of 100,000, and 1,000 per repetition bucket.
• Decoding strategies cover greedy decoding and beam search (100 beams); the authors explicitly do not run random sampling experiments, since their goal is to maximize discoverability rather than text novelty.
• The criterion is exact match with the true suffix; a loose criterion (generation appears anywhere in the training set) is used only as a control and not in the main experiments.
• The paper does not report the number of repeated runs per measurement (T5 masking was measured in only one setting); apart from the T5 repetition buckets (99.7% confidence) and the deduplication comparison (p < 10^-15), the main experiments report no statistical significance tests.

This is an analysis paper on memorization of training data in large language models (ICLR 2023); the authors state that it is the first comprehensive quantitative analysis of memorization in large models.

• Problem: Existing extraction attacks only give loose lower bounds on memorization (e.g., 600 examples on GPT-2), and there is a lack of quantitative understanding across scales and model families; memorization brings privacy, utility, and fairness risks.
• Method: Define "extractable"—using a prefix prompt of length k from the training set, verbatim reproduction of the suffix by greedy decoding counts as memorization; measured on about 500,000 sequences sampled in a normalized way by length (50–500 tokens) and number of repetitions, using a suffix array to efficiently locate sequences with a specified number of repetitions.
• Scaling effect: Within the GPT-Neo family, a 10× increase in model size increases the extractable proportion by 19 percentage points (log-linear fit R2=99.8%); the GPT-2 control not trained on The Pile is only about 6% (GPT-Neo 1.3B is 40%), and the authors therefore argue that the growth comes from memorization rather than generalization.
• Repetition and context effects: The more times a sequence is repeated, the more extractable it is, but memorization still occurs with few repetitions; GPT-J 6B extracts 33% with a 50-token context, rising to 65% at 450 tokens—the authors call this a discoverability phenomenon.
• Total amount estimate: Under uniform sampling, GPT-J 6B has an extraction rate of 7% for length-1000 sequences, and the authors estimate that it memorizes at least 1% of The Pile.
• Cross-family replication: The T5 scaling trend holds but absolute memorization is an order of magnitude lower (T5-XL 3B memorizes 3.5% of sequences repeated 100 times, versus 53.6% for GPT-Neo 2.7B); deduplicated C4 models reduce memorization of sequences repeated fewer than 35 times from 3.6% to 1.2%, but are ineffective for repetitions above about 100; OPT shows the same trend but the effect is orders of magnitude smaller, and the reason cannot be determined.
• Authors' conclusions: Memorization is more prevalent than previously thought and may become more severe as models continue to grow larger; deduplication is a feasible mitigation but not complete; actually extracting these memorized sequences requires entirely new attack strategies, and privacy audits may require training-data prompts.