Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

1. 01What problem does this paper attempt to solve?

   The authors argue that the definition of verbatim memorization is too narrow, and that defenses against only verbatim copying cannot prevent training data leakage.

   Core problem: Language models output approximate paraphrases of training data, and defenses that only block verbatim memorization cannot prevent this leakage.

   • Scenario and importance: Language models reciting training data brings privacy and copyright risks (the authors cite Carlini et al. 2021 saying GPT-2 can output personally identifying information); GitHub Copilot has deployed a filter that "blocks suggestions matching public code."
   • Shortcomings of existing methods: The authors consider "verbatim memorization" (generating substrings identical to the training set) too narrow a definition to capture subtler forms of memorization; deduplication also cannot guarantee the model does not memorize individual samples.
   • Core observation: The authors find that Copilot's filter can be bypassed by "style-transfer" prompts (e.g., translating variable names into French), and that the model itself "evades" verbatim checks through typos, punctuation changes, and the like.
   • Proposed in the paper: Design and implement M EM F REE decoding, a defense that blocks generation of training-set n-grams at decoding time; and use it to systematically study approximate memorization.
   • Threat model:
     • Goal: Extract the model's training data (including approximately memorized content).
     • Knowledge: The training sets of Copilot, GPT-3, and PaLM are not public; the training set of GPT-Neo/Pythia (the Pile) is public.
     • Capability: Modify prompts (style transfer), generate text through an API.
     • Victim systems: GitHub Copilot, GPT-3, PaLM, GPT-Neo, Pythia.

   Collapse this answer

2. 02What related research exists?

   Related work covers memorization measurement, memorization defenses, and definitions of near-verbatim memorization; the authors state that this paper is closest to Lee et al. (2021).

   Memorization metrics

   • Carlini et al. (2021) proposed eidetic memorization: if there exists a prompt p that makes the model output s and s is in the training set, then s is memorized; Kandpal et al. (2022), Tirumala et al. (2022) and others use similar verbatim definitions.
   • Zhao et al. (2022), Stock et al. (2022) explored probabilistic leakage definitions based on differential privacy; Zhang et al. (2021) proposed counterfactual memorization.

   Memorization defenses

   • Differential privacy training (Abadi et al., 2016) is the gold standard for provable non-memorization, but the authors cite Anil et al. (2021) as saying it harms generation quality, and currently no large model is trained with DP.
   • Data deduplication (Lee et al., 2021; Kandpal et al., 2022; Carlini et al., 2022) is a practical countermeasure, but the authors point out that it cannot guarantee the model does not memorize an individual sample.

   Approximate memorization definition

   • Lee et al. (2021) advocate a looser definition of memorization: output within a certain edit distance of the true continuation counts as memorization; the authors say this paper is closest to it, but believe a small edit distance still cannot cover cases such as style transfer.
   • The image generation field has accepted "blurry" memorization (Fredrikson et al., 2015; Zhang et al., 2020; Haim et al., 2022; Balle et al., 2022); the plagiarism detection literature also goes far beyond verbatim matching (Roy et al., 2009; Potthast et al., 2010).

   Baselines and benchmarks: the verbatim memorization baseline is taken from the 30k Pile substrings released by Carlini et al. (2022); the defense controls are undefended generation and retroactive censoring.

   Authors' positioning: this paper proposes MEMFREE, which perfectly prevents verbatim memorization, to show that even a "perfect" verbatim defense cannot prevent training data leakage.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors propose M EM F REE decoding, which blocks training-set n-grams token by token during generation, and use BLEU and edit similarity to measure near-verbatim memorization.

   Overall approach: first build an inference-time defense, MEMFREE decoding, that guarantees blocking all verbatim memorization, then use it to show that even when verbatim memorization is completely blocked, the model still leaks approximate memorization.

   Problem setup

   • Consider an autoregressive language model: given a prefix p, the model outputs a probability distribution over the next token; generation starts from p and repeatedly samples and concatenates.
   • Memorization definition: if the generation contains an n-gram from the training set, it is regarded as verbatim memorization; the defense goal is to prevent such generation at decoding time.

   MEMFREE decoding

   • At each generation step, check whether any n-gram in the concatenated sequence p||t is in the training set D; if so, remove that token's probability mass from the model output distribution before sampling, guaranteeing the generation probability of the memorized token is 0.
   • Unlike retroactive censoring (rejecting the whole passage after generation), MEMFREE filters at n-gram granularity, preserving novel substrings in the generation.
   • Can be combined with any sampling strategy (e.g., top-k): first remove tokens that cause memorization, then truncate to the top k.

   Training set query optimization

   • Only check n-grams ending with the currently predicted token, reducing the number of queries.
   • Only store n-grams that appear more than once in the training set (the authors cite prior work saying repeated text is most likely to be memorized).
   • Store n-grams with a Bloom filter: no false negatives, controllable false positives allowed; on the 800GB Pile, the filter storing 10-grams appearing 10 or more times is 1.6GB (storing all 10-grams is 40.5GB).
   • Key parameters: n=10 (the authors call it conservative, filtering out common phrases longer than 10 tokens); Bloom filter false positive rate about 1%; number of hash functions k=7.

   Measuring approximate memorization

   • Use BLEU and length-normalized character-level Levenshtein similarity to measure the similarity between the generation and the true continuation.
   • Binary judgment: BLEU > 0.75 is regarded as approximate memorization; the authors say the threshold was chosen through qualitative inspection and caution that it may miscount (both overcount and undercount).

   Experimental design

   • For models whose training sets are not public (Copilot, GPT-3, PaLM), test approximate memorization leakage with style transfer prompts.
   • For GPT-Neo/Pythia, use the 30k Pile substrings from Carlini et al. (2022) (divided into 30 buckets by repetition count, 1k per bucket), take the first 50 tokens as the prompt, generate a 50-token continuation, with arg max sampling.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors state that style-transfer prompts can bypass the verbatim defenses of Copilot and GPT-3/PaLM; M EM F REE reduces verbatim memorization by 3000-fold but near-verbatim memorization still exists.

   Experimental Setup

   • Models tested: GitHub Copilot (Codex); GPT-3 DaVinci Original and v2; PaLM 62B and 540B; GPT-Neo 125M–6B; Pythia and its deduplicated version (6.9B).
   • Datasets: 30k Pile substrings from Carlini et al. (2022) (30 buckets × 1k); style-transfer experiments used open-source licenses (n=42), famous speeches (n=100), monologues (n=60), novel openings (n=77), and 2011 and 2021 song lyrics (the n annotations are inconsistent across the paper's figures and tables: Figure 4 labels n=100, Table 2 labels n=11).
   • Baselines: undefended generation; retroactive censoring (a conceptual comparison in the paper).
   • Metrics: BLEU (NLTK default parameters), character-level edit similarity, memorization proportion (BLEU > 0.75), number of verbatim memorization instances.
   • Evaluation method: no human or LLM review; automatic determination using a BLEU threshold; the threshold was chosen by the authors through qualitative inspection.
   • Samples and repetitions: M EM F REE experiments used 5,000 prompts (Figure 6); Bloom statistics used 6,000 50-token generations; the paper does not report the number of repetitions.

   Main Results

   Swipe horizontally to read the table

   Model	Condition	Result	Source
   GPT-Neo 6B	Verbatim memorization (undefended → M EM F REE)	12,000+ instances → 4 instances (over 3000-fold reduction)	Appendix B.6
   GPT-Neo various sizes	Near-verbatim memorization (BLEU)	BLEU remains around 0.6 after M EM F REE, effective across all sizes	Figure 8
   GPT-Neo	High-repetition samples	The more repetitions, the more M EM F REE reduces BLEU	Figure 7
   Pythia 6.9B	Near-verbatim memorization	More near-verbatim memorization than GPT-Neo; M EM F REE still effective but slightly weaker	Figure 14, 15

   • The authors state that M EM F REE barely changes outputs on generations that originally had low similarity (points on the diagonal in Figure 6).
   • The authors state that the larger the model, the higher the similarity between undefended generations and the original source, but M EM F REE is effective at all sizes.

   Style-Transfer Bypass of Defenses

   • Copilot: Style-transfer prompts such as Python-style comments and French variable naming can make Copilot output memorized content when the "block matching public code" option is enabled (Figure 2, Appendix F); even passive prompting with high-repetition text from the Pile can bypass it (Figure 5).
   • GPT-3 and PaLM: Applying 3 style transfers (double spaces, all lowercase, all uppercase) to 6 document categories, the authors state the models can still generate memorized continuations (Figure 4, 13). Table 2 gives GPT-3 counts: under original prompts from open-source licenses (n=42), DaVinci Original had 23 successful memorizations and v2 had 30; all-uppercase transfer was least likely to trigger memorization.
   • The authors state that PaLM 540B memorizes more easily than 62B, and the two GPT-3 versions differ greatly in memorization tendency (v2 is more sensitive to double spaces).

   Scale of Near-Verbatim Memorization

   • The authors repeated the prefix-extraction experiments of Carlini et al. (2022) using a near-verbatim definition, and state that prior literature underestimated memorization leakage: in Figure 3, the proportion of near-verbatim memorization (BLEU > 0.75) is higher than verbatim memorization, in the worst case by a factor of two (the authors call it a factor-of-two increase).

   Failure Cases and Negative Results

   • When M EM F REE changes only a few tokens, generations can still be highly similar to the true continuation (points near and above the diagonal in Figure 6); the authors state such examples are mostly numeric lists (Appendix G).
   • 0.16% of samples actually increased in similarity by more than 0.1 after M EM F REE; the authors state many of these have clear overlap with the true continuation.

   Other Ablations and Analyses

   • Downstream task impact: over 86% of validation targets in BillSum and Arxiv summarization tasks contain 10-grams in the Bloom filter; non-English tasks (e.g., WebNLG-ru 99.6%) are affected more (Table 1).
   • Computational cost: 49.8 milliseconds for 10,000 queries on a single CPU core; an average of 42.1 queries per generation; 12.6% of generations had more than 50 Bloom hits (Figure 9).
   • Most generations require fewer than 5 changes to pass the check (Figure 11).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that the definition of near-verbatim memorization is domain-dependent and difficult to exhaust, and that experiments cover only English and code models.

   Limitations and Future Directions Noted by the Authors

   • The near-verbatim memorization definition of BLEU > 0.75 is imprecise and may both overcount and undercount (Section 3.3).
   • The definition of near-verbatim memorization is domain-dependent; different languages and tasks require different considerations (Section 6).
   • New near-verbatim cases must be discoverable by the definition; the authors state this will become a "cat-and-mouse game," and exhausting all styles is infeasible (Section 6).
   • The boundary between memorization (e.g., personal information) and knowledge (e.g., common sense) is blurry; loosening the definition may make it harder to understand memorization and generalization (Section 6).
   • Studying the interaction between M EM F REE and model utility is future work (Section 6).
   • Drawing on plagiarism-detection techniques to improve the near-verbatim memorization definition is a direction the authors consider valuable (Section 6).

   Experimental Coverage

   • 8 models tested in total: Copilot, two GPT-3 versions, two PaLM sizes, four GPT-Neo sizes, two Pythia versions.
   • Languages and domains: only English natural language and code; style-transfer experiments cover five document categories: licenses, speeches, monologues, novels, and song lyrics.
   • M EM F REE was applied only to GPT-Neo and Pythia, whose training sets are public; Copilot, GPT-3, and PaLM were used only for bypass experiments because their training sets are not public.
   • The paper does not report the number of experimental repetitions, nor the agreement between BLEU threshold judgments and human judgments.
   • The song-lyrics sample size is inconsistent within the paper: Figure 4 labels n=100, Table 2 labels n=11.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors build the M EM F REE defense, which can fully block verbatim memorization, and show that models still leak approximate memorization, so verbatim defenses give a false sense of security.

   An empirical study on defenses against memorization in language models, whose core claim is: defenses that only block verbatim memorization cannot prevent training data leakage.

   • Problem: Language models reciting training data brings privacy and copyright risks; existing defenses (such as Copilot's filter) only check for verbatim matches, which the authors consider too narrow a definition.
   • Method: Propose M EM F REE decoding—at generation time, check token by token whether it forms a training-set n-gram, and if so remove it from the probability distribution; implement a 1.6GB filter over the 800GB Pile using a Bloom filter; use BLEU > 0.75 to judge approximate memorization.
   • Result one: Style-transfer prompts (e.g., French variable names, Python comments) can bypass Copilot's verbatim filter; GPT-3 and PaLM are also susceptible to similar prompts for documents such as licenses and speeches (Figure 2, 4).
   • Result two: On GPT-Neo 6B, M EM F REE reduces verbatim memorization from over 12,000 to 4 (a reduction of over 3000×), with almost no effect on generations that were not memorized in the first place (Appendix B.6, Figure 6).
   • Result three: Approximate memorization still persists—the authors say the proportion of approximate memorization is up to twice as high as verbatim memorization (Figure 3); 0.16% of samples actually increase in similarity by more than 0.1 after M EM F REE.
   • Authors' conclusion: Preventing verbatim copying is necessary but insufficient; the definition of memorization needs to cover semantic similarity, but the definition of approximate memorization is domain-dependent and hard to exhaust, making it a difficult but crucial open problem.

   Collapse this answer

Related papers