Skip to content

Information-guided method identifies training data traces in proprietary large language models

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models

arXiv· arXiv:2503.12072· Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Yuchen Lin, Maria Antoniak, Niloofar Mireshghallah, Chandra Bhagavatula, Yejin ChoiUniversity of Washington; University of Copenhagen; ChipStack AI; Stanford UniversityIndexed by this site · 2026-10-14· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

Information-guided probes identify text memorized by LLMs under black-box access. On GPT-4 confabulated text, the Person probe Fβ is 82.7, higher than LCS's 56.9.

Threat modelThe candidate text is known. For closed, black-box models, only input-output access is available, with no weights, logits, token probabilities, or pretraining data. Discoverable memorization is tested: given part of a sample, see whether the model can recover the masked high-surprisal tokens.

Questions
Commercial LLMs rarely disclose training data, making it hard for outsiders to verify copyright, data misuse, and evaluation contamination. The question is: with only inputs and outputs, how can one determine whether a candidate text is memorized by a closed-source model.
Methods
First, a reference model (BERT, 110M) finds high-surprisal tokens; optionally, a small instruction-tuned model filters out tokens guessable from context; then each token is masked in turn and the target model fills in the blank. On fictional text, reconstructing at least two tokens counts as memorized.
Experiments and results
On GPT-4 confabulated text, the Person probe Fβ=0.1 is 82.7, higher than LCS's 56.9 in the same table. On NYT, GPT-4 with Rank+filtering has precision 70.0, while GPT-3.5 has higher LCS. On GPQA, the EM difference before and after contamination with Prob+IF is 78.23%.
Limitations and future work
The authors note that membership is vaguely defined, the method relies on memory, and closed-source variance and post-hoc filtering hinder reproducibility. Experiments include GPT-3.5, GPT-4, Llama-2, and BookMIA, NYT, GPQA, CommonsenseQA, ARC. The paper does not report the number of repetitions for the main experiments.
Experimental setupGPT-3.5 (gpt-3.5-turbo-0125)、GPT-4 (gpt-4-0613) et al. · BookMIA、New York Times Exhibit-J et al. · Precision、Recall et al.
Evaluated models
GPT-3.5 (gpt-3.5-turbo-0125)GPT-4 (gpt-4-0613)Llama-2-70BLlama-2 7BLlama-2 13B
Benchmarks
BookMIANew York Times Exhibit-JCNN 2023 articlesGPQACommonsenseQAARC-Challenge
Metrics
PrecisionRecallFβ=0.1Exact Matchreconstruction rate
AI summaryThe paper proposes an information-guided probing method that can identify training data memorized by proprietary LLMs such as GPT-4 without needing access to model weights or token probabilities. The authors note that commercial LLM providers rarely disclose training data details, limiting external oversight of copyright infringement audits, data author rights, and research on data contamination and data selection. The method is based on the observation that text passages with high surprisal are suitable search material for memory probes. By evaluating whether the model can successfully reconstruct high-surprisal tokens in the text, a large amount of text memorized by the LLM can be identified.

The paper proposes an information-guided probing method that can identify training data memorized by proprietary LLMs such as GPT-4 without needing access to model weights or token probabilities. The authors note that commercial LLM providers rarely disclose training data details, limiting external oversight of copyright infringement audits, data author rights, and research on data contamination and data selection. The method is based on the observation that text passages with high surprisal are suitable search material for memory probes. By evaluating whether the model can successfully reconstruct high-surprisal tokens in the text, a large amount of text memorized by the LLM can be identified.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    In a black-box setting without weights and token probabilities, determine whether candidate text is memorized by a closed-source LLM.

    Given a piece of text and a closed-source black-box model, can one determine from inputs and outputs alone whether the sample is memorized.

    • Scenario: The authors state that commercial models rarely disclose training data, making it hard for outsiders to check copyright and data misuse, hard for data authors to control use, and hard for researchers to judge generalization and evaluation contamination. The text cites Gemini as described as containing web pages, books, and code, and Llama-2 as described as coming from public sources and not containing Meta product data.
    • Limitations of existing work: Character-name cloze assumes the text contains character names (Chang et al., 2023). Prefix completion often does not generate verbatim, so it relies on heuristics such as longest common subsequence (Karamolegkou et al., 2023a). The authors consider these tests either genre-bound or insufficiently precise.
    • Core observation: High-surprisal spans are suitable as memory probes. The method's premise is that if a token is hard to reconstruct from context alone, the remaining mechanism for successfully filling it in is memory.
    • Proposed probes: information-guided probes. Select high-surprisal tokens for cloze, use reconstruction rate to identify memorized text, and apply this to copyrighted content and evaluation-set contamination.
    • Threat model
      • Goal: Determine whether a given sample is likely memorized, rather than extracting unknown training text from scratch.
      • Knowledge: closed, black-box. Only inputs and outputs; no weights, logits, or token probabilities, and no access to pretraining data.
      • Capability: Submit cloze prompts and see whether the model can reconstruct masked tokens. The paper calls this discoverable memorization.
      • Targets: LLMs whose training data is not public. Main experiments use GPT-3.5, GPT-4, and Llama-2-70B.
  2. What related research exists?

    Related work is grouped into memory tests, surprisal, copyright and contamination, and membership inference; baselines are prefix probing and TS-SLOT.

    Memory tests without probability access

    • Discoverable memorization: Carlini et al. (2022), Liu et al. (2023), Chang et al. (2023), D'Souza and Mimno (2023) test whether a model can recover the rest of a training sample when given part of it. The authors paraphrase Carlini et al. (2022) as stating that about 1% of training data can be recovered this way, and state that cloze requires specific features such as character names, while prefix completion performs poorly.
    • Books and poetry: Chang et al. (2023) use character-name cloze to check which books ChatGPT/GPT-4 has seen. D'Souza and Mimno (2023) and Walsh et al. (2024) address poetry memorization and poetry evaluation respectively; the authors use these to illustrate that prior tests are often genre-bound.
    • Prefix completion: Karamolegkou et al. (2023a) and Grynbaum and Mac (2023) give a text prefix and compare continuations. The authors state that modern LLMs may have post-training or output filtering, so they often cannot reproduce text verbatim, and evaluation instead uses longest common subsequence (LCS).

    Surprisal

    • Information content: Shannon (1948), Hale (2001), Levy (2008) use negative log probability under context to represent the information carried by a linguistic unit. Predictable units have low surprisal; unexpected units have high surprisal.
    • Link to comprehension behavior: McDonald and Shillcock (2003), Goodkind and Bicknell (2018), Oh and Schuler (2023), Giulianelli et al. (2023) treat LLMs as estimators of token surprisal and compare them with human comprehension measures. The authors use the same concept to distinguish "reconstruction from context" from "reconstruction from memory."

    Copyright, evaluation contamination, and membership inference

    • Copyright chain: The authors place the risk in data collection (Min et al., 2023; Shi et al., 2023; Chang et al., 2023; Karamolegkou et al., 2023a), training (Vyas et al., 2023), and generation deployment (Meeus et al., 2024; Ippolito et al., 2023). Duarte et al. (2024)'s DE-COP turns black-box copyright detection into multiple-choice questions; the authors state it targets document-level membership inference, is not the focus of this paper, and relies on proprietary LLMs and is more expensive.
    • Evaluation contamination: Oren et al. (2023), Golchin and Surdeanu (2023), Weller et al. (2024), Xu et al. (2024), Sainz et al. (2024) infer whether test samples entered training. The authors state that some of these methods use metadata such as sample order, which this paper does not have. Deng et al. (2024)'s TS-SLOT first has ChatGPT mark informative words, then has the target model guess the masked words.
    • Membership inference: Shokri et al. (2017) and subsequent LOSS, likelihood-ratio, Zlib entropy, curvature, Min-k%, etc., rely on thresholding membership scores. Duan et al. (2024) state that membership inference on LLMs is near random, partly due to large datasets and few iterations. The authors position this paper as finding memories that leave strong traces, not as determining all members and non-members.

    Data documentation and corpus retrieval

    • Accessible data: Bender and Friedman (2018), Gebru et al. (2021), Liu et al. (2024b) promote data documentation; Elazar et al. (2024) and Liu et al. (2024a) retrieve and index pretraining corpora. The authors state that these works target available datasets, whereas this paper infers training data from the model side when there is no data transparency.

    Baselines and data: Copyright experiments compare against prefix probing/LCS, as well as Random and Majority; contamination experiments compare against TS-SLOT. When token probabilities are available, comparisons are also made with perplexity, PPL/ZLib, and Min-K%. Appendix D compares against DE-COP. Data are BookMIA, New York Times litigation materials and CNN 2023 articles, GPQA, CommonsenseQA, and ARC-Challenge.

    The authors position this paper on probes that require only inputs and outputs and can be implemented with smaller language models, and state that it is complementary to prefix probing and applicable to proprietary LLMs.

  3. How does the paper address this problem?

    Use a reference model to select high-surprisal tokens, mask them, and see whether the target model can reconstruct them, thereby identifying memorization.

    The overall approach is called information-guided probes: first find high-surprisal tokens, then construct a cloze masking one token at a time, and finally use the target model's reconstruction rate to determine whether the text is memorized. Figure 1 depicts the pipeline as selecting tokens, constructing probes, and counting successful reconstructions.

    Probe form

    • Inputs and outputs: Mask one high-surprisal token at a time, keeping the rest of the context unchanged. The target model only returns the fill-in-the-blank result; its token probabilities are not needed.
    • Reconstruction rate: For a text, count the number of high-surprisal tokens the target model successfully reconstructs. On fictional text, at least two successes mark the sample as memorized, to avoid a single chance hit (Appendix A).
    • Prompts: Appendix B gives the full prompts for fictional text, news, and contamination probing, as well as prompts for knowledge filtering; these contain instructions and in-context examples. The specific constraints of these prompts are not restated here.

    How to select high-surprisal tokens

    • Concept: The information content of a token is I(wt)=−log P(wt|w<t),即可预测性越低,surprisal 越高。
    • 两种实现:负对数概率 Prob,以及排序名次 Rank(wt)=|{x:P(x|hc)>P(wt|hc), x∈ V}|. Rank counts how many words are more likely than the current token, rather than looking only at its own probability. For fictional text, character names from domain knowledge (Person) can also be used directly.
    • Reference model: The target model does not provide probabilities, so a secondary model estimates Prob and Rank. Ideally this model has not memorized the sample; the authors consider this unattainable and therefore choose a low-capacity base model. In experiments this is BERT (110M). The authors cite Carlini et al. (2022) stating that low-capacity models have much lower memorization rates.
    • Threshold: Selected on a held-out 1870 BookMIA entries, log likelihood < -12,或 rank > 2000. If a text has more than 10 candidates, only the 10 with the lowest probability or highest rank that pass the threshold are kept (Appendix A).

    Knowledge filtering

    • Purpose: Different models know different facts. After the reference model produces candidates, optionally a low-capacity instruction-tuned model reconstructs the masked tokens; tokens guessable from context are filtered out.
    • Difference from the reference model: The reference model is a base model and provides probabilities; the filter is an instruction-tuned model that fills in blanks in a way closer to the final target model. The authors write that one or more such models can be used.
    • Use in experiments: New York Times uses Mistral-V2 and Alpaca-7B. The contamination experiment uses Mistral-V2. Appendix A states that NYT is a real domain, so an additional filter is used.

    Judgment and baseline protocol

    • Copyrighted text: For fictional works, the first 50 words of a passage are used for prefix probing; NYT uses the prefixes given in litigation Exhibit-J. Continuations and original text are compared with LCS, then a logistic regression is fit on LCS over a validation set for test-set classification (Appendix E).
    • Contamination: The baseline is TS-SLOT, which first has ChatGPT select informative words, then constructs masked prompts. The controlled experiment fine-tunes an uncontaminated model on target data and compares Exact Match and its difference before and after contamination.
    • Membership labels: For fictional text, "memorized" positive examples are popular book passages, and negative examples are excerpts from books published in 2023, after the knowledge cutoff of the studied models. NYT positives are articles that litigation materials claim GPT-4 memorized. These are dataset setups, not training-set membership independently verified by the authors.
  4. What experiments did the paper conduct?

    On confabulated text, the surprisal probe's Fβ is higher than LCS; on NYT, GPT-3.5 is the opposite, while on GPT-4 the Rank probe is higher.

    Experimental setup

    • Target models: gpt-3.5-turbo-0125, gpt-4-0613, Llama-2-70B. Appendix A states that all three have knowledge cutoffs before 2023. Figure 3 additionally uses Llama-2 7B and 13B. Table 4 writes the controlled contamination target as gpt-3.5-turbo, with the table note writing ChatGPT; the paper does not directly state whether it is the same checkpoint as gpt-3.5-turbo-0125.
    • Auxiliary models: The reference model is BERT (110M). NYT knowledge filtering uses Mistral-V2 and Alpaca-7B; the contamination experiment filter is Mistral-V2. Appendix D uses Claude-2 to rewrite passages for DE-COP.
    • Data: BookMIA test set of 8k entries, with 1870 held out for threshold tuning. Positives follow Chang et al. (2023) as popular book passages; negatives are excerpts from books published in 2023. NYT positives are 100 litigation Exhibit-J articles; negatives are 100 CNN 2023 articles. GPQA was written by experts in biology, physics, and chemistry and released in 2023; the authors therefore treat it as unlikely to be contaminated for these models. CommonsenseQA and ARC-Challenge use test sets; the paper does not state the total number of questions, only the number of probes.
    • Metrics: Precision is primary, i.e., the proportion of samples labeled as memorized that are positive; when precision is close, Fβ=0.1 is considered, which weights precision more. Contamination experiments report Exact Match and the difference before and after contamination. The paper does not use LLM evaluation.
    • Baselines: Random, Majority, LCS prefix probing, TS-SLOT. Llama-2-70B is additionally compared with perplexity, PPL/ZLib, and Min-K%, which have token probability access. Appendix D compares DE-COP on 600 entries.
    • Unstated protocol: The paper does not report sampling temperature or number of repetitions.

    Main results

    Identification results on fictional text are as follows. Numbers are from Table 2, β=0.1. Random has P/R/Fβ of 50.2/50.5/50.2 and Majority 49.8/100/50.2, not separated by model.

    Swipe horizontally to read the table

    ModelPerson PPerson FβProb FβRank FβLCS Fβ
    GPT-3.583.182.571.969.353.4
    GPT-482.282.781.682.356.9
    Llama-2-70B75.874.660.959.652.9
    • The authors state that relative to suffix completion, surprisal tokens more precisely identify book passages memorized by all three models; the baseline row in the table is LCS. Their goal is to reduce false positives labeled as memorized.
    • The authors state that Person tokens are highly informative when domain knowledge is available; the Prob and Rank tokens from the reference model also work. The examples in Table 1 are not limited to character names, but also include fictional place names and common words in unusual contexts.
    • Prob/Rank recall is low: GPT-3.5 has 11.7 and 10.5, Llama-2-70B has 8.5 and 7.0. On GPT-4, Person recall is 75.3 and Rank precision is 82.6 (Table 2).

    New York Times articles

    Positives are 100 articles that litigation materials claim GPT-4 memorized; negatives are 100 CNN 2023 articles. Only GPT-3.5 and GPT-4 are tested. The table below is from Table 3, omitting Prob and Prob+IF.

    Swipe horizontally to read the table

    ModelMethodPRFβ
    GPT-3.5LCS59.86159.82
    GPT-3.5Rank53.52352.8
    GPT-3.5Rank+IF37.5333.7
    GPT-4LCS46.75046.8
    GPT-4Rank64.74464.4
    GPT-4Rank+IF70.01467.3
    • The omitted GPT-3.5 Prob and Prob+IF have Fβ of 50.5 and 36.7; GPT-4 has 57.7 and 53.7. Majority in the same table has Fβ 50.3. GPT-4's LCS is 46.8.
    • The authors state that as of May 2024, the near-verbatim NYT reproductions from the litigation no longer occur, and suggest this may come from additional post-training or output filtering.
    • The authors state that verbatim prompts work better on GPT-3.5, with few correct surprisal-token guesses; on GPT-4, surprisal probes have fewer false positives. After knowledge filtering, GPT-3.5 recall drops to 3 and 4; the paper gives no additional reason.

    Dataset contamination

    The controlled experiment starts from GPQA, which the authors treat as uncontaminated, and fine-tunes GPT-3.5-turbo following Deng et al. (2024). Numbers are from Table 4.

    Swipe horizontally to read the table

    MethodUncontaminated EMContaminated EMΔ#Tokens
    TS-SLOT38.84%88.84%50%448
    Prob16.96%89.51%72.55%448
    Rank12.28%66.07%53.79%448
    Prob+IF6.59%84.82%78.23%258
    Rank+IF7.25%78.64%71.39%207
    • The authors state that all surprisal approaches have greater discriminability than TS-SLOT, with low-probability tokens plus instruction filtering having the largest difference.
    • Two existing test sets are then probed (Table 5). CommonsenseQA EM is: Llama-2-70B 0.96%/0.0%, GPT-3.5 6.73%/2.33%, GPT-4 4.81%/1.16%. Before the slash is the low-probability probe, after the slash is the filtered probe; probe counts are 104/86. The authors state they find essentially no evidence of contamination.
    • ARC EM is: Llama-2-70B 11.43%/1.96%, GPT-3.5 24.76%/6.0%, GPT-4 22.86%/13.73%; probe counts are 105/51, 105/50, 105/51. The authors state they see slight signs of contamination on GPT-4-0613 and cite high-surprisal tokens such as "photosynthesis" and "HCL" being correctly filled in.

    Complementarity, scale, and probability upper bound

    • The authors state that prefix probing, Person, and Prob identify different memorized samples, so they are suitable as a complementary suite. Figure 2 shows the overlap of verbatim prefixes and surprisal (person, prob) on GPT-4 fictional text; the main text gives no overlap counts.
    • Figure 3 compares the ratio of high-surprisal tokens recovered by Llama-2 7B, 13B, and 70B. The authors report that 70B recovers roughly 7 times more than 7B and believe probes are more likely to work on larger models. The specific ratios at each point in the figure are not separately listed in the main text.
    • Table 6 is only for Llama-2-70B with token probability access, on fictional text. Min 5% has P/R/Fβ of 99.5/10.0/91.4, PPL 96.5/9.7/88.6, Person 75.8/29.6/74.6. The authors state that probability-based methods are roughly 17 Fβ points apart from surprisal methods and advocate probability-based membership inference when probabilities are available.

    Other ablations and analyses

    • Appendix D, GPT-3.5, BookMIA 600 entries: DE-COP-Passage has P/R/Fβ of 58.38/63.11/58.43, Person 85.47/47.57/84.8, Majority 51.5/100/51.75.
    • Table 10: TS-SLOT keeping only nouns, adjectives, and verbs has #Tokens 442, EM 38.91% and 88.91%, Δ still 50%; the reconstruction column is the same as Table 4.
    • Other Min-K in Table 6: 10% has Fβ 90.5, 20% 90.0, 30% 89.3, 40% 88.9; PPL/ZLib is 90.7.
    • Appendix C: The authors compare cost by token price and state that DE-COP is more expensive when passages are longer than 15 tokens; this estimate does not include Claude rewriting and does not report measured dollar costs.
  5. What are some points that can be further explored?

    The authors note that membership is vaguely defined, the method relies on memory, and closed-source variance and post-hoc filtering hinder reproducibility.

    Limitations and future directions noted by the authors

    • Vague membership definition: Whether paraphrases count as members, whether derivative works count, and how much memorization counts are unclear; the authors state they are not certain how close the identified text is to training samples (Section 9).
    • Reliance on memory: If the model has not memorized the training data, the method does not work. The authors believe reconstruction probes may only work for large, strong instruction-tuned models and suggest future probes that do not rely on memory, such as calibrating recovery rates with rare or common tokens (Section 9).
    • No metadata used: The experiments do not use metadata such as the authors of fictional passages; the authors believe identification accuracy could improve with it (Section 9).
    • Use of existing datasets: There may be time gaps or frequency differences between members and non-members (Section 9, citing Duan et al., 2024).
    • Reference model: Its knowledge may differ from the target model, or it may already have been trained on the probed samples, affecting the distribution of high-surprisal tokens. The text may also be too ordinary to find such tokens, or spurious high-surprisal tokens may appear due to reference model limitations (Section 9).
    • Reproducibility and future work: Closed-source models have variance, and companies can use post-training to evade the method; when proprietary changes are not disclosed, using probes on future models may be difficult (Section 9). Section 6 hopes for multiple complementary probes and studies how training samples affect downstream behavior. Appendix H proposes finding surprisal tokens with domain phrases, n-grams, or human annotation, and extending to spans or structures.

    Experimental coverage

    • Main experiments target gpt-3.5-turbo-0125, gpt-4-0613, and Llama-2-70B; Figure 3 additionally reports token recovery ratios for Llama-2 7B, 13B, and 70B (Section 4, Section 5).
    • Fictional text uses the BookMIA test set of 8k entries and a tuning set of 1870; the NYT experiment uses 100 articles from Exhibit-J plus 100 CNN 2023 articles, and reports only GPT-3.5 and GPT-4 (Sections 4.1, 4.2).
    • Contamination experiments first fine-tune GPT-3.5-turbo on GPQA, then probe the test sets of CommonsenseQA and ARC-Challenge; all three target models have EM in Table 5 (Section 4.3).
    • Baselines include LCS prefix probing, TS-SLOT, Random, and Majority; membership inference with token probability access is compared only on Llama-2-70B; DE-COP is compared on 600 BookMIA entries (Table 2, Table 6, Appendix D).
    • The paper does not report the number of repetitions or sampling temperature for the main experiments. The reference model is BERT (110M); the NYT filter uses Mistral-V2 and Alpaca-7B, and the contamination experiment filter is Mistral-V2 (Section 4, Appendix A).
  6. Summarize the main content of the paper

    Black-box cloze probes use the reconstruction rate of high-surprisal tokens to identify memorized text, with Fβ higher than prefix probing on fictional text.

    Information-guided probes, given only inputs and outputs, use the cloze reconstruction rate of high-surprisal tokens to determine whether a closed-source model has memorized a piece of text.

    • Approach: A low-capacity reference model BERT selects hard-to-predict tokens; optionally, an instruction-tuned model filters out tokens guessable from context; then the target model fills in each blank in turn. On fictional text, filling in at least two correctly counts as memorized. Positive labels come from the dataset setup, not from independently verified training-set membership.
    • Fictional text: On the BookMIA test set, GPT-4's Person probe has Fβ=0.1 of 82.7 and LCS of 56.9; GPT-3.5 has 82.5 vs. 53.4; Llama-2-70B has 74.6 vs. 52.9 (Table 2). Prob and Rank often have much lower recall.
    • News: In the NYT 100-article comparison, GPT-4's Rank with knowledge filtering has precision 70.0 and LCS 46.7; on GPT-3.5, LCS has Fβ 59.82, higher than the surprisal probes (Table 3). The authors state that as of May 2024 these articles are no longer reproduced verbatim and suggest this may relate to post-training or output filtering.
    • Contamination: After fine-tuning GPT-3.5-turbo on GPQA for controlled contamination, low-probability tokens with filtering go from 6.59% to 84.82% EM, a difference of 78.23%, while TS-SLOT's difference is 50% (Table 4). The authors state that on CommonsenseQA they see essentially no contamination; on ARC, GPT-4 has EM of 22.86% and 13.73% (Table 5).
    • Upper bound and scale: On Llama-2-70B, Min 5% with access to token probabilities has Fβ 91.4; the authors state this is roughly 17 Fβ points apart from the surprisal methods (Table 6). The authors also report that Llama-2 70B recovers roughly 7 times more high-surprisal tokens than 7B (Figure 3).
    • Authors' conclusion: Different probes identify different samples, so a suite of methods is needed. The authors hope this improves data transparency in the LLM ecosystem and gives data contributors more agency when interacting with systems.

Related papers

Both use: ARC-Challenge
Read originalarxiv.org