Extracting memorized fragments of copyrighted books from open-weight language models

1. 01What problem does this paper attempt to solve?

   The authors attempt to quantify the specific degree to which open-source LLMs memorize particular copyrighted books, assessing the legal and technical risks of constituting infringing copies or derivative works.

   The paper attempts to solve how to precisely measure the verbatim memorization of specific copyrighted books by open-source large language models, and to clarify its practical impact on copyright litigation and legal policy.

   • Litigants' technical claims are oversimplified: Plaintiff copyright holders accuse LLMs of being infringing copiers, while defendant tech companies argue that models only contain abstract statistical associations and do not copy specific works; the authors believe the reality is more complex than these two polarized claims.
   • Existing extraction metrics cannot correspond to copyright infringement elements: Previous research often calculates average extraction rates on randomly sampled sequences, but copyright law assesses infringement against specific works themselves; average metrics obscure a model's complete memorization of individual books.
   • Book memorization is highly specific: Models do not memorize all training data; the degree of memorization varies significantly across models and books; some works may be substantially or even completely memorized.
   • Defining infringement boundaries for model weights and generated outputs: It is necessary to clarify whether protected expression encoded internally in model weights constitutes a "copy" or "derivative work" under copyright law, and under what prompt conditions users can actually reconstruct entire books in outputs.

   Collapse this answer

2. 02What related research exists?

   The paper reviews research at the intersection of verbatim extraction, probabilistic extraction, and copyright law; the authors state that existing work focuses on average extraction rates without quantifying memorization of individual specific works.

   Group 1: Greedy decoding-based verbatim extraction studies

   • Carlini et al. (2021, 2023) and Lee et al. (2022) — propose methods that prompt the model with a prefix from training sequences and check whether the greedy decoding suffix exactly matches; report that larger models and more repeated training are associated with more memorization. The authors note that such methods give only binary judgments and that reliance on greedy decoding underestimates extraction risk under non-deterministic sampling.

   Group 2: Non-deterministic decoding and probabilistic extraction studies

   • Hayes et al. (2025) — propose the probabilistic discoverable extraction metric ((n, p)-discoverable extraction), computing the theoretical probability that the model verbatim generates a target suffix under stochastic decoding schemes like top-k. The authors inherit this probabilistic framework but adapt it to require only a single forward pass to read logits, greatly reducing computational cost.

   Group 3: Generative AI copyright and fair use studies

   • Cooper and Grimmelmann (2024) and Lee et al. (2023, 2024) — discuss from a copyright law perspective whether model weights themselves satisfy the statutory definition of "copy," whether extracted outputs constitute substantial similarity, and the applicability boundaries of transformative fair use in open-source distribution scenarios.
   • Judicial precedents and industry views — Initial rulings in Kadrey v. Meta and Bartz v. Anthropic suggest that training use may constitute fair use; defendants often cite Author's Guild v. Google (2015) arguing that models are similar to search engines displaying snippets. The authors note that models can extract entire book-length texts, which differs from purely internal indexing.

   Group 4: Baseline methods and benchmark datasets

   • Baseline methods and benchmarks: Greedy decoding discoverable extraction and (n, p)-discoverable extraction serve as technical baselines; evaluation benchmarks include the Books3 corpus with nearly 200,000 books (sampled 200 books) and the Enron dataset.
   • Positioning difference from existing work: The authors state that existing work mostly measures average extraction rates on random samples, whereas this paper establishes sliding window and coverage metrics to quantify the model's specific memorization of individual books and the possibility of whole-book reconstruction.

   Collapse this answer

3. 03How does the paper address this problem?

   The authors propose a sliding-window-based single forward pass probability extraction algorithm, define an extraction coverage metric, and use beam search to achieve whole-book reconstruction from a single seed prompt.

   The authors propose a sliding window probabilistic extraction framework, which uses overlapping slices and a single forward pass to compute the verbatim extraction probability distribution for a single book, and establishes an extraction coverage metric to quantify the proportion of the book memorized; for highly memorized books, combined with a beam search autoregressive process to achieve long-text reconstruction.

   Sliding Window Slicing and Probabilistic Extraction Process

   • Overlapping sliding window slicing: For the text to be tested in Books3, start from the beginning, take a 100-token window (50-token prefix plus 50-token suffix), then slide forward by 10 characters and repeat; each book generates a test sequence every 10 characters on average.
   • Single forward pass computation: Using teacher forcing, feed the 100-token sequence into the model for a single forward pass to obtain logits, avoiding the computational cost of multiple autoregressive generations (Appendix A.3).
   • Suffix probability formula: Under decoding scheme ϕ (top-k decoding with temperature T=1, truncation k=40), the conditional probability of the 50-token suffix z_a+1:a+j of sequence z is defined as the product of conditional probabilities at each position:

   p_z \triangleq \prod_{t=a+1}^{a+j} \Pr_{\theta,\phi}(z_t \mid z_{1:t-1}) = \exp\left(\sum_{t=a+1}^{a+j} \log \Pr_{\theta,\phi}(z_t \mid z_{1:t-1})\right) This formula represents the joint probability that the model exactly matches the target suffix token-by-token given the prefix.

   • Extraction success determination: Set a conservative threshold τ_min = 0.1%; if p_z ≥ τ_min, the sequence is considered successfully extracted. The geometric mean conditional probability per suffix token is approximately 87.1%.

   Extraction Coverage Metric

   • Mathematical definition of coverage: To quantify the proportion of a book memorized, the authors define the extraction coverage metric:

   \mathrm{coverage}(\tau) = \frac{1}{L} \left| \bigcup_{z: p_z \ge \tau} \mathrm{span}(z) \right| where L is the total number of characters in the book, and span(z) is the set of character indices covered by the 50-token suffix with extraction probability not below threshold τ.

   • Design rationale and properties: By computing the proportion of characters covered by the union of suffixes meeting the condition, duplicate counting from highly overlapping sliding windows is eliminated, reflecting the actual breadth of the book memorized with high probability at the character level (Section 4.2).

   Negative Controls and Measurement Validity Verification Design

   • Context length extension test: On known training books, test prefix lengths from 25 to 800 tokens (suffix remains 50 tokens), observing whether longer prefixes consistently increase conditional probability (Section 5.1).
   • Dual negative controls: One uses PHI 4, which was not trained on full copyrighted books, to test Books3 books; the other uses LLAMA series models to test 4 new books published in 2025 after the July 2024 training cutoff, verifying that non-training data does not falsely report high-probability extraction (Section 5.2).

   Single Seed Prompt Long-Text Reconstruction Process

   • Beam search sliding window autoregression: Use a short opening text of the book as the initial seed prompt, with beam width 8, maximum context window 3000 tokens, and generating 50 tokens each time (Section 7).
   • Chapter EOS marker handling mechanism: The model tends to output EOS at chapter ends; the process detects EOS, removes it, and replaces it with formatted next chapter title text (e.g., "CHAPTER TWO") to encourage the model to continue autoregressive generation of the next chapter.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The paper completed over 3000 experiments; most models have coverage <2% on most books, but LLAMA 3.1 70B reaches 96.27% on Harry Potter.

   Experimental Setup

   • Models under test: A total of 14 open-weight completion models, including LLAMA 1 (13B, 65B), LLAMA 2 (7B, 13B, 70B), LLAMA 3 (8B, 70B), LLAMA 3.1 (8B, 70B) and PYTHIA 12B, which are confirmed to have been trained on Books3; DEEPSEEK V1 67B, QWEN 2.5 72B and GEMMA 2 27B, which are suspected of using related corpora; and the negative control PHI 4 (14B) (Section 6.1).
   • Evaluation corpus and scale: 200 books were selected from the approximately 200,000 books in Books3 (about 0.1%), of which 100 were randomly sampled and another 100 were manually selected (covering works by plaintiffs in the Kadrey lawsuit, well-known bestsellers, public domain works, etc.); the negative control uses 4 new books published in 2025 (Table 3 and Table 5).
   • Baselines and decoding settings: Greedy decoding extraction is compared with probability-based extraction based on top-k (T=1, k=40); the full-book sliding window splits 100-token sequences with a 10-character step.
   • Metrics and decision criteria: The probability extraction metric pz is used, with decision threshold τ_min = 0.1%; extraction coverage coverage(τ) is computed; long-text reconstruction uses word-level and sentence-level longest common substring (LCS) match and TF-IDF cosine similarity.
   • Sample sizes and repetitions: The full-book sliding window experiments were evaluated on 200 books over 394 GPU-days, totaling more than 3000 experiments; the average extraction rate experiments took 10 sequences from each of 4000 books (40,000 sequences per run), repeated 3 times for a total of 120,000 sequences.

   Main Results

   Swipe horizontally to read the table

   Book	LLAMA 1 65B	LLAMA 2 70B	LLAMA 3.1 70B	QWEN 2.5 72B	PHI 4
   Harry Potter and the Sorcerer’s Stone	34.83%	36.41%	96.27%	58.21%	1.73%
   Nineteen Eighty-Four	67.27%	71.21%	81.19%	30.59%	5.09%
   The Great Gatsby	73.60%	71.82%	88.77%	26.86%	8.71%
   The Alchemist	11.90%	10.75%	49.06%	5.84%	1.04%
   Beloved	1.87%	3.58%	12.54%	1.55%	0.32%
   Sandman Slim	0.32%	0.32%	0.43%	0.38%	0.25%

   According to Table 4 (book extraction coverage at τ=0.1%, some models omitted).

   • Memorization shows a clear long-tail and specific distribution: The authors state that for most books and models, extraction coverage is below 2%, but on a very small number of popular books it shows high memorization, with LLAMA 3.1 70B reaching 96.27% coverage on Harry Potter.
   • Differences exist across organizations at the same size and across generations of the same series: The authors point out that even at comparable scale and with all having been exposed to related corpora, LLAMA 1 65B and QWEN 2.5 72B have coverage of 34.83% and 58.21% on Harry Potter respectively; within Meta's series, LLAMA 3.1 generally memorizes more than LLAMA 2 and LLAMA 1.
   • Plaintiff books generally have low memorization but with individual cases: Among the books involved in the Kadrey lawsuit, most such as Sandman Slim have coverage below 0.5%, but Coates's We Were Eight Years in Power reaches 14.26% coverage under LLAMA 3.1 70B (Table 4).

   Single-Seed Prompt End-to-End Book Reconstruction Experiment

   • Reconstruction method and setting: LLAMA 3.1 70B was selected, with only the first 6 tokens of the first line of Harry Potter's first chapter ("Mr. and Mrs. D") as the initial input, combined with beam width 8 and a 3000-token context to perform sliding generation (Section 7).
   • Experimental results: The authors report that within 9 attempts the entire 304-page book was successfully reproduced almost verbatim; after removing non-body content at the beginning and end, the word-level greedy LCS match with the original Books3 version reaches 0.992, the sentence-level LCS reaches 0.934, and the TF-IDF cosine similarity reaches 0.9999.
   • Differences and edition analysis: The authors' analysis points out that the differences between the generated text and the original book are mainly in typesetting and British/American spelling; some word and sentence differences that appear to be "hallucinations" were verified to be the original text of early print editions before 2004 (Appendix F.1).

   Negative Controls and Measurement Validity Verification

   • What was measured: The sliding-window extraction procedure was run on PHI 4, which was not trained on copyrighted books, and on 4 new books published in 2025 after the training cutoff (Section 5.2 and Appendix E.4).
   • Results: On the 2025 new book Great Big Beautiful Life, apart from the copyright notice, no sequence has p_z ≥ 0.1%; among about 62,000 sequences under 200- to 800-token prefixes, only 1 has p_z ≥ 10⁻¹⁴%; PHI 4 on Books3 books exceeds the threshold only at widely repeated public famous lines on the internet such as Bible quotations and Obama's speeches.
   • The authors' interpretation: The authors believe this proves that τ_min=0.1% is sufficiently conservative, that what the experiments capture is genuine memorization rather than coincidental generation, and that the extraction probability measurement is valid.

   Ablation Experiment on Context Prefix Length

   • What was measured: On 4 Books3 books, the prefix length was extended from 25 to 800 tokens (suffix fixed at 50 tokens) to test changes in extraction coverage of models such as LLAMA 3.1 70B (Section 5.1 and Appendix E.3).
   • Results and interpretation: As the prefix grows, extraction coverage gradually rises and levels off (Figure 6), for example Beloved reaches close to 80% coverage under an 800-token prefix, but on This Is How You Lose Her it remains extremely low throughout; the authors state that longer prefixes can reveal more latent memorization but cannot conjure extraction of content that was not memorized.

   Other Ablations and Analyses

   • Decoding scheme ablation: greedy decoding (k=1) has a lower extraction rate than non-deterministic top-k sampling (left of Figure 3 and Appendix C).
   • Step size ablation: increasing the sliding window from 1 character to 10 characters causes a slight signal loss but reduces computational cost by 10 times (Appendix D.2).
   • Floating-point precision ablation: float16 and float32 show no significant difference in extraction rate on Enron data (Table 1 and Appendix B).
   • BOS token ablation: after adding a BOS token before the sequence for the PYTHIA model, the maximum probability extraction rate rises slightly from 5.27% to 5.52% (Table 2).
   • Failed reconstruction test: running the full-book reconstruction procedure on the low-memorization book Sandman Slim, the model fell into a repetition loop and failed to reproduce the body text (Appendix F.3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The limitations noted by the authors include that the sample covering 200 books cannot represent the whole, the global threshold conservatively underestimates memorization, and whole-book reconstruction is sensitive to configuration.

   Limitations and future directions noted by the authors

   1. Limited book and model sample size: The authors note in Section 6.1 and Section 8.1 that the experiments evaluated only 200 books out of approximately 200,000 in Books3 (about 0.1%) and 14 open-source models, a sample size insufficient to generalize to the overall memorization of all books or all open-source models.
   2. Unified threshold may underestimate memorization: The authors note in Section 5.3 and Appendix E.1.1 that the global fixed conservative τ_min=0.1% biases measurements toward a lower bound of memorization; future work needs to calibrate specific thresholds for different models to further lower τ_min.
   3. Feasibility boundaries of full-book reconstruction not fully explored: The authors note in Section 7 and Appendix F that single-seed prompt reconstruction of an entire book was only preliminarily validated as an existence proof on extreme high-memorization cases like Harry Potter; the method is sensitive to settings such as step size, and future research is needed to improve reconstruction stability and extend to other texts.
   4. Inability to definitively determine full-book attribution for models with undisclosed training sets: The authors note in Appendix E.6.4 that for models like DEEPSEEK and QWEN with undisclosed training sets, successful extraction only proves that the specific extracted sequence belongs to the training set, and cannot assert that the entire book was included in training.
   5. Causal attribution of training data influence not explored: The authors note in Appendix E.6.5 that extraction metrics cannot measure the potential influence of texts not verbatim memorized on model outputs and representations; causal attribution of training data requires different technical approaches.

   Experimental coverage

   1. Scope of tested models: The tested subjects cover 14 open-weight completion language models (7B to 72B), excluding commercial closed-source API models or Chat/Instruct models fine-tuned for dialogue.
   2. Test book coverage: The evaluation corpus comes from 200 books in Books3 and 4 newly published books from 2025, not including other shadow libraries or non-English books.
   3. Evaluation task settings: The main measurement is verbatim extraction of 50-token suffixes guided by 50-token prefixes; the paper does not report extraction risks for non-verbatim semantic paraphrasing.
   4. Defense mechanism testing: The evaluation targets unprotected base model weights; the paper does not report mitigation effects of defenses such as alignment, output filtering, or machine unlearning.
   5. Computation and reconstruction budget: Long-text reconstruction was only deeply tested on 3 works: Harry Potter, We Were Eight Years in Power, and Sandman Slim, not fully extended to all 200 books.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that open-source LLMs exhibit clear book- and model-specific memorization of books, and entire books can be reconstructed from a single prompt, complicating polarized claims in copyright litigation.

   This study measures the degree to which model weights memorize specific copyrighted books and their legal implications through large-scale verbatim extraction measurements on open-source large language models.

   • Research motivation and core questions: Addressing the polarized technical claims of plaintiffs and defendants in AI copyright litigation, existing average extraction rate studies cannot answer the core legal dispute of whether models memorize specific copyrighted works.
   • Method and metric innovation: Designed a sliding window single forward pass algorithm and extraction coverage metric, detecting full-book extraction hotspots with a 10-character step, using τ_min=0.1% as a conservative threshold to determine the lower bound of memorization.
   • Core experimental findings: Extraction coverage varies significantly across models and books; most models have coverage <2% on most books, but LLAMA 3.1 70B reaches 96.27% on Harry Potter and 81.19% on 1984 (Table 4).
   • Full-book end-to-end reconstruction validation: Using LLAMA 3.1 70B with beam search and chapter EOS replacement logic, the entire 304-page book is reconstructed from only a 6-token first-line input, with word-level LCS similarity of 0.992 (Section 7).
   • Validity and control verification: On PHI 4, not trained on copyrighted books, and new books published in 2025, unique content shows no extraction signals above τ_min, ruling out interference from coincidental generation (Section 5.2).
   • Legal and policy implications: The authors believe the results support that some models may constitute derivative works of some high-memorization books, complicating defendants' fair use defenses; also, because memorization varies greatly per book, it poses obstacles to plaintiffs' class actions (Section 8).

   Collapse this answer

Related papers