Skip to content

Extracting alignment data from open models

Extracting alignment data in open models

arXiv· arXiv:2510.18554· ICML 2026· Federico Barbero, Xiangming Gu, Christopher A. Choquette-Choo, Chawin Sitawarin, Matthew Jagielski, Itay Yona, Petar Veličković, Ilia Shumailov, Jamie HayesUniversity of Oxford; National University of Singapore; Google DeepMind; Anthropic; MentaLeap; AI Sequrity CompanyIndexed by this site · 23 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors state that SFT/RL alignment data can be extracted from open-source models using a chat template prefix, that string matching underestimates the memorization rate by at least 10×, and that retraining on the extracted data can recover part of the performance.

Threat modelThe attacker's goal is to extract the alignment (post-training) training data of open-weight models; knowledge is open-weight, the user controls tokenization and the chat template prefix; capability is repeatedly sampling with the chat template prefix and using embedding vector retrieval to match the training set; the victim system is open models with public training mixtures (OLMo 2, Open-Reasoner-Zero). The attack does not directly apply to closed models.

Questions
The alignment training data (SFT/RL data) of open-source models is a source of competitive advantage; the authors study whether models memorize and emit this data, and whether string-matching metrics underestimate this leakage.
Methods
Exploiting the fact that chat template special tokens are introduced only during post-training, it repeatedly samples with the template prefix, then uses gemini-embedding-001 to compute embedding similarity and perform vector retrieval over the training set, judging approximate semantic memorization.
Experiments and results
The authors report that under string matching the memorization rate appears negligible, whereas under embeddings it is much higher (they state it is underestimated by at least 10×); after SFT of OLMo 2 7B on about 930k extracted samples, most benchmarks are close to the original model, with only IFE clearly lower; RL training samples are also reproduced verbatim.
Limitations and future work
The authors state that the attack relies on the user controlling the chat template and applies only to open-source models, leaving the threat on closed-source models for future work; the experiments cover only two model families, OLMo 2 (SFT) and Open-Reasoner-Zero (RL), and the embedding threshold of 0.95 was chosen manually.
Experimental setupOLMo 2 13B、Open-Reasoner-Zero 7B et al. · OLMo 2 SFT dataset (939k samples)、ORZ 57k dataset et al. · embedding similarity score、normalised Levenshtein similarity et al.
Evaluated models
OLMo 2 13BOpen-Reasoner-Zero 7BQwen2.5 7B InstructOLMo 2 7BQwen2.5 7B
Benchmarks
OLMo 2 SFT dataset (939k samples)ORZ 57k datasetBBHMMLUMATHGSM8KPOPQATQAIFEDROPAMCMATH500Minerva MathOlympiadBench
Metrics
embedding similarity scorenormalised Levenshtein similaritynormalised Indel similaritymemorisation ratecoveragebenchmark accuracylikelihood
AI summaryThe researchers demonstrate a method for extracting large amounts of alignment training data from post-trained open models. The authors argue that high-quality embedding models identify semantic similarity better than metrics such as edit distance, and that approximate string matching conservatively underestimates the amount of extractable data by about 10 times due to trivial artifacts. The study finds that models recite data used in the SFT or RL post-training stages, and that this data can be used to train base models and recover a considerable portion of the original performance. The authors state that this exposes a potentially overlooked risk of extracting alignment data, and note that distillation can be viewed as indirectly training on the model's original dataset.

The researchers demonstrate a method for extracting large amounts of alignment training data from post-trained open models. The authors argue that high-quality embedding models identify semantic similarity better than metrics such as edit distance, and that approximate string matching conservatively underestimates the amount of extractable data by about 10 times due to trivial artifacts. The study finds that models recite data used in the SFT or RL post-training stages, and that this data can be used to train base models and recover a considerable portion of the original performance. The authors state that this exposes a potentially overlooked risk of extracting alignment data, and note that distillation can be viewed as indirectly training on the model's original dataset.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The study investigates whether open-source models leak alignment training data, and whether string-matching metrics underestimate such leakage.

    Core question: Do open-weight models memorize and emit their post-training (alignment) data, making this competitively valuable data extractable.

    • Problem setting and importance: The authors argue that model developers gain a competitive advantage from carefully curated alignment data (SFT and RL data, as well as targeted data for math, reasoning, long context, etc.); if models memorize and reproduce this data, that advantage itself is at risk. This also bears on distillation practice: when training one's own model on a strong model's outputs, one may indirectly train on the original data.
    • Shortcomings of existing methods: The authors point out that prior memorization research mostly measures extraction success by exact or approximate string matching, focusing on privacy and copyright risk; such metrics are not aligned with the extraction goal that "semantic structure is equally valuable" and miss samples that are semantically equivalent but literally different.
    • Core observation and hypothesis: The chat template and its special tokens are introduced only during post-training; the authors hypothesize that prompting the model with the template as a prefix will generate alignment data, and that the semantic closeness of the generations to the training set far exceeds expectations.
    • What the paper proposes: An extraction pipeline (template-prefix sampling + embedding vector retrieval matching) that uses embedding similarity instead of string matching to measure "approximate semantic memorization," and verifies that the extracted data can be used directly to retrain another model.
    • Threat model:
      • Goal: Extract alignment training data (SFT question-answer pairs, RL prompts, etc.) from post-trained open-source models.
      • Knowledge: Open-weight models, with tokenization controlled by the end user; the paper states the attack does not directly apply to closed-source models.
      • Capability: Repeated sampling with the chat template start token as prefix (temperature default 1), and vector retrieval over the training set using an embedding model.
      • Victim systems: Open-source models that publish their post-training mixture (OLMo 2, Open-Reasoner-Zero).
  2. What related research exists?

    Related work covers four areas: verbatim memorization and extraction, measuring non-verbatim memorization, alignment data synthesis (Magpie), and distillation.

    Verbatim memorization and training data extraction

    • Carlini et al. (2021, 2022), Nasr et al. (2023, 2025)—study verbatim memorization and regurgitation of training data by LLMs, propose metrics such as discoverable extraction, and mainly focus on privacy and copyright risks. The paper notes that such work judges success by string matching, which differs from this paper's goal of semantic extraction.
    • Hayes et al. (2025b)—propose probabilistic discoverable extraction, replacing Boolean judgments with the joint likelihood of multiple queries.

    Non-verbatim memorization metrics

    • Ippolito et al. (2022)—note that preventing verbatim memorization gives a false sense of security, and that "style-transfer" can bypass string matching; this paper cites it as motivation when extending the definition of memorization.
    • Shilov et al. (2024)—propose mosaic memory, in which the model can stitch together partially overlapping fragments; Wang et al. (2024)—propose distributional memorisation and state that reasoning tasks rely more on generalization; Huang et al. (2025)—propose the Entropy-Memorisation Law. The authors state that the memorization rate measured in this paper under the embedding definition is far higher than these works might have expected, and that, unlike Wang et al., this paper finds that in reasoning tasks the model also clearly exploits memorization of training data.

    Alignment data synthesis and chat template

    • Xu et al. (2024) (Magpie)—use chat template prefixes to prompt aligned models to synthesize alignment datasets, and conjecture that the generated content may be training data. This paper generalizes their prompting strategy and answers this open conjecture from the perspective of memorization.

    Model distillation

    • Hinton et al. (2015)—propose knowledge distillation. The authors reframe distillation as a memorization problem: if the teacher model can regurgitate its training data, the student model indirectly comes into contact with that data.

    Baselines and benchmarks: The baseline in Table 1 is a single-token prefix using only <|endoftext|>; the baselines in Table 2 are the SFT checkpoint released by OLMo et al. (2024) and a version reproduced by the authors with the original data, with benchmarks BBH, MMLU, MATH, GSM8K, POPQA, TQA, IFE, DROP; the baseline for the RL part is a model trained with Dr. GRPO on the ORZ 57k dataset, with benchmarks AMC, MATH500, Minerva Math, OlympiadBench.

    The authors position this paper at the intersection of three areas: alignment, memorization and distillation, and state that it reveals that distillation can be regarded as indirect training data extraction.

  3. How does the paper address this problem?

    Use chat template prefixes to induce the model to emit alignment data, then use embedding similarity to retrieve matches in the training set to measure memorization.

    Overall approach: exploiting the fact that chat template special tokens are only introduced during post-training, repeatedly sampling with the template start token as a prefix, then using an embedding model to semantically match generated samples against the training set, thereby extracting and measuring alignment data leakage. The paper does not give this pipeline a dedicated name.

    Extraction pipeline

    • Prefix sampling: generate samples using the chat template's start token as a prefix. The SFT experiment uses <|endoftext|><|user|>\n, the RL experiment uses the conversation template prefix released by the model developer (Appendix C gives the full prefix); temperature kept at the default value of 1.
    • Vector retrieval: use an embedding model to embed the entire post-training set into a vector search engine; embed each generated sample and take the similarity argmax, retrieval can be implemented as a single matrix multiplication.
    • Decision: similarity above the threshold counts as (approximate semantic) memorization.

    Measuring approximate semantic memorization

    • Embedding Score: use gemini-embedding-001 to generate a single embedding for each sample (removing all special tokens, keeping only plain text), embeddings normalized to unit norm, dot product is the similarity.
    • Threshold: manually chosen 0.95, which the authors call the more conservative choice; samples near this threshold would be indisputably judged as memorization under reasonable human evaluation; Appendices D and E give random samples at different thresholds.
    • Control metrics: normalised Levenshtein similarity (1 − Levenshtein(A,B)/max(len(A),len(B)), i.e. edit similarity normalized by the longer string's length) and normalised Indel similarity; following Comanici et al. (2025), a 0.9 threshold is used to judge approximate memorization.

    SFT data extraction (Section 4)

    • Subject: OLMo 2, whose SFT-stage dataset contains 939k samples (Appendix A writes 939,344 question-answer pairs); the authors say it was chosen because the full training mixture and details are public.
    • Scale: the string-matching experiment uses 100k generated samples; the embedding experiment generates 1M samples.
    • Coverage analysis: for each training sample, report its maximum embedding score against the 1M generated samples.

    RL data extraction (Section 5)

    • Subject: Open-Reasoner-Zero 7B (trained from Qwen 2.5 base with PPO, training data public); RL samples contain only question-answer pairs, reasoning traces are rollout products, so only the question-answer part is extracted.
    • Likelihood analysis: compare the likelihood of Qwen 2.5 base and the post-RL model for each PPO training prompt, bucketed by order of magnitude.

    Retraining validation of extracted data

    • SFT: collect about 930k synthetic samples (about 850M tokens, the original SFT set is about 1.3B tokens), apply basic filtering with Gemini 2.5, SFT-train OLMo 2 7B.
    • RL: first train Qwen2.5 7B base with Dr. GRPO on the ORZ 57k dataset to get the Baseline, then generate 100k samples from the Baseline, filter with Gemini 2.5 and randomly take 57k, train Qwen2.5 7B base.
  4. What experiments did the paper conduct?

    Measure extraction rates, differences between string matching and embedding, and the effect of retraining on extracted data, on OLMo 2 and Open-Reasoner-Zero.

    Experimental setup

    • Models tested: OLMo 2 13B (SFT extraction), OLMo 2 7B (SFT retraining), Open-Reasoner-Zero 7B (RL extraction), Qwen2.5 7B base (RL retraining and likelihood comparison), Qwen2.5 7B Instruct (qualitative example in Figure 2).
    • Data: OLMo 2 SFT set of 939k samples; ORZ 57k dataset; Open-Reasoner-Zero training set of 142,770 samples (Figure 8).
    • Generation scale: 100k generations for the string-matching experiments; 1M generations for the embedding experiments; 1,000 each for the prefix comparison experiments; 100k generations for RL extraction.
    • Decision method: gemini-embedding-001 computes embedding similarity, threshold 0.95 (manually selected); string-matching threshold 0.9 (following Comanici et al. 2025).
    • Evaluation benchmarks: SFT retraining uses the 8 benchmarks and evaluation pipeline of OLMo et al. (2024); RL retraining uses AMC, MATH500, Minerva Math, OlympiadBench.
    • Number of repetitions: not reported in the paper.

    Main results

    SFT retraining (OLMo 2 7B, per Table 2):

    Swipe horizontally to read the table

    ModelBBHMMLUMATHGSM8KPOPQATQAIFEDROP
    Baseline (SFT only)0.49530.61330.20730.74070.23640.48580.65620.5960
    Baseline (Reproduction)0.49440.61230.20770.73770.25290.51100.66170.5945
    Extracted Data0.51610.60520.17050.78470.24900.55290.50280.5923
    • The authors state that the reproduction is close to the released SFT checkpoint, validating the experimental setup.
    • The authors state that the model trained on extracted data is comparable to the baseline on most benchmarks, but IFE is clearly lower; they suspect too few samples targeting that benchmark were generated by the pipeline, and believe mixing in targeted data could improve it.
    • The authors note that the synthetic set has fewer tokens (about 850M) than the original set (about 1.3B), because the filtered generations are shorter.

    String matching vs embedding

    • Under string matching the memorization rate appears negligible: Figure 4 shows that the Levenshtein/Indel similarity distribution of 100k generations is concentrated in the low range, and when judged by the 0.9 threshold the memorization rate appears negligible (the authors' wording).
    • Embeddings reveal a higher memorization rate: Figure 5 (1M generations) shows the embedding score distribution shifted right; the scatter plot shows that Levenshtein and embedding scores do not align, and the authors state that string matching has a length bias (longer texts score lower).
    • The authors state that string matching conservatively underestimates by at least 10× (abstract and contributions section), because trivial differences such as numbers and options lower the score; the example in Figure 3 has embedding 0.986 vs Levenshtein 0.699.

    RL data extraction and likelihood analysis

    • Verbatim reproduction: Figure 7 shows Open-Reasoner-Zero 7B outputting RL training samples' questions and answers verbatim, along with its own reasoning traces; the authors state this is counterintuitive, since the PPO objective appears not to align with memorizing training samples.
    • Likelihood increase: Figure 8 shows the likelihood of RL post-training prompts increasing, e.g., at the 1e-8 threshold from 287 for the base to 7,101, and at 1e-5 from 18 to 678; the authors state they found samples rising from 1e-11 to 1e-5, that the mechanism is not yet clear, and leave it as a future direction.
    • RL retraining: Figure 9 shows that training Qwen2.5 7B base with Dr. GRPO on the extracted 57k synthetic data is comparable on AMC, MATH500, Minerva Math, and OlympiadBench to the Baseline trained on the original ORZ 57k (the figure caption says "recover most of the performance"; no specific values are given in the converted text).

    Other ablations and analyses

    • Prefix comparison (Table 1, 1,000 generations each): the full chat template prefix gives an average best embedding score of 0.892, versus 0.857 for <|endoftext|> only; the authors believe this explains why Magpie-style techniques work.
    • Coverage (Figure 6): some training samples are easier to memorize than others; the authors' investigation finds that samples are easier to memorize if similar samples also appear in pre/mid-training data, or if multiple versions of the same sample appear frequently.
    • Appendix B: the authors propose a mathematical speculation of "accidental gradient alignment," arguing that even if the loss masks the question part, the conditional gradient update may still increase the likelihood of the question itself; the authors note this is only speculation.
  5. What are some points that can be further explored?

    The authors state that the attack applies only to open-source models, and leave the threat on closed-source models and the mechanism of RL memorization for future work.

    Limitations and future directions noted by the authors

    • Only applies to open models (Conclusion): the attack exploits the chat template; closed models force the template and users cannot control it; for closed models one would need to forge the behavior of these tokens, and the authors say future work will evaluate the degree of threat to closed models, citing Geng et al. (2025) to show this is not impossible.
    • RL memorization mechanism unclear (Section 5, Conclusion): the authors say the exact mechanism of the post-RL training prompt likelihood increase is unclear, left as a future research direction.
    • Extraction pipeline can be improved (Section 5): the authors believe more sophisticated pipelines could obtain higher-performance, higher-quality datasets, left as a future direction since this paper focuses on training sample extraction.
    • Threshold choice affects measurement (Section 3.3): the embedding threshold (0.95) is manually chosen, and the choice affects the measured memorization rate; the authors note the string-matching threshold has the same limitation.
    • Limitations of the likelihood metric (Section 5, Appendix B): single-sequence likelihood only measures "pointwise" memorization and may miss template-level memorization; the authors believe one should integrate over all reasonable sequences, but this quickly becomes infeasible.
    • Open questions on post-training memorization (Appendix A.2): Jagielski et al. (2022) found that data later in training is more easily memorized; the authors say whether this finding applies to post-training is an interesting open question.

    Experimental coverage

    • Models tested are OLMo 2 (7B, 13B), Open-Reasoner-Zero 7B, Qwen2.5 7B (base and Instruct), extraction experiments on two model families in total (OLMo 2 for SFT, Open-Reasoner-Zero for RL).
    • Extraction experiments require models to publish their post-training mixture, which the authors call a significant constraint of the work (Section 3).
    • Generation scale is 100k (string matching, RL extraction) and 1M (embedding experiment); prefix comparison uses 1,000 each.
    • The embedding model used is only gemini-embedding-001; filtering uses Gemini 2.5.
    • The paper does not report the number of experimental repetitions, nor consistency data for human evaluation (threshold reasonableness is supported only by qualitative examples in Appendices D and E).
  6. Summarize the main content of the paper

    The authors say open models recite alignment training data, the embedding metric reveals leakage at a scale far exceeding string matching, and the extracted data can be retrained on.

    An attack paper on training data extraction, studying whether open-weight models leak their competitively valuable alignment (post-training) data.

    • Problem: alignment data (SFT, RL, and targeted data such as math and reasoning) is a competitive asset for model developers; the authors study whether models memorize and recite this data, and argue that mainstream string-matching metrics are misaligned with semantic-level leakage.
    • Method: exploiting the fact that chat template special tokens are only introduced during post-training, repeatedly sample with the template prefix (e.g. <|endoftext|><|user|>\n), then use gemini-embedding-001 embedding similarity (threshold 0.95) to do vector retrieval in the training set, judging "approximate semantic memorization".
    • Metric comparison: on OLMo 2 13B, under string matching (0.9 threshold) the memorization rate appears negligible (Figure 4), while under embedding the distribution shifts markedly right (Figure 5); the authors claim string matching conservatively underestimates by at least 10×.
    • SFT retraining: SFT-train OLMo 2 7B on about 930k extracted samples, most benchmarks close to the original model (GSM8K 0.7847 vs baseline 0.7407), but IFE only 0.5028 (baseline 0.6562) (Table 2).
    • RL findings: Open-Reasoner-Zero 7B verbatim recites RL training samples; likelihood of post-RL training prompts rises (287 → 7,101 at the 1e-8 threshold, Figure 8); Dr. GRPO training on the extracted 57k data is comparable on four math benchmarks to the model trained on original data (Figure 9).
    • Authors' conclusion: model distillation can be viewed as indirect training data extraction, and the advantage open models gain from alignment data carries leakage risk; the attack only applies to open models, and the threat on closed models and the RL memorization mechanism are left for future work.

Related papers

Show 3 more papers
Read originalICML 2026