Identically trained language models are broken by different jailbreaks

1. 01What problem does this paper attempt to solve?

   Whether rerunning pretraining once on the same data and applying the same safety training will be broken on the same jailbreaks.

   Core question: to what extent does which jailbreaks can break a checkpoint depend on its pretraining run rather than its pretraining data.

   • Problem setting: jailbreak benchmarks and red teaming usually test only one checkpoint and treat "which jailbreaks can break it" as a property of its pretraining data and safety training (the authors' summary of existing work). The authors argue that this reading ignores the randomness of pretraining itself.
   • Gap in existing evidence: prior work has shown that training seeds change predictions, bias and safety (e.g., fine-tuning seeds changing harmfulness, sampling seeds flipping refusals), and some studies compare branches of the same run; the authors state that, to their knowledge, no study has tested whether a second full pretraining on the same data, after the same safety training, would be broken by the same jailbreaks.
   • Core observation: two post-trained copies of the same pretraining run (MZ twins) have a per-prompt non-refusal rate correlation of 0.984, which drops to 0.811 with a different pretraining seed and only further to 0.779 with a different data recipe; the authors conclude from this that the pretraining run matters far more than the data recipe.
   • Contributions: measure the above "ancestry gap" across 25 data recipes and 3 scales (150M/300M/1B); use same-seed retraining and partial retraining to separate the roles of seed, initialization, data order and hardware/software; and give two implications for single-checkpoint evaluators—low transfer of the best attack template, and per-prompt profiles that can "fingerprint" the pretraining run.
   • Nature of the work: this is a measurement and analysis study, proposing no new attacks or defenses; the systems tested are 150 open small models under controlled training, all prompts come from public benchmarks, and the authors state that they wrote no new jailbreaks or harmful requests.

   Collapse this answer

2. 02What related research exists?

   Related research falls into four groups: training randomness, variance in safety behavior, refusal mechanisms, and noise correction and fixed-seed retraining.

   Training randomness

   • Zhong et al. (2021), D'Amour et al. (2022), McCoy et al. (2020), Juneja et al. (2023)—models differing only in seed have similar overall accuracy, but differ in per-sample predictions and out-of-distribution generalization.
   • MultiBERTs (Sellam et al., 2022) nests task fine-tuning seeds under pretraining seeds; PolyPythias (van der Wal et al., 2025) varies pretraining seeds on the same corpus and evaluates base models. The paper notes that none of these suites include models trained to refuse.

   Variance in safety behavior

   • Fraser et al. (2025)—seeds of benign fine-tuning shift the model's overall harmfulness; Larsen (2026)—sampling seed and temperature flip safety decisions on about a quarter of prompts.
   • Hughes et al. (2025)—jailbreak difficulty rankings are broadly similar across multiple attack runs on the same model and across different models (except for one defended model).
   • Lu et al. (2026)—branches of the same run differing only in the final data window have similar refusal after SFT but different refusal after identical post-training; Itzhak et al. (2025)—cognitive biases likewise track the base model more than the instruction data. The paper notes that these studies all vary seeds after pretraining or take only the final segment of the same run, and never change the entire pretraining run.

   Refusal mechanisms (within and across models)

   • Arditi et al. (2024)—refusal is mediated by a single direction in activation space; Wollschläger et al. (2025) propose a low-dimensional cone, Zhao et al. (2025b) propose that refusal and harmfulness are encoded separately.
   • Labunets (2026), Reblitz-Richardson (2026)—trace this structure within the same pretraining run; Bosco & Srinivasan (2026), Oozeer et al. (2025)—transfer refusal interventions across architectures (rotation) and across scales (mapping); McCann (2026) compares features and steering vectors across nine Pythia-70m seeds (excluding refusal). The paper notes that refusal has never been compared across different runs of the same data recipe, and this paper makes this comparison in Appendix F.

   Noise correction and fixed-seed retraining

   • Spearman's (1904) attenuation correction (dividing by the square root of the product of reliabilities) has been used to compare models with human ratings or brain recordings (Depaolini & Hasson, 2026; Tan & Deng, 2026); Geirhos et al.'s (2020) error consistency corrects only for chance agreement.
   • Summers & Dinneen (2021), Zhuang et al. (2022), Bhojanapalli et al. (2021), Altıntaş et al. (2025)—even with all seeds fixed, nondeterministic kernels or tiny early perturbations cause runs to diverge. The paper notes that neither line of work applies noise correction to comparisons where "two models are each measured by sampled responses"; this paper does.

   Positioning: The authors state that this paper is the first study to compare jailbreak results across different complete pretraining runs under the same data recipe and the same safety training ("To our knowledge, no study has tested…"). This paper has no attack-method baseline; the comparison objects are the 25 recipes × 3 seed pretraining runs released by DataDecide (Magnusson et al., 2025), with prompts from six public benchmarks: HarmBench, SORRY-Bench, StrongREJECT, XSTest, OR-Bench-Hard-1K, and WildJailbreak.

   Collapse this answer

3. 03How does the paper address this problem?

   Use the difference in per-prompt correlation between "same base vs. different base under the same recipe" (ancestry gap) to isolate the effect of the pretraining run.

   Overall approach: turn "which prompts break the model" into a per-prompt vulnerability profile for each model, then use the correlation differences across three types of model pairs with different degrees of shared ancestry to separate the contributions of the pretraining run and the data recipe.

   Models and training

   • Base models: 1B models from DataDecide's 25 data recipes × 3 pretraining seeds, each recipe trained on the same 100B tokens, with the seed determining both initialization and data order; the main data recipe is DCLM-Baseline. The authors disabled dropout so that the seed is the only controlled randomness.
   • Self-trained reruns: on the authors' own H200s, retrained twice with base model 1's seed (same-seed reruns, differing from the original only in GPU nondeterminism/hardware-software); also trained two partial reruns sharing only base model 1's initialization or only its data order, respectively.
   • Post-training: all base models uniformly underwent Tülu 3's SFT (including WildGuardMix, WildJailbreak, CoCoNot safety data) + DPO, stopping before OLMo 2's two RL stages; each base model used 2 post-training seeds (4 for the main recipe), with the post-training seed determining only the order of training samples. 150 models in total.

   Pair types and prompt set

   • Three pair types (Figure 1): MZ twins share a base model and differ only in post-training seed; same-recipe pairs share only the data recipe; cross-recipe pairs share neither. rMZ − rS is called the ancestry gap, measuring the increment from sharing one pretraining run; rS − rX measures the increment from the data recipe.
   • Prompt set: from 7 public sources, after Tülu 3's 8-gram decontamination and length filtering, 6,226 prompts remained; the main analysis subset is 3,212 adversarial-harmful prompts (harmful requests wrapped in jailbreak templates), of which HarmBench's 1,212 human-jailbreak prompts consist of 300 harmful requests × 91 attack templates.

   Sampling, judging and statistics

   • Sampling: vLLM, temperature 1.0, top-p 1, 20 responses per prompt, at most 512 new tokens; rule-based checks removed incoherent samples that repeated or paraphrased the prompt.
   • Judging: the primary judge is WildGuard's refusal label, with Qwen3Guard-Gen-8B as second judge; a model's vulnerability on a prompt = non-refusal rate among coherent samples, and vulnerability across all prompts constitutes its profile.
   • Statistics: Pearson r between two models' profiles, then Spearman correction divided by the square root of the product of reliabilities to obtain the corrected correlation (reliability from odd-even split-half correlation extrapolated by Spearman–Brown); report means and 95% bootstrap intervals for each pair type, with predictions and interpretation rules preregistered before the experiments (Appendix H).

   Mechanism analysis (Appendix F)

   • Following Arditi et al. (2024), estimate each model's refusal direction (mean activation difference between harmful and harmless instructions), perform ablation, cross-run rotated transfer (orthogonal Procrustes), projection and within-pair link to vulnerability, and a "transplant" intervention setting a model's projection on each prompt to the value of another run's model, measuring the proportion of behavioral difference explained by this scalar (closure).

   Collapse this answer

4. 04What experiments did the paper conduct?

   25-recipe grid + three scales + same-seed/partial reruns + single-checkpoint inference + human audit and mechanistic analysis.

   Experimental setup

   • Models tested: 150 1B models (25 data recipes × 3 pretraining seeds × 2 post-training seeds; the main recipe had 4 post-training seeds); the main recipe additionally had 150M and 300M with 3 base models × 4 post-training seeds each; 2 same-seed reruns and 2 partial reruns each with 4 post-training seeds.
   • Prompt set: 6,226 prompts (3,212 adversarial-harmful, 1,240 plain harmful, 1,774 benign), from HarmBench, SORRY-Bench, StrongREJECT, XSTest, OR-Bench-Hard-1K, WildJailbreak.
   • Metrics: per-prompt non-refusal rate (20 coherent samples), corrected Pearson correlation, ancestry gap (rMZ − rS), mean vulnerability.
   • Judging: WildGuard refusal labels as primary, Qwen3Guard-Gen-8B for review; manual audit by two authors blinded (2,800 responses across three annotation rounds), Cohen's κ 0.65–0.768 on refusal.
   • Preregistration: primary predictions and interpretation rules were registered before the corresponding experiments (Appendix H); intervals are mostly 95% bootstrap (by prompt or by recipe).

   Main results

   According to Table 1 (mean corrected correlation, 95% intervals in parentheses):

   Swipe horizontally to read the table

   Model scope	MZ twins	Same-recipe	Cross-recipe	Ancestry gap
   All 25 recipes, 1B	0.984 [0.982, 0.985]	0.811 [0.804, 0.817]	0.779 [0.766, 0.788]	0.173 [0.167, 0.179]
   Main recipe, 1B	0.985 [0.983, 0.987]	0.814 [0.801, 0.825]	Not reported	0.171 [0.160, 0.183]
   Main recipe, 300M	0.981 [0.978, 0.985]	0.770 [0.756, 0.784]	Not reported	0.211 [0.198, 0.224]
   Main recipe, 150M	0.988 [0.983, 0.993]	0.784 [0.769, 0.799]	Not reported	0.204 [0.189, 0.218]

   • The authors state that the correlation drop caused by not sharing a base model is 5.4 times that of not sharing a data recipe (Table 1, interval [4.0, 7.2]); by recipe, the ancestry gap ranges from 0.145 to 0.209, and the intervals for all 25 recipes are above zero (Figure 2).
   • The authors state that "which prompts break the model" mainly depends on the pretraining run, while "how often it is broken" mainly depends on the data recipe: of the variance in mean vulnerability, data recipe accounts for 49% [12%, 70%], base model 32% [16%, 59%], and post-training seed 19% [11%, 37%] (§3.1).
   • After swapping in a different safety dataset (Nemotron-SFT-Safety-v2, PKU-SafeRLHF, BeaverTails, HH-RLHF, etc. replacing Tülu 3's SFT safety data), the gap for the main recipe 1B was +0.178, versus 0.175 for Tülu 3 (Appendix G.6).

   Same-seed and partial reruns (§3.2)

   • Same-seed reruns: the three pairwise means among the two same-seed reruns, base model 1, and each other were 0.844 [0.835, 0.853], only 0.030 [0.021, 0.040] higher than same-recipe pairs, far below the 0.985 of MZ twins (Figure 3); the authors conclude from this that the high correlation of MZ twins comes from sharing the same training run, not from the seed itself.
   • Partial reruns: pairs sharing only initialization or only data order were nearly identical to pairs sharing neither (0.822 / 0.822 / 0.823, all three difference intervals covering zero); the authors state that no contribution of initialization or data order alone was detected.
   • Hardware/software switch: the correlation between the two reruns was 0.011 [−0.001, 0.023] higher than that between the first rerun and base model 1, with the interval covering zero; the authors state that no correlation loss from switching hardware/software was detected.

   Inferences from a single checkpoint (§3.3)

   • Attack ranking transfer: the best attack template remained best in 67% [55%, 79%] of pairs under a new post-training seed, but in only 27% [19%, 35%] under a new pretraining run; a 10–20 percentage point template lead was overturned 2.2% of the time by an MZ twin and 25.3% by a same-recipe partner (Figure 4a).
   • Run fingerprinting: using profiles from 3,211 prompts for nearest-correlation matching, all 75 pretraining runs were identified correctly (random guessing would be 1/75); with a random 50 prompts, accuracy averaged 98% (Figure 4b). Identifying the data recipe (unseen base model) succeeded only 24/50 (48%).
   • Single prompt and recipe ranking: when a prompt broke a model at least 9 out of 10 times, its MZ twin also met this 75.0% of the time, same-recipe partners 60.9%, and cross-recipe 58.7%; when comparing the mean vulnerability of two data recipes using one model each, 23.4% of comparisons ranked opposite to the six-model mean, with a single-model standard error of 3.44 points, dropping to 1.99 points with three base models.

   Judge audit and mechanism analysis

   • Judge audit: on the two adversarial sources, WildGuard labeled many responses that humans judged as refusals as compliant—precision 0.528 [0.436, 0.615], recall 0.977 in the 25-recipe grid audit (Table D.3); uncorrected mean vulnerability was 0.528, and after precision correction it was 0.278 [0.230, 0.324]. On human labels for four models, the difference between MZ and same-recipe correlations was 0.156 [0.036, 0.294], with the interval above zero (Table D.6); the authors attribute the difference to responses rather than the judge.
   • Refusal direction (Appendix F): ablating each model's own refusal direction at least doubled plain-harmful vulnerability in all 12 models (Table F.1); after Procrustes rotation, another pretraining run's direction met the registered criterion in 11 of 12 targets (Table F.2). Setting a model's projection on each prompt to the value of another run's model, the mean closure across 12 directed pairs was 0.492 [0.469, 0.515] (Table F.5); the authors state that about half of the behavioral distance moves with this scalar.

   Other ablations and analyses

   • Template/request effects: after removing the main effects of template strength and request difficulty in HarmBench human-jailbreak, the gap rose from 0.210 to 0.525 (Appendix G).
   • Swapping Qwen3Guard in as judge: the gap for the main recipe 1B was 0.147 (Table G.1), with the direction of the conclusion unchanged.
   • On benign prompts, same-seed pairs (0.793) were lower than the same-recipe mean (0.851) (Table B.2); the authors offer no interpretation.
   • The data recipe DCLM-Baseline (QC FW 3%) had a notably low cross-recipe mean of 0.647, and its MZ twins mean of 0.973 was the lowest of the 25 recipes (Appendix G.2).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list two limitations, model family and judge; the experiments cover 25 recipes, three scales and multiple manual audits.

   Limitations and future directions noted by the authors

   • Single model family (Section 5): all models come from the DataDecide pretraining family and Tülu 3's SFT and DPO stages; the authors state that the gap intervals exclude zero across all 25 recipes and that the gap reproduces after swapping in a different safety dataset, indicating that the gap does not depend on a single corpus or that one safety dataset.
   • Judge misses refusals (Section 5): the primary judge WildGuard misses refusals on jailbreak prompts, with uncorrected mean vulnerability 0.528 and 0.278 [0.230, 0.324] after audit correction; human labels still give a difference above zero, 0.156 [0.036, 0.294].
   • Scale ceiling (Section 5): the largest model is 1B, which the authors explain is limited by the cost of multiple full pretrainings (citing Madaan et al. 2025: ten 7B runs at about 15,000 A100 hours each), and state that the released code and per-prompt labels facilitate future measurement of the ancestry gap on larger models.
   • Scope of mechanism analysis (end of Appendix F.8): the transplant intervention covers only one data recipe, 1B, a single layer and single position, one scalar per prompt, and uses only the edited model's own direction.

   Scope of experiments

   • 150 1B models tested (25 recipes × 3 pretraining seeds × 2 post-training seeds) + 12 each for the main recipe at 150M/300M + 4 derived models from self-trained reruns; all went through the same Tülu 3 SFT+DPO pipeline.
   • Prompt set of 6,226 prompts, with 3,212 adversarial-harmful used in the main analysis; 20 responses sampled per model per prompt.
   • Judges were WildGuard (primary) and Qwen3Guard-Gen-8B (review); manual annotation covered 2,800 responses across three rounds, annotated by two of the authors, not independent external annotators.
   • Only 2 same-seed reruns and 2 partial reruns, all based on base model 1/2 of the main recipe; the cross-run fingerprinting experiment matched only across post-training seeds under the same post-training recipe, and the authors explicitly did not test across different fine-tuning schemes.
   • The paper reports no main experiments on closed-source models, other post-training recipes (aside from one safety data swap), or larger models; Appendix H records the distinction between preregistered predictions and post-hoc analyses.

   Collapse this answer

6. 06Summarize the main content of the paper

   Which jailbreaks break a model depends mainly on the pretraining run rather than the data; jailbreak conclusions from a single checkpoint can only represent one run.

   A measurement study: the authors ask whether, when the same data recipe is used to rerun pretraining once and given the same safety post-training, the model will be broken by the same jailbreak prompts.

   • Problem addressed: jailbreak evaluations treat "which jailbreaks work" as a property of the data and training, ignoring pretraining randomness; no prior study had compared two full pretrainings on the same data.
   • Method: 1B base models from DataDecide's 25 data recipes × 3 pretraining seeds, uniformly given Tülu 3 SFT+DPO (2 post-training seeds), 150 models in total; each model sampled 20 times on 3,212 adversarial harmful prompts, with WildGuard judging refusal, comparing corrected correlations of per-prompt non-refusal rates across three pair types (MZ twins / same-recipe / cross-recipe).
   • Main results: MZ twins correlation 0.984, dropping to 0.811 when the pretraining seed changes, and only to 0.779 when the data recipe also changes (Table 1); this ordering holds across all 25 recipes and at 150M/300M; same-seed reruns are only 0.844, and the authors state that the high correlation belongs to that one run rather than the seed.
   • Implications for evaluators: the best attack template remained best only 27% of the time after changing the pretraining seed (67% after changing the post-training seed); per-prompt profiles identified the run in 75/75 of 75 runs, and 98% with 50 prompts; when comparing two data recipes using one model each, 23.4% of rankings reverse.
   • Authors' conclusion: a jailbreak found on a single checkpoint is evidence about one pretraining run, and only partly about the data behind it; the authors argue that pretraining data comparisons and jailbreak evaluations should both be read across multiple runs, and that a new pretraining run may need its own red teaming.
   • Boundaries: conclusions are limited to the DataDecide family and Tülu 3 post-training, with a maximum of 1B; the primary judge misses refusals on jailbreak prompts, and the direction of the conclusions is unchanged after the authors' manual annotation review.

   Collapse this answer

Related papers