Safety Pretraining: Toward the Next Generation of Safe AI

1. 01What problem does this paper attempt to solve?

   Use data filtering, rewriting, refusal education, and a harmfulness tag to build safety during the pretraining stage.

   How to build safety during pretraining, rather than relying only on post-hoc alignment.

   • Scenario: The authors argue models have entered healthcare, education, and public policy, and harmful or toxic generation remains a core risk.
   • Limitations of existing work: Post-hoc methods such as RLHF, DPO, and Constitutional AI are viewed by the authors as at most superficially suppressing toxic generation under adversarial analysis. The authors say unsafe information, once internalized during pretraining, is difficult or even impossible to forget; their summary is alignment is not unlearning.
   • Hypothesis: In the discussion the authors propose the Outer Onion Hypothesis: surface-level risks can be peeled away with post-training, while risks absorbed early in pretraining are harder to remove.
   • What this paper does: safety filtering, synthetic recontextualization, RefuseWeb and Moral Education, a harmfulness tag, and Safe Beam Search at inference; also evaluates base models not yet safety fine-tuned, and releases the 1.7B SafeLM.
   • Threat settings in evaluation:
     • Target: reduce compliance with harmful requests, measured by ASR.
     • Adversarial setting: §6.2 uses adversarially learned suffixes, learned on the authors' own models, targeting 100 items from JailbreakBench. The paper does not state white-box or black-box.
     • Another stress: benign supervised fine-tuning on GSM8K, to see whether alignment built only at the instruction-tuning stage is broken.
     • Subject: a 1.7B model trained with SmolLM2 settings.

   Collapse this answer

2. 02What related research exists?

   Nearest neighbors are post-hoc alignment, heuristic filtering, and special token fine-tuning; the authors move the intervention into pretraining.

   Section 2 divides related work into post-hoc alignment, pretraining interventions, and safety evaluation.

   Post-hoc alignment

   • RLHF and supervised fine-tuning (Ouyang et al., 2022; Bai et al., 2022a): Adjust the model's tendency to refuse harmful queries. The paper states these are resource-intensive, require large amounts of preference annotations, and harmful content can still often be extracted under adversarial pressure (Zou et al., 2023; Chao et al., 2023).
   • Shi et al. (2023): Use DPO during the instruction fine-tuning stage to improve safety response rates while maintaining helpfulness. The paper states such work still centers on post-hoc tuning.
   • The introduction lists DPO (Rafailov et al., 2023) and Constitutional AI (Bai et al., 2022b) as post-hoc alignment; Section 2 does not further expand the comparison.

   Pretraining and data intervention

   • Korbak et al. (2023): Conditional pretraining on human preference scores, with lower toxicity at the 100M parameter scale than fine-tuned models. The paper also places this work on the same track as special token fine-tuning.
   • FineWeb (Penedo et al., 2024): Heuristic filtering to reduce toxicity. The paper paraphrases Vidgen et al. (2024): such data still has residual harmful content and requires stronger filtering.
   • Bianchi et al. (2023): Adding safety samples during fine-tuning can improve safety without harming performance. The paper states this prompted them to extend the practice to pretraining.
   • Special token fine-tuning (Xhonneux et al., 2025; Jain et al., 2024; Korbak et al., 2023): Annotate harmful data with special tokens and fine-tune, so that generation can steer away. The authors state their tag extends this idea to pretraining, and that associations established only through fine-tuning fail under benign fine-tuning.
   • Qi et al. (2024b): The paper paraphrases their view: safety alignment is fragile, at most modifying the first few generated tokens, and thus vulnerable under jailbreaks or benign fine-tuning (He et al., 2024).

   Safety evaluation

   • SafetyPrompts, MLCommons AI Safety Benchmark v0.5 (Vidgen et al., 2024), and AdvBench, JailbreakBench, HarmBench: used to measure toxicity or jailbreak tendencies. The paper states it adds datasets, evaluation tools, and Data Safety Report Cards on top of these.

   Baselines and benchmarks

   • Baselines: Standard pretraining on raw web data following SmolLM2, then instruction and safety fine-tuning on Ultrachat-200k, WildGuardMix, WildJailbreak; ablations also compare keeping only score-0.
   • Benchmarks: General tasks are the nine items in Table 1; safety is HarmBench, TDC, JailbreakBench, AdvBench, and the authors' base completion evaluation; over-refusal uses Alpaca.

   The authors position this paper before post-training: using data rewriting, refusal education, and tags during pretraining, rather than establishing safety associations only during fine-tuning.

   Collapse this answer

3. 03How does the paper address this problem?

   Filter and rewrite data, add refusal and moral education, then steer with tags and Safe Beam Search.

   The framework consists of four data interventions and one decoding shift. The authors say they release the 1.7B SafeLM.

   Safety scoring and filtering

   • Score: web text is scored 0–5, 0 for all-ages appropriate, 5 for highly unsafe, with a brief reason. The full scoring prompt is in Appendix C.1.
   • Annotation: annotated on the same subset of FineWeb as the FineWeb-Edu classifier. The introduction writes the classifier is trained on 10K samples annotated by GPT-4; §3.1 writes scores are given by GPT-4o-mini. A portion is held out as reference labels.
   • Scaled scoring: §3.1 overview writes GPT-4 and LLaMA-3.1-8B; later text writes Qwen 2.5-7B scores with the same prompt, and an embedding classifier is fine-tuned from gte-base-en-v1.5. The final score is the maximum of the two, to maximize recall of unsafe samples. The overview also writes thresholds are iteratively tuned on 600K scored samples.
   • Report card: the authors propose releasing a Data Safety Report Card with the dataset (Figure 1): safety score distribution, and 14 harmful n-gram categories merged from Vidgen et al. (2024) and Chao et al. (2024), counted per million tokens with Infini-gram. The query table is in Appendix D.

   Rewriting, refusal, and moral education

   • SafeWeb: instead of directly discarding unsafe samples, rewrite them into text with disclaimers and ethical explanations, requiring that a single sentence be safe out of context, while preserving facts. §3.2 writes the generation model is LLaMA-3.1-8B. The introduction writes 12 prompts and 100 billion tokens; §3.2 writes over 100B tokens; related work writes 300B tokens of synthetic data. Appendix C.2 gives style templates and the full prompt.
   • RefuseWeb: text in FineWeb scored 4 or 5 is turned into user–assistant dialogues. The assistant refuses harmful requests and explains the reason, with categories including harassment, discrimination, malware, physical harm, economic harm, fraud, misinformation, adult content, and privacy. At tokenization, role names are replaced with names or occupations. The full prompt is in Appendix C.3.
   • Moral Education: LLaMA 3.1-8B-Instruct turns the above dialogues into web-style ethical essays, with scores still taken as the maximum of the language model scale and the embedding classifier. The full prompt is in Appendix C.4.

   Harmfulness tag and Safe Beam Search

   • Annotation: for segments judged unsafe by the safety score, insert <potentially unsafe content> at a random position, defaulting to 5% of input length (probability p in Algorithm 2). Safe text does not get the tag.
   • Safe Beam Search (Algorithm 1): at each step, do a one-step lookahead on candidates, compute the probability that the next token is the tag, drop the 50% with highest probability, then take top-k by log-likelihood. The paper writes beam size as k, without giving a value for k.

   Training setup

   • Pretraining: follows SmolLM2, training 1.7B. Initial corpus is FineWeb-Edu, StackOverflow, FineMath, Cosmopedia. Uses LitGPT, FlashAttention-2, and mixed precision. Optimization hyperparameters are the same as SmolLM2; the paper does not list learning rate, batch, or sequence length.
   • Post-training: all models are instruction and safety fine-tuned on Ultrachat-200k, WildGuardMix, WildJailbreak. Refusal data is about 10% of instruction data. For models pretrained with the tag, 10% of WildGuardMix completions are additionally made into tagged samples.
   • Success determination: ASR on harmful requests is given by the HarmBench classifier and a rewritten judging template, answering yes or no (Appendix C.5); the paper does not state the underlying model of that classifier. Base models turn harmful requests into completion-style prompts rather than dialogue completions.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The introduction writes ASR drops from 38.8% to 8.3%, the abstract writes 8.4%; keeping only score-0 raises it to 43.8%.

   Experimental setup

   • Generation model: 1.7B trained per SmolLM2, variants are standard pretraining, safety pretraining, safety pretraining plus SafeBeam. The harmful completion examples in appendix Table 2 target SmolLM-1.7B-Instruct and are not quoted here.
   • Corpus and post-training: FineWeb-Edu, StackOverflow, FineMath, Cosmopedia; post-training is Ultrachat-200k, WildGuardMix, WildJailbreak.
   • Safety evaluation: HarmBench, TDC, JailbreakBench, AdvBench; base evaluation turns harmful requests into completions. Adversarial suffixes are learned on the authors' models, 100 items from JailbreakBench in total.
   • Other evaluations: nine general benchmarks in Table 1. Overrefusal draws 300 items from Alpaca. Benign fine-tuning uses GSM8K, then safety sets are retested. The default ablation metric is ASR after GSM8k benign fine-tuning.
   • Judging: ASR uses the HarmBench classifier and the template in Appendix C.5; the underlying model is not stated. Overrefusal is classified into 5 categories by GPT-4o-mini. The paper does not report number of repetitions, nor agreement between judging and humans.

   Main results

   Statement in the paper	ASR	Source
   Standard pretraining, base	44% (figure label 44.1%)	§6.2, Figure 2
   Safety pretraining, base	11% (figure label 11.6%)	§6.2, Figure 2
   Reduction endpoint in abstract	8.4% (compared to 38.8%)	Abstract
   Reduction endpoint in introduction	8.3% (compared to 38.8%)	§1

   • Authors' comparison: the authors say the standard pretrained base's 44% is four times the safety-pretrained 11% (original text: four times higher). Both the abstract and introduction write a drop from 38.8%, with endpoints 8.4% and 8.3% respectively. Neither states whether the number belongs to base, after instruction tuning, or after benign fine-tuning.
   • Instruction tuning: the authors say after safety instruction tuning all models have ASR near zero, and call this a false sense of safety. Figure 2 labels 1.6%, 0.7%, 0.0% at that stage; the text does not map these three numbers one-to-one to the three variants. The figure also has labels such as 28.8% and 23.0%, which the text likewise does not map one-to-one; no further attribution is made here.
   • General tasks: the authors say safety interventions do not hurt general capability. Table 1 nine-task average: Raw Data 42.8%, Score 0 Only 42.7%, + Rephrasing 43.5%, + Refusal + Moral Ed. 42.9%. In the same table, GSM8K goes from 7.0% with Raw to 4.7% with the full intervention, TriviaQA from 35.2% to 33.3%; with only Rephrasing added, GSM8K is 7.7%.

   Data intervention ablation

   The metric is ASR after GSM8k benign fine-tuning (Figure 4, §7.1).

   Data	ASR	Source
   Raw Data	38.8%	Figure 4, §7.1
   Score 0 Only	43.8%	Figure 4, §7.1
   + Rephrasing	33.6%	Figure 4, §7.1
   + Native Refusal	25.1%	Figure 4, §7.1
   + Moral Ed.	23.0%	Figure 4

   • Keeping only the safe subset: rises from 38.8% to 43.8%. The authors argue the model has not seen unsafe patterns and therefore cannot handle them.
   • Rewriting and refusal: the authors say rewriting brings ASR from 38.8% to 33.6%; adding refusal data brings it to 25.1%, which they call 8.5 percentage points.
   • Two numbers for moral education: the authors say moral education gives the lowest ASR in the study and write it drops to 20.0%. Figure 4 labels + Moral Ed. as 23.0%. Relative to the 25.1% in the figure, this step is smaller than the refusal drop the authors state.

   Benign fine-tuning and overrefusal

   • Benign fine-tuning: §6.4 does supervised fine-tuning on GSM8K and then retests safety sets. The authors say all models have low ASR after safety instruction tuning, but the standard pretrained model then degrades, and they say its ASR nearly quadruples, with alignment largely superficial; the safety-pretrained model rises only slightly. That paragraph does not give the exact before/after values for this quadrupling.
   • Adversarial suffixes: §6.2 describes learning suffixes on 100 items and evaluating ASR; the text does not give numbers for that attack. The authors state one limitation: these suffixes were learned without knowing SafeBeam's modified decoding, and future work could do attacks aware of the inference algorithm.
   • Alpaca: 300 items, GPT-4o-mini classifies into compliant and useful, compliant but not useful, partially refused, fully refused; Appendix C.5 also has shortcircuit, and Figure 3's legend has the first 4 categories. The paper does not give counts per category. The authors say safety pretraining does not reduce compliance on benign requests, and SafeBeam slightly increases overrefusal.

   Other ablations and analyses

   • Tag in pretraining vs. only in instruction tuning (§7.2): after benign fine-tuning, ASR with the tag in pretraining is about 8%, with the tag only in instruction tuning about 30%; the authors say the latter is about 4 times the former.
   • Tag proportion (§7.3): among 3%, 5%, 10%, the authors say 5% gives the lowest ASR; no ASR values are given for the three proportions.
   • Classifiers (Table 4, GPT-4o-mini held-out set): Ensemble Recall@1 is 0.9536, F1 is 0.3615; gte-large-en-v1.5 F1 is 0.4533; LLaMA Guard Recall@1 is 0.1037; profanity checker Recall@1 is 0.0310. The authors say existing filters do not align with GPT-4o-mini scores, and prioritize Recall@1.
   • Scoring failures (Appendix B.2): LLaMA's JSON error rate is about 25%, and it tends to score 4 or 5 relative to GPT-4o-mini and human annotation.
   • N-grams after rewriting (Figure 6): the authors say rewriting reduces the frequency of various harmful queries, and because sensitive topics still need to be explained, the frequency should not be zero. The text does not give per-category frequencies.

   Collapse this answer

5. 05What are some points that can be further explored?

   The limitation the authors state is that the SafeBeam evaluation did not use attacks from the perceptual decoding algorithm.

   Limitations and future directions noted by the authors

   The paper does not have a separate Limitations section. What is explicitly written as limitations or future work includes:

   • SafeBeam evaluation (§6.2): The authors write that adversarial jailbreaks were learned without awareness of the modified inference algorithm; future work could develop and evaluate attacks that are aware of the inference algorithm.
   • Classifier (Appendix B.2): The authors state that future work could explore fine-tuning to reduce excessive risk aversion while maintaining filtering performance.

   Experimental coverage

   • The generation model experiments are on a 1.7B model trained on SmolLM2, with corpora FineWeb-Edu, StackOverflow, FineMath, Cosmopedia (§5). Appendix Table 2 also gives examples of harmful completions from SmolLM-1.7B-Instruct.
   • The safety set is HarmBench, TDC, JailbreakBench, AdvBench, and a self-built base completion evaluation; the general set is the nine items in Table 1; Alpaca over-refusal is 300 examples per model, reviewed by GPT-4o-mini (§6).
   • Benign fine-tuning uses GSM8K. Adversarial suffixes are learned on 100 JailbreakBench examples on the authors' own models (§6.2, §6.4).
   • The paper does not report the ASR of this adversarial suffix attack, the number of tokens for 1.7B training, the beam size k of Safe Beam Search, the number of repetitions, or the agreement between GPT-4o-mini or the HarmBench classifier and human judgment.
   • The safety classifier is compared on a held-out set of FineWeb-Edu labeled by GPT-4o-mini; the final score used for data intervention takes the maximum of Qwen 2.5-7B and gte-base-en-v1.5 (§3.1, Table 4).

   Collapse this answer

6. 06Summarize the main content of the paper

   Using filtering, rewriting, refusal, moral education, and tags for safety pretraining, the authors say ASR can be reduced.

   SafeLM puts safety into the 1.7B pretraining data and representation separation, rather than only doing post-hoc alignment.

   • Problem: The authors argue that post-hoc alignment methods such as RLHF, DPO, and Constitutional AI struggle to erase unsafe information already learned during pretraining; alignment is not unlearning.
   • Method: A classifier scores web text 0–5 and filters; unsafe text is rewritten into contextualized SafeWeb; RefuseWeb and Moral Education teach refusal and ethical reasoning; unsafe segments get a harmfulness tag; Safe Beam Search at inference drops beams more likely to generate that tag. Training setup follows SmolLM2.
   • Safety numbers: The introduction writes ASR drops from 38.8% to 8.3%, the abstract writes to 8.4%. §6.2 writes the standard pretrained base is 44% and safety-pretrained is 11% (Figure 2 labels 44.1% and 11.6%), and the authors say the former is four times the latter. After instruction tuning the authors say ASR is near zero, but benign fine-tuning on GSM8K breaks alignment that relies only on post-training.
   • Data ablation: With only score-0 kept, ASR after GSM8k benign fine-tuning rises from 38.8% to 43.8% (Figure 4). Rewriting and refusal go further down; the endpoint with moral education is written as 20.0% in the text, while Figure 4 labels it 23.0%.
   • General scores: Table 1 nine-task average: Raw Data 42.8%, Refusal+Moral Ed. 42.9%. GSM8K goes from 7.0% to 4.7%.
   • Authors' conclusion: Safety should be built during pretraining through contextual exposure, refusal, and moral education. The authors argue that simply removing unsafe samples does not yield a more robust model, and post-training cannot replace unlearning.

   Collapse this answer

Related papers