Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

Safety added in post-training may be lost when subsequent data contains no safety data, and increasing refusal often comes at the cost of general capability.

• Where it happens: The authors consider LLM safety already a core part of development, but most alignment is still after pretraining, adding refusal via SFT, preference learning or related methods. Recent work has also put safety data into pretraining, or used alignment-oriented documents in mid-training.
• Existing gap: The authors state that if subsequent SFT has almost no safety data, even if its purpose is general capability, the model may quickly lose safety. Safety fine-tuning also often focuses on refusal, which may interfere with the general reasoning and instruction following already present from pretraining.
• Shared structure: The authors argue that safety responses and general tasks both require understanding intent or goals, invoking relevant knowledge and constraints, and reasoning about possible consequences, so the two need not be treated as separate training objectives.
• This paper's approach: Propose safety mid-training, placed between pretraining and post-training. Given safety and utility prompts, the base model self-generates structured documents and reasoning traces, and after filtering, training continues, so that subsequent SFT and RL start from a model that has already applied these capabilities to safety.

Safety post-training

• SFT and preference optimization: Bianchi et al. (2024) do safety SFT; Bai et al. (2022)'s RLHF, Rafailov et al. (2023)'s DPO, and Shao et al. (2024)'s GRPO perform alignment after pretraining.
• Safety reasoning supervision: Guan et al. (2024), Zhang et al. (2025a), Wang et al. (2025b), Zhao et al. (2025) train safety reasoning with chain of thought, safety rationales or process-level rewards.
• Safety data: WildJailbreak (Jiang et al., 2024) constructs adversarial safety data; XGuard-Train (Rahman et al., 2025) extends to multi-turn jailbreaks. Other work adds safety rationales, answer-then-check, semantically related safety responses or long CoT. The authors state this work shows that the structure and coverage of data matter for post-training.

Mid-training

• Stage definition: OLMo et al. (2024), Lambert et al. (2024), Tu et al. (2025), Wang et al. (2025a), Mo et al. (2025) place mid-training after general pretraining and before instruction tuning or preference optimization, still using the language modeling objective, with data volume usually larger than post-training.
• Capability to use: Yang et al. (2025), Grattafiori et al. (2024), Liu et al. (2024) use it to improve reasoning, multilingual ability, coding, tool use or domain adaptation.
• Alignment-oriented documents: Korbak et al. (2026) use synthetic documents about alignment behavior; Li et al. (2026) derive documents from model specifications. The authors state that their data comes directly from safety and utility prompts, and the two types of samples share the same reasoning pattern.

Self-generation

• Using the trained model to produce data: Agarwal et al. (2024), Zhao et al. (2026), Shenfeld et al. (2026) and others sample training data from the model itself. The authors relay that prior work studies how it mitigates distribution mismatch and catastrophic forgetting and improves generalization. This paper uses it to construct documents and reasoning traces for the two types of prompts.

Baselines and benchmarks

• Baselines: base without this method, Base+SFT, Base+SFT (Distill Sonnet-4.6), Safety Pretraining (Maini et al., 2025, about 2.7B tokens), Alignment Pretraining (Tice et al., 2026, about 1.3B tokens).
• Benchmarks: on the safety side HarmBench, AdvBench, WildJailbreak, DoNotAnswer, SimpleSafetyTests, JailbreakHub, StrongREJECT, BeaverTails, AIR-Bench; on the utility side GSM8K, MATH-500, TruthfulQA, AlpacaEval, MMLU, ARC-Challenge, MMLU-Pro, HellaSwag, BBH.

The authors position this paper among these lines: not only adding refusal in post-training, nor only using synthetic documents about alignment behavior, but learning safety and general reasoning together in mid-training with the same reasoning structure.

safety mid-training继续训练base model：安全与utility提示词都变成结构化文档和推理轨迹，过滤后放进同一因果语言建模语料。作者给出的理由是，用base model自己生成，就不需要更大的生成模型。

提示词与语料

• 字段：每条含用户提示词、safe/unsafe标签、可选越狱策略标注和参考答案。参考答案只用于生成时的事实锚定，以及检查生成回答。
• 规模：提示词池约185k条唯一提示。默认的推理加文档语料约367k–368k条记录、约200M token。附录A.1：77%为safe、23%为unsafe；unsafe中75%是直接有害请求，25%套了越狱模板。
• 隔离：评测样本不进入mid-training、SFT和GRPO。来源包括Alpaca、BeaverTails、WildJailbreak、Dolly-15k、Competition Math、PKU-SafeRLHF、UltraChat、GSM8K、Alert、SaladBench、JailbreakPair，计数为约数（Table 8）。

文档与推理轨迹

• 文档：同一base model先生成固定七段文档：Task、Core goal、Relevant knowledge、Key constraints、Possible consequences、Best feasible path、Verification。安全查询与通用查询共用该模式。
• 轨迹：再以文档和参考答案为条件，采样推理轨迹和答案。轨迹把意图、相关知识与后果写成连续说明；安全提示词与不安全提示词都走这一结构。
• 变体：reasoning-only只把提示词、轨迹和生成答案放进训练序列。附录A.2与A.3给出完整生成说明；采样温度为0.7，自生成侧top-p为0.95，最大新token为2048。

裁判过滤

• 裁判：Qwen3-32B，温度0。检查对文档的忠实性、逻辑与伦理连贯性，以及与参考答案和安全标签的一致性，并打1到5的质量分。附录A.4给出rubric。
• 接受：overall_pass为真且质量至少4；同一提示词多条通过时留质量最高者。无人通过则按裁判反馈重采样，最多三轮，仍失败就从推理语料删除。文档用另一套rubric，通过与质量标准相同。

训练目标

• 两种记录：放在同一JSONL并打乱。推理记录套chat template；文档记录是无模板纯文本。二者打包为长度3072的块。
• 损失：全部token计入因果语言建模，包括chat记录里的提示词。损失为 L_mid(θ)=−∑_j=1^B∑_t=1^T_jlog p_θ(z_j,t|z_j,<t) ，即对每个打包块中每个token的条件对数概率求和后取负。
• 超参：1个epoch，有效batch 64，峰值学习率2.0e-5，cosine、5% warmup，weight decay 0.01，梯度裁剪1.0，AdamW。以Qwen3-4B估算，reasoning-only约105M token、约540步；推理加文档约198M token、约1010步。作者称步数随打包和最后一批处理而变。

接到SFT与GRPO

• SFT：损失只在assistant token上。3个epoch，每套mix共10000条，其中9000条训练、1000条验证。pure_safe只有安全例，pure_general只有通用例，balanced二者数量相等。轨迹沿用同一推理模式，并过滤到与参考答案和安全标签一致。
• GRPO：5000条提示词、1个epoch，学习率1e-6，rollout温度0.9、top-p 0.95，每条8个rollout。奖励裁判仍是Qwen3-32B，五分制。安全提示词按安全与有用性评分；通用提示词只评任务表现，不打安全分。因此pure_general GRPO既无安全提示词，也无安全奖励。

Experimental Setup

• Models evaluated: Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, and Llama-3.2-1B, Llama-3.2-3B. mid-training is added on the corresponding base; Qwen3 is evaluated both before and after SFT, and the Llama experiments are without SFT.
• Comparisons: Base, Base+SFT, Base+SFT (Distill Sonnet-4.6), SP (Safety Pretraining data, about 2.7B tokens), AP (safety mid-training data for Alignment Pretraining, about 1.3B tokens). The SFT data for SP/AP is generated by reasoning-only safety-midtrained checkpoints of the same scale. Self-generation uses the base model itself; the ablation additionally uses Claude Sonnet 4.6 to produce documents or traces.
• Benchmarks and samples: 9 safety and 9 utility benchmarks, listed in the setup field. The main comparison's safety average excludes WildJailbreak and BeaverTails, and the utility average excludes GSM8K and MATH-500, because they share the same source as mid-training. 300 examples per benchmark, or the full set if fewer than 300. Table 22, Figure 1 and Figure 4(a) report the full-set averages.
• Metrics and judge: Lower ASR is safer; Safety is defined as 100%−ASR. Utility is accuracy except for AlpacaEval, which is win rate, then averaged across benchmarks. Free-form responses are judged by GPT-4.1-mini (temperature 0); multiple-choice questions use rule-based extraction. Filtering and GRPO reward use Qwen3-32B.
• GRPO scope: Only on Qwen3-1.7B and Qwen3-4B, comparing balanced and pure_general, 5000 examples each, 1 epoch.
• Judge agreement: In Appendix C.4, on 500 stratified samples, the agreement rates of Qwen3-Next-80B-A3B, Claude Sonnet 4.6, GLM-5 and GPT-4.1-mini are 97.8%, 98.2%, 98.2%, respectively, mean 98.1%. The paper does not report the number of training repetitions.

Main Results

The table below is the comparison average from Table 2 (excluding the same-source benchmarks above), not the full-set average.

Swipe horizontally to read the table

Model	balanced Ours ASR	pure_general Ours ASR	pure_general Base+SFT ASR	pure_general Distill ASR	pure_general Ours Utility
Qwen3-1.7B	8.9%	11.2%	26.3%	37.3%	61.7%
Qwen3-4B	5.5%	7.6%	13.6%	29.1%	74.9%
Qwen3-8B	3.4%	5.1%	11.6%	20.4%	78.2%

• Without SFT: In Table 1, Qwen3-1.7B's ASR/utility goes from 30.4%/29.8% to 8.4%/58.6%, 4B from 34.1%/47.6% to 6.6%/73.1%, 8B from 25.4%/70.3% to 4.9%/77.1%. The authors state the relative ASR reduction is 72% to 81%. The introduction also states that relative to the base in Tables 1 and 5, the reduction is 68% to 86%.
• After SFT: The authors state that under balanced, Ours+SFT has the lowest average ASR at every scale, with higher or comparable utility. The exception is Qwen3-8B, where Distill utility is 78.3% and Ours is 78.1%, but ASR is 10.5% vs 3.4% (Table 2). On Qwen3-4B, Base+SFT has ASR 10.8% and utility 65.6%, while Ours+SFT has 5.5% and 74.9%.
• Pure general SFT: The authors state that Ours+SFT has lower ASR than every baseline, with a reduction of 70% to 75% relative to Distill, and the highest utility at every scale. On Qwen3-4B, SP's ASR goes from 7.7% in balanced to 17.3% in pure_general, AP from 7.3% to 14.5%; Ours goes from 5.5% to 7.6%, with utility still 74.9%.

GRPO Trajectories

• Setup: Figure 3 compares Base+GRPO and Ours+GRPO on Qwen3-1.7B and 4B, each with balanced and pure_general. The x-axis is GRPO step, labeled up to 4000; the paper gives endpoint differences, not point-by-point readings.
• Mixed data: The authors state the mid-trained model leads throughout, with higher final safety and utility. Qwen3-4B's final advantage is 5.9 points in safety and 4.0 points in utility. The authors state that on Qwen3-1.7B, Base+GRPO loses part of its earlier gains later on, while Ours+GRPO is more stable at the end.
• No safety reward: When switched to pure general prompts, the base's final safety drops more. On Qwen3-4B, the base drops 13.2 points and Ours drops 1.4 points, with little change in utility. The authors state Qwen3-1.7B shows the same pattern; the paper does not give the corresponding point values for that scale.

Pattern Fields and Unsafe Proportion

• Removing fields: Figure 4(a) and Tables 29–30, on Qwen3-4B without SFT, remove one of intent, knowledge or consequence at a time. The authors state the full pattern has the best average ASR and utility, and removing any component lowers both. The appendix states that the average ASR on DoNotAnswer and JailbreakHub rises from 10.9% to 19.2%, 16.7%, 21.9%. When knowledge or consequence is removed, the comparison utility average drops by 11.7 and 16.4 points, respectively. The full-set utility in the figure is labeled 74.4, 73.0, 65.0, 61.7.
• Proportion sweep: Figure 4(b) and Table 31 fix the total amount, with unsafe proportions of 0%, 25%, 50%, 75%, 100%. The safety average in that table includes only HarmBench, AdvBench, WildJailbreak. At 0%, average ASR is 46.6%, XSTest over-refusal 1.1%, utility average 80.2%; at 25%, 8.5%, 9.7%, 81.6%; at 100%, ASR 0.0%, over-refusal 88.0%, utility 45.0%. The authors state 25% gives the best overall balance. The unsafe proportion in the corpus statistics is 23% (Figure 5), which is not the same number as the 25% in this sweep.

Generation Model and Cross-Family

• Who generates the data: Table 4. Without SFT, the authors state the two are overall similar; on Qwen3-8B, Sonnet-4.6's ASR is 4.0% and self-generation is 4.9%. After balanced SFT, self-generation has lower ASR at every scale: 1.7B 8.9% vs 10.9%, 4B 5.5% vs 9.2%, 8B 3.4% vs 5.8%. For 8B, post-SFT utility is Sonnet 78.9%, self-generation 78.1%.
• Llama-3.2: Table 5, without SFT. 1B's ASR/utility goes from base 31.3%/23.6% to self-generation 10.1%/29.4%; 3B from 41.4%/24.9% to 5.9%/47.8%. Sonnet generation on 1B has utility 22.1%, lower than base's 23.6%, and ASR 14.4%, still lower than base. The authors state self-generation gives the strongest overall average at both scales.

Other Ablations and Analyses

• Stage rather than just swapping data (Table 3, Qwen3-4B, reasoning-only): standard reasoning SFT has ASR/utility 22.8%/68.1%; doing SFT directly on the same data is still 22.8%/68.9%; changing to mid-training gives 7.5%/74.7%; adding SFT gives 5.4%/74.6%.
• Document records (Table 6, average of three SFT mixes): reasoning-only already brings the main change; adding self-generated documents gives 1.7B 9.2%/61.3%, 4B 5.9%/75.0%, 8B 3.9%/78.4%. On 4B, Sonnet documents have ASR 4.9%, lower than self-generated documents' 5.9%, but utility 71.6%, lower than 75.0%.
• SFT trace source (Table 7, mix average): after mid-training, self-generated SFT has ASR/utility 9.2%/61.3% for 1.7B, 5.9%/75.0% for 4B, 3.9%/78.4% for 8B; switching to Sonnet distilled traces gives 11.9%/59.9%, 9.4%/73.7%, 6.7%/78.0%.
• Filter pass rate (Figure 6, Appendix A.4): of 372,486 reasoning candidates, 343,088 have overall_pass true (92.1%); of 185,290 documents in the first round, 112,783 pass (60.9%). The proportion with quality at least 4 does not include the condition "all rubrics pass".
• Qualitative examples: Appendix E compares the output differences of Base+SFT and the full pipeline on directly harmful requests, jailbreak-wrapped requests and factual recall. The appendix gives examples; requests or answers are not reproduced here.

Limitations and future directions noted by the authors

• Model scale: Experiments concentrate on Qwen3 and Llama-3.2, up to 8B parameters (Limitations).
• Evaluation scope: The evaluation suite covers a set of safety and utility benchmarks, but cannot cover every domain, language, or real interaction mode (Limitations).
• Future directions: The authors write that future work can extend to larger models, and to more pre-training and post-training settings (Limitations).

Experimental coverage

• Models: The experiments state they ran Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base, as well as Llama-3.2-1B, Llama-3.2-3B (Section 4, Table 5).
• Post-training: SFT has three mixes: pure_safe, pure_general, balanced (Table 2). GRPO is on Qwen3-1.7B and Qwen3-4B, each with 5000 prompts, 1 epoch, including pure_general with no safety prompts and no safety reward (Section 4.2, Appendix B.3).
• Average scope: The main comparison removes WildJailbreak, BeaverTails, GSM8K, MATH-500, which share the same source as mid-training; the full-set numbers are in Appendix D, and Table 22 and Figure 1 report the full-set average. At most 300 examples per benchmark (Section 4.1).
• Judge: Free-form metrics use GPT-4.1-mini; Appendix C.4 reports agreement rates with Qwen3-Next-80B-A3B, Claude Sonnet 4.6, GLM-5 on 500 entries, mean 98.1%.
• Repetitions not stated: Appendix B lists the hyperparameters and time cost of mid-training, SFT, and GRPO; the paper does not report the number of repetitions of these trainings.

The authors treat safety as a foundational capability that can be learned together with general reasoning, rather than only supplementing refusal in post-training. Safety mid-training is inserted between pre-training and post-training: the base model, following the same structure, self-generates documents and reasoning traces for safety and utility prompts, and after filtering by Qwen3-32B, continues training with causal language modeling.

• No SFT: In Table 1, the average ASR for Qwen3-1.7B/4B/8B goes from 30.4%, 34.1%, 25.4% to 8.4%, 6.6%, 4.9%, and utility goes from 29.8%, 47.6%, 70.3% to 58.6%, 73.1%, 77.1%. The authors state a relative ASR reduction of 72% to 81%. The introduction gives another figure for the base of Tables 1 and 5: 68% to 86%.
• SFT: Under the average in the comparison of Table 2, the ASR of Ours+SFT in both balanced and pure_general is lower than the listed baselines. In pure_general, the Ours ASR for 1.7B/4B/8B is 11.2%, 7.6%, 5.1%, and utility is 61.7%, 74.9%, 78.2%. The authors state a relative ASR reduction of 70% to 75% versus Distill.
• GRPO: On Qwen3-4B, the final advantage of mid-training initialization is safety 5.9 points, utility 4.0 points. Without safety reward, safety declines by 1.4 points for Ours and 13.2 points for base (Figure 3).
• Elsewhere: For Llama-3.2-3B without SFT, the average ASR/utility of self-generated mid-training is 5.9%/47.8%, and base is 41.4%/24.9% (Table 5). When the same reasoning data is directly used for SFT, ASR is still 22.8%, and after changing to mid-training it is 7.5% (Table 3).

The authors' conclusion is: putting safety and general reasoning together in mid-training can mitigate the trade-off between safety and utility, and keeps safety effective when only general data follows, or even when GRPO has no safety reward; starting from a mid-trained model, subsequent RL can reach a better safety–utility frontier. The authors plan to release the generated data and documents.