Large reasoning models learn better alignment from flawed thoughts

1. 01What problem does this paper attempt to solve?

   How to train an LRM to recover from misleading reasoning, rather than relying on self-correction to emerge spontaneously.

   How to train an LRM to recover from misleading reasoning, rather than relying on self-correction to emerge spontaneously at inference time.

   • Where it arises: A frontier LRM first generates a structured CoT and then a final answer, but filling a phrase into the CoT can bypass alignment and elicit an unsafe completion. The authors state that both open-source models and commercial APIs support CoT prefilling, and models are still easily led astray by flawed premises injected into the thought process.
   • Limitations of existing work: Online RL commonly uses algorithms such as GRPO, with rewards mostly on the correct final answer; at inference time, however, the trajectory may start from an incorrect step. The authors report in Section 2 that once starting from a flawed CoT, the model follows unsafe reasoning into a harmful completion and cannot recover on its own.
   • Core observation: Continuing the reasoning trace of a safer model raises the safety score, while continuing the trace of a weaker model lowers it. The authors call this a generalization gap: training rewards only the final answer, while inference must handle noisy trajectories. Appendix B states the same phenomenon appears in over-refusal and math.
   • Threat model:
     • The goal is to make the completion follow an injected flawed premise, thereby bypassing alignment, or to elicit over-refusal on benign questions.
     • An attacker can inject a prefix into the CoT. Full-text CoT hijacking in Section 5.3 assumes a malicious user fully controls the reasoning trajectory and forces the model to skip its own reasoning and directly generate an answer.
     • IPR first injects a prefix, then appends a transition sentence asking the model to ignore prior reasoning and re-evaluate, then re-injects the same or slightly modified prefix, resetting for a repeatable k rounds.
     • The victim system is an LRM that generates a CoT first. The counter-aligned prefixes used in training come from external models; the authors state deployment does not require exposing the CoT. Section 5.5 describes GCG as white-box gradient optimization.
   • What this paper does: Proposes RECAP (Robust Safety Alignment via Counter-Aligned Prefilling), which mixes counter-aligned CoT prefills with standard prompts in post-training, using standard RLHF rewards to force the model to rewrite flawed trajectories. The authors state this adds no RL optimization steps and does not change the architecture.

   Collapse this answer

2. 02What related research exists?

   The paper situates RECAP outside SFT safety reasoning, auxiliary losses, RLHF that optimizes only the final answer, and inference-time reflection or external guardrails.

   Appendix L and Section 6 divide prior work into data fine-tuning, loss shaping, online RL, and inference-time intervention. The authors' point of comparison is: after these lines of work, CoT prefilling can still bypass alignment.

   SFT on safe reasoning

   • SafeChain and STAR-1: SafeChain by Jiang et al. (2025) and STAR-1 by Wang et al. (2025b) construct safety data with reasoning and use SFT to improve safety and reduce over-refusal. Table 2 lists STAR, SafeChain, and same-data SFT as baselines.
   • Comparison in the paper: The text states that training only on safety data improves safety scores but reduces helpfulness on the over-refusal benchmark. The Table 2 exception for this sentence is discussed in Question 4; here, the exception is not written as these works' own conclusions.

   Additional losses

   • Intermediate reasoning shaping: SARO by Mou et al. (2025), SafePath by Jeung et al. (2025), STAIR by Zhang et al. (2025c), RealSafe-R1 by Zhang et al. (2025b), and Zhang et al. (2025d) use additional losses to penalize unsafe intermediate reasoning or reward safe trajectories. Appendix L only categorizes them.
   • SafeKey: Zhou et al. (2025c) amplifies the aha-moment in safe reasoning. The paper also only lists it under this direction.

   Online RL

   • GRPO and DAPO: GRPO by Shao et al. (2024) and DAPO by Yu et al. (2026) are used to improve reasoning and alignment. RECAP modifies the token range involved in updates in DAPO; the main experiments use DAPO as the comparison, and Appendix F additionally uses GRPO.
   • Only final answers rewarded: The authors note that most RLHF still optimizes only the final answer (Lambert et al., 2024). Rager et al. (2025) show that prefilling CoT with a phrase can bypass alignment.

   Inference-time intervention

   • Backtracking: Zhang et al. (2024) introduce the RESET token, restarting generation after undesirable trajectories, trained with SFT and DPO, performing only one explicit restart. Appendix G compares with RECAP on DSLlama-8B.
   • External guardrails and inference-time reflection: Llama Guard by Inan et al. (2023), and inference-time sampling discussed by Ichihara et al. (2025). The authors state that such methods rely on runtime intervention.

   Baselines and benchmarks: Alignment baselines are STAR, SafeChain, same-data SFT, and DAPO. Safety benchmarks are StrongREJECT and its prefill variant, WildJailbreak, Fortress; over-refusal is XSTest and FortressOR; math is MATH500, GSM8K, AIME2024.

   The authors position RECAP as: turning anti-alignment prefill directly into training supervision, forcing the model to cover flawed trajectories, making robustness not dependent on inference-time restarts or external guardrails.

   Collapse this answer

3. 03How does the paper address this problem?

   RECAP uses online RL with counter-aligned CoT prefixes, optimizing only tokens after the prefix, forcing the model to rewrite toward safe or helpful answers.

   RECAP writes opposite-direction CoT prefixes into online RL rollouts, then uses a reward that looks only at the final answer to force the policy to rewrite that trajectory. The prefix appears only during training.

   Problem Setting

   An LRM is denoted πθ. Given a prompt x, the output is split into intermediate reasoning ycot and final answer yresp. To study brittleness, an externally generated, syntactically fluent but semantically biased or counter-aligned prefix is injected. The model continues on the prompt plus prefix, then gives an answer. The full output is y=(y_cot^pre∥ y_cot^gen, y_resp), i.e., the concatenation of prefix, continuation, and final answer.

   Counter-Aligned Prefixes

   • Harmful prompts: The prefix comes from a weakly safety-aligned or helpful-only model πharm. The main experiments use DSQwen-7B, temperature 0.9, top-p 0.95, sampling 5 trajectories per question. LlamaGuard keeps trajectories judged unsafe; when none of the five is unsafe, the one with the lowest "safe" class probability is taken, then truncated to ℓpre words.
   • Benign prompts: First SFT DSQwen-7B on STAR-1's safety-only subset to obtain πrefuse, which tends to refuse, then sample refusal-oriented reasoning as an over-conservative prefix.
   • Scope of use: Augmentation is applied only to a proportion α of samples in the training set. The prefix is fixed context, not optimized tokens. Following the prefix gives unsafe instructions or over-refusal.

   Optimization Objective

   The framework is DAPO. With a prefix, the importance sampling ratio and within-group advantage are computed only for tokens after the injected prefix. With prefill, t₀(x)=ℓ_pre+1; otherwise t0(x) is 1. Clipping thresholds follow DAPO's εlow and εhigh. Dynamic sampling drops prompts in a group where all rewards are 0, or all are above a threshold τ; the paper does not give τ. The reward Ri is determined only by the prompt and final answer.

   Rewards, Data, and Hyperparameters

   • Three rewards: Safety uses Granite-Guardian-3.1-8B's logits as a continuous reward; over-refusal is scored by Llama-3.1-8B-Instruct on the scale in Appendix C, with allowed values 0, 0.5, 1, and Appendix C gives the full prompt; math uses RLVR, giving reward for verifiable correct answers.
   • Corpus: 5K items: 1K harmful prompts from BeaverTails, 1K over-refusal prompts from STAR-1, and 3K math prompts from GSM8K and MATH. Math prompts are not CoT-prefilled.
   • Default settings: α=0.5, ℓpre=500, G=16 rollouts per question, truncation length 8000, εlow=0.2, εhigh=0.28, training for 1 epoch. Main policies are DSLlama-8B and DSQwen-14B.

   Deployment and Evaluation

   Counter-aligned prefills are used only in training. The authors state the learned rewriting transfers to ordinary inputs without prefixes, and inference does not require exposing the CoT. Main evaluations use GPT-4o to judge the proportion of completions that are safe or helpful; Fortress uses an item-by-item scale. Math reports pass@K.

   Collapse this answer

4. 04What experiments did the paper conduct?

   RECAP is higher than DAPO in Table 2's prefill columns, and DSQwen-14B's StrongREJECT is the same at 99.04.

   Experimental Setup

   • Policy models: Main results are DSLlama-8B and DSQwen-14B. Table 5 additionally trains Qwen3-4B. Table 1 also reports DSQwen-1.5B, DSQwen-7B, DSQwen-32B, and Qwen3-4B-Thinking-2507.
   • Data and prefixes: Training uses 5K prompts from Question 3, with prefixes from DSQwen-7B. The text states that robustness evaluations use different prefill sources. The prefill variants in Table 2 inject DSQwen-1.5B's flawed CoT; the paper does not state the word count for that variant.
   • Baselines: STAR, SafeChain, SFT on the same data, DAPO. Appendix G additionally compares Backtracking.
   • Benchmarks: StrongREJECT has 313 items total; jailbreak uses WildJailbreak and Fortress (500 expert adversarial prompts); over-refusal uses XSTest and the benign subset of Fortress; math uses MATH500, GSM8K, AIME2024. Unseen attacks are CodeAttack with three encodings and GCG on AdvBench, plus 520 items from ReNeLLM.
   • Metrics and evaluation: Safety score and helpfulness score are the percentage of completions judged safe or helpful. Unless otherwise stated, main evaluations use GPT-4o; Fortress uses an item-by-item scale. pass@K is 1 on MATH500 and GSM8K, and 16 on AIME2024. Table 2 is the average of three independent runs. CoT hijacking throughout uses greedy decoding with a maximum of 8000 tokens. The paper does not state the decoding temperature for the main evaluations. The math averages in Appendix I are not the same set of numbers as Table 2; the main results follow Table 2.
   • Training-side models: The safety reward is IBM Granite-Guardian-3.1-8B, the over-refusal judge is Llama-3.1-8B-Instruct, and prefix filtering uses LlamaGuard.

   Main Results

   The table below follows Table 2. XSTest is helpfulness score, MATH500 is pass@1, and the rest are safety scores. STAR, SafeChain, SFT, and Fortress, Fortress OR, GSM8K, AIME2024 are omitted.

   Swipe horizontally to read the table

   Model and method	StrongREJECT	REJ-Prefill	WildJailbreak	XSTest	MATH500
   DSLlama-8B Original	57.83	45.05	24.40	90.40	81.00
   DSLlama-8B DAPO	96.81	79.23	72.90	78.00	82.20
   DSLlama-8B RECAP	99.68	98.70	88.75	91.87	83.60
   DSQwen-14B Original	68.69	49.20	34.40	96.00	86.40
   DSQwen-14B DAPO	99.04	80.51	77.60	96.80	88.80
   DSQwen-14B RECAP	99.04	98.08	91.65	96.80	90.00

   • The authors state the gains transfer to prompts without prefill at inference time. The introduction states relative to vanilla DAPO an average of +12.3% (directly harmful), +21.0% (jailbreak), +7.8% (over-refusal helpfulness score). In Table 2, on StrongREJECT and XSTest for DSQwen-14B, RECAP and DAPO are tied.
   • Fortress, not listed in the table above: on DSLlama-8B, RECAP is 86.84 and DAPO is 68.86; on DSQwen-14B, 80.17 and 67.85. On GSM8K, DSLlama-8B's RECAP is 93.72 and DAPO is 93.71. On AIME2024, RECAP is 70.00 (DSLlama-8B) and 86.67 (DSQwen-14B), the same as each respective Original.
   • The authors state that STAR and SafeChain, trained only on safety data, reduce over-refusal helpfulness scores. In Table 2, DSLlama-8B's SafeChain on XSTest is 90.40, the same as Original; DSQwen-14B's SafeChain on Fortress OR is 96.40, higher than Original's 95.00.

   Adaptive Attacks and Self-Reflection

   • Full-text hijacking: The entire CoT is replaced, followed by an end-of-thinking marker. In Table 3 on StrongREJECT, DSLlama-8B's DAPO is 70.29 and RECAP is 98.08; DSQwen-14B is 73.48 and 96.49. The authors state an average improvement of 35.4%.
   • IPR: On StrongREJECT, the prefix comes from DSQwen-1.5B, reset for k rounds. Table 4 does not state the policy model. DAPO drops from 79.23 at k=1 to 69.65 at k=3; RECAP drops from 98.72 to 97.44. The authors state the decrease flattens as k increases.
   • Reflection: The Figure 5 caption states RECAP more often corrects mid-trajectory; it does not restate the example text. In Table 7 on StrongReject-Prefill, the proportions of first reflections falling in the four equal intervals of the generated CoT are 31.1%, 32.1%, 24.8%, 12.0%. On WildJailbreak without prefix, 74.2% of RECAP examples exhibit self-reflection, versus 43.9% for DAPO. The paper does not state the policy model for these two items.

   Cross-Family Transfer, Unseen Jailbreaks, and Judges

   • Table 5: Qwen3-4B policy. With prefill from DSQwen-7B, Strong Reject is 99.82; from LLaMA-3.2-1B, 99.68; DAPO is 95.60. Both RECAP variants have Rej-Prefill of 99.68 and Fortress OR of 92.80. The authors state the key is that the prefix is counter-aligned with the target behavior, not that it comes from the same model family.
   • Table 6: DSLlama-8B, these attacks were not used in training. RECAP is higher than DAPO in all five rows, e.g., CodeAttack (Stack) is 78.85 vs. 51.92, and GCG is 88.08 vs. 70.96. The authors state the gains cover code obfuscation, semantic rewriting, and gradient optimization.
   • Appendix E.1: 250 items total from StrongReject and WildJailbreak, three human annotators, plus GPT-4o, Granite-Guardian-3.1-8B, Llama-3.1-8B. Human safety scores are 82.00±1.02 for DAPO and 92.80±0.74 for RECAP; Cohen's κ against the human majority label is 0.76–0.83. The paper does not state the policy model.

   Other Ablations and Analyses

   • Table 1, first 200 words: the authors state that after the four DS models continue the DSQwen-1.5B prefix, the average safety score drops by 36.4%; after continuing the DSQwen-32B prefix, it rises by 91.7%. Qwen3-4B goes from 95.21 to 76.68 and 84.35, which the authors state is a drop of 19.5% and 11.4%.
   • Table 17, DSLlama-8B, only 2K alignment prompts: WildJailbreak at α=0.25 is 67.15, lower than 69.15 at α=0; XSTest at α=1 is 89.80, lower than 92.00 at α=0.5. The authors adopt α=0.5.
   • Table 17 length: StrongREJECT at ℓpre=700 is 98.72 and XSTest is 86.80, lower than 99.35 and 91.20 at 500 words; Fortress rises from 82.99 to 86.45. The authors still use 500 as the default and state that at 700 words both safety and helpfulness scores decline.
   • Table 17 source: StrongREJECT with unsafe prefix is 98.05, without prefill is 97.44, and with safe prefix is 71.88. The authors state that safe prefixes are directly exploited and the model does not learn to correct.
   • Figure 2, DSQwen-14B: RECAP's CoT/total tokens on safety tasks is 318/707, DAPO's is 286/625. The authors state the total budget is comparable to DAPO. Figure 3 has no coordinate readings; the authors state that without prefill, DAPO is higher initially, then RECAP surpasses it.
   • Table 13: Backtracking's REJ-Prefill is 67.51, lower than DAPO's 79.23. In Table 11's binary reward and Table 12 (whose title says GRPO, though the row names still say DAPO), the RECAP row is higher than the comparison row.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors leave the selection of supervision from flawed math trajectories as an open question and list hybrid reasoning and multilingual-multimodal directions.

   Limitations and Future Directions Noted by the Authors

   • A deeper understanding of alignment and reasoning failure modes may also open new avenues for misuse or model manipulation; the authors state that publicly researching these weaknesses is necessary (Limitations).
   • Future work could extend to hybrid reasoning models that combine multiple reasoning strategies (Limitations).
   • In multilingual and multimodal settings, reasoning brittleness may be more pronounced (Limitations).
   • Math differs from safety alignment: multiple trajectories can lead to correct or incorrect solutions. The authors state that determining which flawed trajectories provide effective supervision remains an open question (Limitations).

   Experimental Coverage

   • The main policies are DSLlama-8B and DSQwen-14B (Table 2). Table 5 uses Qwen3-4B as the policy, with prefills from DSQwen-7B or LLaMA-3.2-1B. Table 1 additionally has prefill cases for DSQwen-1.5B, 7B, 32B, and Qwen3-4B-Thinking-2507.
   • Training uses 5K prompts, 1 epoch, with DAPO as the main optimizer. Appendix F reports GRPO, as well as Granite-Guardian's binary labels and continuous logits. Math prompts are not prefilled during training.
   • Main safety and helpfulness scores are judged by GPT-4o; Fortress uses an item-by-item scale. Appendix E.1 re-evaluates 250 items with three human annotators and three automated judges. Math uses pass@1 on MATH500 and GSM8K and pass@16 on AIME2024; Appendix I additionally gives means and standard deviations over three seeds.
   • Stress tests include full-text CoT hijacking (Table 3), IPR with k=1 to 3 (Table 4), and CodeAttack, ReNeLLM's 520 items, and GCG (Table 6). Ablations vary α, ℓpre, and prefix source on DSLlama-8B's 2K alignment prompts (Table 17). Appendix B reports prefill cases on MATH500 and XSTest.
   • The paper does not report statistical significance tests. Tables 4, 7, and the 250-item human subset do not state the policy model.

   Collapse this answer

6. 06Summarize the main content of the paper

   RECAP uses RL post-training with counter-aligned prefills to cover flawed CoTs, and is compared against DAPO on safety, helpfulness, and math.

   RECAP is a post-training recipe for LRMs: in RL, deliberately provide a CoT prefix with the opposite direction, making the model steer the trajectory back to a safe and helpful answer.

   For harmful questions, the prefix comes from the unsafe reasoning of a weakly aligned model; for benign questions, the prefix comes from the refusal reasoning of an over-conservative model. Following the prefix does not yield high reward. Updates cover only tokens after the prefix. Math questions are not prefilled, and no prefix is injected at inference time. Main experiments train DSLlama-8B and DSQwen-14B in DAPO, with training prefixes from DSQwen-7B.

   In the three-run average of Table 2, on DSLlama-8B, RECAP's StrongREJECT, prefill variant, and WildJailbreak safety scores are 99.68, 98.70, 88.75, while DAPO's are 96.81, 79.23, 72.90; XSTest helpfulness score is 91.87. DSQwen-14B's prefill safety score goes from DAPO's 80.51 to 98.08, while StrongREJECT is the same as DAPO at 99.04. Under full-text hijacking in Table 3, the two models' RECAP scores are 98.08 and 96.49. In Table 6, DSLlama-8B's GCG safety score is 88.08.

   The authors state that over-refusal decreases, math pass@K is maintained, reasoning token budget is similar to DAPO, and self-reflection appears more often; even with repeated CoT resets, safety scores remain higher than DAPO. The authors conclude that writing flawed reasoning into RL can make alignment more robust under both clean prompts and adversarial traces, without changing the standard RLHF process.

   Collapse this answer

Related papers