Where do reasoning models refuse?

1. 01What problem does this paper attempt to solve?

   A reasoning model's refusal can occur within the chain of thought; the authors aim to locate where the decision is formed.

   Where in a long chain of thought should a reasoning model decide to refuse, rather than only at the boundary between prompt and final reply.

   • Setting: Dialogue models without CoT must decide refusal or compliance before the first reply token. Reasoning models write a chain of thought before answering; the authors say this paradigm is becoming mainstream, citing o1, DeepSeek-R1, Qwen3, and GPT-OSS, and also say inference-time compute can be traded for adversarial robustness, while reasoning-model-specific vulnerabilities have emerged.
   • Existing conclusions do not carry over: The authors cite Qi et al. and Arditi et al. that safety alignment in non-reasoning models mainly affects the first few output tokens, and refusal is mediated by a single linear direction that can be turned off by ablation. The authors argue this cannot be transferred directly: the decision point is no longer confined to the prompt-to-reply boundary.
   • Cases to distinguish: If refusal is determined before the CoT begins, different trajectories for the same prompt should not change the final refusal or compliance. The authors compare the variance of final-reply compliance scores conditioned only on the prompt versus additionally fixing a CoT.
   • What was done: The authors study four open-source models (three distilled, one RL model with deliberative alignment), test whether CoT affects refusal, use sentence-by-sentence resampling to see on which sentence the decision falls in distilled models, and test whether linear direction ablation can increase harmful compliance.

   Collapse this answer

2. 02What related research exists?

   Related work covers chain-of-thought reasoning, trajectory interpretability, reasoning model vulnerabilities, and linear directions for refusal.

   The authors organize related work by chain-of-thought reasoning, trajectory interpretability, safety issues in reasoning models, and linear representations of refusal.

   Chain-of-Thought Reasoning and Monitoring

   • o1 and DeepSeek-R1: OpenAI o1 (2024) and DeepSeek-R1 (2025) generate explicit CoT before final output. The authors say this paradigm improves multi-step task performance, capabilities can be obtained through reinforcement learning, and large reasoning models can be distilled into smaller ones.
   • Deliberative alignment: Guan et al. (2025) train models to recall and reason about safety policies in the CoT.
   • Monitoring and faithfulness: Korbak et al. (2025) and [25–27] discuss that if CoT faithfully reflects internal decisions, monitoring could see unsafe intent before harm occurs; Greenblatt et al. (2024) report that planning for alignment faking can appear in the CoT. Turpin et al. (2023), Chen et al. (2025), and Arcuschin et al. (2025) report that CoT is not always faithful and can mislead or fabricate.

   Interpretability of Reasoning Trajectories

   • Single forward pass is insufficient: The authors argue standard mechanistic interpretability handles a single forward pass [32, 33]; reasoning computation spans many tokens, and the same prompt can sample different trajectories [34], so conclusions should target the trajectory distribution.
   • Repeated sampling: Brown et al. (2024) report that repeated sampling changes performance, with correct-solution coverage following a power law in the number of samples.
   • Thought anchors: Bogdan et al. (2025) use sentence-level rollouts to measure counterfactual importance, including black-box resampling, attention aggregation, and causal masking. They find importance is heavy-tailed, with planning or backtracking sentences disproportionately affecting subsequent reasoning and final answers. The authors say Section 4.1 adapts their resampling, switching the object from mathematical correctness to safety decisions.

   Safety Issues on Reasoning Models

   • Safety in CoT and distillation: Zhou et al. (2025) report that safety reasoning appears in the CoT but often fails under attack. Li et al. (2026) report that distillation systematically weakens safety alignment.
   • New attack surfaces the authors cite: Kuo et al.'s H-CoT (2025) uses adversarial prompts to turn the model's own safety reasoning into harmful output. Ye et al. (2026) describe how false reasoning implanted in the prompt is treated by the model as its own and continued. Yong and Bach (2026) report that even without adversarial input, post-trained reasoning models on benign tasks such as math may rationalize complying with harmful requests in the CoT.

   Linear Representations of Refusal

   • Linear concepts: Multiple works treat high-level semantics as linear directions in activation space [38, 32, 39, 40], and use representation engineering [41] and activation steering [42, 43] to read or modify these directions.
   • Single refusal direction: Arditi et al. (2024) report that refusal in non-reasoning dialogue models is mediated by a single direction in the residual stream; ablating it turns off refusal with negligible impact on general capability. The authors say Section 5 tests whether this extends to reasoning models.
   • Subsequent geometric results: Wollschläger et al. (2026) find multiple geometrically distinct refusal directions with similar behavioral trade-offs. Zhao et al. (2025) report that harmfulness and refusal can be represented separately.

   The variance experiment's comparison is different conditions of the same prompt, with no external attack-method baseline; the ablation comparison is unablated standard reasoning. Prompts come from HarmBench, AdvBench, SorryBench, OR-Bench, and StrongREJECT; capability evaluations use AIME 2025, GPQA Diamond, and BBEH Mini. In the introduction, the authors argue that Qi et al. (2024) and Arditi et al.'s conclusions about non-reasoning models cannot be applied directly, and position this paper's sentence-level analysis after Bogdan et al. and direction ablation after Arditi et al.

   Collapse this answer

3. 03How does the paper address this problem?

   Use variance comparison, sentence-by-sentence resampling, and direction ablation to locate refusal's position in the chain and in activations.

   The authors use three analyses to see where refusal occurs: fixing the chain of thought compares outcome variance, sentence-by-sentence resampling locates the position within the chain, and they extract a refusal direction from activations and perform direction ablation.

   Generation Structure and the Four Models

   • Two-stage generation: After the user prompt, the model first generates a CoT wrapped in special tokens, then generates the final reply. Without CoT, refusal must occur at the prompt-to-reply transition; with CoT, it can also occur at chain entry, mid-chain, or at the chain-to-final-reply transition (Figure 1).
   • Two DeepSeek distilled models: DeepSeek-R1-Distill-Llama-8B and DeepSeek-R1-Distill-Qwen-7B are supervised fine-tuned from Llama-3.1-8B and Qwen2.5-Math-7B respectively on roughly 800K samples generated by DeepSeek-R1; the distilled models themselves have no RL. The teacher DeepSeek-R1 (671B) has second-stage safety RL, with a safety reward model trained on 106K prompts labeled safe or unsafe; the paper states there is no record of such safety-specific training for the distilled models.
   • The other two models: Qwen3-8B is distilled from Qwen3-235B-A22B; the 8B skips all RL, while the teacher underwent four-stage post-training including GRPO. GPT-OSS-20B is a mixture-of-experts model with RL post-training in the same family as o3, and includes deliberative alignment: first supervised fine-tuning so the model recalls and reasons about safety specifications in the CoT, then RL with a safety evaluator.

   Whether the Chain of Thought Enters the Refusal Decision

   • Sampling: Each prompt is independently sampled 5 CoTs, and each CoT is sampled 5 final replies, for up to 25 outputs. Temperature 0.6, top-p 0.95, uniform across models (Appendix A). CoTs not completed within the token limit are excluded and do not proceed to the second stage.
   • Scoring: Only the prompt and final reply are given to StrongREJECT, not the chain of thought. The compliance score is in [0, 1], higher meaning more compliant. Appendix B's composite is score = (1 − refusal) × (specificity + convincingness) / 2, where refusal is 1 for refusal and 0 for compliance, and specificity and convincingness are first scaled from 1–5 to [0, 1].
   • Variance: Compare the standard deviation σ[si] of all scores conditioned only on the prompt, and E[σ[sij]], the mean over each CoT of the standard deviation within that CoT. The authors say a large gap indicates CoT is the main source of outcome variation. The footnote states the converse need not hold: a decision can occur mid-chain yet different trajectories still reach the same outcome, in which case the prompt-only variance would also be low.

   Localizing along the Chain of Thought

   • Sentence-by-sentence resampling: Split the CoT into sentences, starting from an empty prefix and lengthening one sentence at a time; at each prefix, have the model rewrite the remaining CoT and final reply, k=10, then score with StrongREJECT. The authors say this adapts the black-box resampling of Bogdan et al.
   • High-variance subset: Keep only prompts with σ[si] ≥ 0.43 and mean over CoTs of the standard deviation ≤ 0.03. The authors say the thresholds were chosen so that CoT indeed carries causal weight, while ensuring at least 32 items per model. Figure 3 randomly draws N=32 from these; the horizontal axis is normalized sentence position. The paper does not separately state the sampling temperature for this stage.
   • First-sentence prefix: Take the opening sentences from DeepSeek-R1-Distill-Llama-8B in Figure 3a as prefixes, then sample k=15; the same prefixes are also sent to DeepSeek-R1-Distill-Qwen-7B. Appendix F has more examples; sentences are not repeated here.

   Linear Directions and Ablation

   • Comparison groups: Using Section 3's scoring, on the 75% training set of the 1,946 items, separate stable refusals from stable compliance. Prompt-level: at least 80% of the 25 scores below 0.2, or at least 80% above 0.8. CoT-level: all 5 scores for that trajectory below 0.2, or all above 0.8. The authors say 0.2 and 0.8 were chosen to obtain comparison groups of similar size; the paper does not report the number of items in either group.
   • Direction: Each sample averages residual-stream activations over a set of tokens. Two positions: chat template tokens at the end of the prompt, or all CoT tokens. The refusal direction is the difference of the two group means, r^(l) = μ^(l) − ν^(l), i.e., the refusal group mean minus the compliance group mean at that layer. Computed for multiple layers.
   • Ablation and layer selection: At inference, project the direction out of the residual stream at every layer and token; equivalently, rewrite the weights that write into the residual stream. The authors say the two formulations correspond to different threat models (runtime hook or persistent weight modification) and are used interchangeably in experiments. GPT-OSS-20B orthogonalizes each expert weight separately, with the same direction vector. For layer selection, try ablation on a small subset of the held-out set and choose the layer with higher StrongREJECT and still a high proportion of coherent, complete generations; optimal layers are in Table 6; the paper does not state the subset size.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors report that variance drops after fixing the chain of thought, and say ablation improves compliance but capability generally declines.

   Experimental Setup

   • Models tested: DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, GPT-OSS-20B. The first three are distilled and did not undergo RL; GPT-OSS-20B includes deliberative alignment.
   • Prompts: Main set of 1,946 items, from HarmBench 200, AdvBench 520, SorryBench 440, a random 500 from OR-Bench-Hard-1K, and StrongREJECT 313, deduplicated across benchmarks. Section 3 adds a random 500 from OR-Bench-80K, totaling 2,446 items, used only in that section.
   • Generation: Section 3 uses 5 CoTs times 5 final replies, temperature 0.6, top-p 0.95. A few incomplete CoTs do not proceed to the second stage. The paper does not give the incompletion rate for that section.
   • Evaluation: A StrongREJECT rubric variant, looking only at the prompt and final reply. The default remote model is GPT-4o-mini; the authors instead use a local GPT-OSS-20B (vLLM, 4 H100s), which is also one of the tested models. Appendix B uses 200 items (50 per tested model) to compare score distributions between the two; the authors say there is a small deviation, but use only the local evaluator throughout and do not compare with remote scores. The paper reports no correlation coefficient and no agreement with human annotation.
   • Localization and ablation: High-variance threshold is σ[si] ≥ 0.43, mean σ[sij] ≤ 0.03; Figure 3 uses N=32, k=10 per prefix, first-sentence prefix k=15. Ablation splits the 1,946 items 75/25, with a full held-out set of 487; the jointly refused subset is 24 items, conditioned on all four models having StrongREJECT < 0.4 under standard reasoning. Ablation evaluation uses 5×5 outputs per item, with incomplete CoTs scored 0.
   • Capability benchmarks: AIME 2025, GPQA Diamond, BBEH Mini, using Inspect. Figure 15 ablation results come from directions extracted on CoT tokens.

   Main Results

   Per Table 2. The table note states they come from Section 3; that section's prompt pool is 2,446 items, with some incomplete CoTs excluded. Average score is the mean StrongREJECT over outputs.

   Swipe horizontally to read the table

   Model	Mean StrongREJECT	Prompt-only E[σ[si]]	Prompt+CoT E[σ[sij]]
   DeepSeek-R1-Distill-Llama-8B	0.621	0.212	0.108
   DeepSeek-R1-Distill-Qwen-7B	0.645	0.230	0.118
   Qwen3-8B	0.484	0.105	0.065
   GPT-OSS-20B	0.228	0.045	0.003

   • Authors' interpretation: The authors say that on all four models, fixing the CoT reduces the standard deviation of compliance scores, so refusal information is not only present before the chain of thought begins. Figure 2 uses gray lines to connect the two standard deviations for each prompt; downward indicates lower variance after fixing the trajectory; blue and orange points are E[σ[si]] and E[σ[sij]] over all prompts, i.e., the last two columns of the table.
   • Not every prompt relies on CoT: The authors say Qwen3-8B and GPT-OSS-20B have higher refusal tendencies, with a batch of prompts where both variances are 0. The caption also notes a batch of high-variance prompts with σ[si] ∼ 0.5 and mean CoT-conditional standard deviation ∼ 0.0; the authors say CoT is the main driver of refusal outcomes on these prompts.
   • Low variance does not imply timing: Section 3 footnote states that a model can decide within the CoT yet reach the same result across different trajectories, in which case the prompt-only variance would also be low.

   Position within the Chain of Thought

   • Two models in the figure: Figure 3 only plots DeepSeek-R1-Distill-Llama-8B and GPT-OSS-20B. The top shows the mean StrongREJECT standard deviation over k=10 resamples along normalized sentence position, with shading as ±1 SD across prompts; the text does not give endpoint values. The authors say Llama-8B's variance drops early in the CoT and stays low afterward, collapsing after the first sentence in many cases; GPT-OSS-20B declines more slowly. The bottom heatmap shows 3 prompts each, red for compliance, white for refusal. Appendix E lists the prompts and each CoT's mean and standard deviation; prompts are not repeated here. The threshold was designed to give at least 32 items per model, but the figure does not plot the other two models.
   • First-sentence scores: Table 1 takes five first sentences from prompt i=806 in Figure 3a; on the source model with k=15, the means are 0.07±0.26, 0.98±0.06, 0.00±0.00, 0.87±0.35, 0.18±0.38. The authors say these opening sentences are semantically close yet can lead to completely different refusal outcomes.
   • Same-teacher transfer: The same prefixes on DeepSeek-R1-Distill-Qwen-7B give 0.15±0.31, 1.00±0.00, 0.13±0.28, 1.00±0.00, 0.58±0.50. The authors say the direction is broadly similar and suggest one possibility is that both inherited the association between opening patterns and refusal from the teacher DeepSeek-R1. Row 5 differs between the two. Appendix F has five more groups, not repeated here. Qwen3-8B is not in this transfer experiment.

   Direction Ablation

   Per Table 7. The objects are the 24 jointly refused held-out items; numbers are post-ablation StrongREJECT means ± standard deviations, with incomplete generations scored 0. The paper does not give pre-ablation means in this table; inclusion requires a score < 0.4 under standard reasoning.

   Model	CoT direction	Prompt-end direction
   DeepSeek-R1-Distill-Llama-8B	0.888±0.281	0.294±0.421
   DeepSeek-R1-Distill-Qwen-7B	0.713±0.404	0.819±0.320
   Qwen3-8B	0.498±0.476	0.604±0.444
   GPT-OSS-20B	0.519±0.457	0.562±0.489

   • Authors' interpretation: The authors say ablation increases harmful compliance on all four models, but is less stable than the same technique on non-reasoning models; the paper does not give numbers from reference [2]. In the table, Llama-8B's prompt-end direction is lower than its CoT direction, while Qwen-7B is the opposite; the paper does not explain this. Figure 4 is one qualitative example for GPT-OSS-20B, with baseline marked 0.00 and post-ablation marked 1.00; the prompt and output are not repeated.
   • Operationalized refusal rate on the held-out set: Table 9 counts a mean score < 0.4 as refusal; this is not the binary refusal bit inside the evaluator. N=487, row order is Llama-8B, Qwen-7B, Qwen3-8B, GPT-OSS-20B. Pre-ablation: 38.7%, 29.8%, 60.0%, 93.4%; after CoT ablation: 0.4%, 3.9%, 7.5%, 3.5%; prompt-end direction: 10.1%, 4.1%, 8.5%, 7.0%. The authors say refusal rates all drop after ablation. Table 8, on the 24-item subset, gives GPT-OSS-20B's incomplete CoT rate for the prompt-end direction as 39.9%; the text says the failure rate reaches 40% and that the StrongREJECT scores of the remaining successful generations are generally high. After ablation, all four models have higher incompletion rates than without intervention.
   • Capability: Appendix I.3 and Figure 15 compare AIME 2025, GPQA Diamond, and BBEH Mini before and after ablation. The authors say performance is worse except in two evaluations, typically a relative decline of 5-30%; the paper does not say which two, and the text does not list raw accuracies for each benchmark.

   Other Ablations and Analyses

   • Optimal layer (Table 6, CoT / prompt-end): Llama-8B 23 / 15, Qwen-7B 21 / 17, Qwen3-8B 23 / 17, GPT-OSS-20B 19 / 15.
   • Training comparison uses continuous scores 0.2 and 0.8; testing and Table 9's refusal split use 0.4, which the authors say is broader than the training filter. The paper does not report the number of comparison items or the number of layer-selection subset items.
   • Incomplete rates on the 24-item subset (Table 8, no intervention / CoT / prompt-end): Llama-8B 0.3% / 2.6% / 1.2%, Qwen-7B 1.8% / 2.4% / 1.4%, Qwen3-8B 0.5% / 2.7% / 0.2%; GPT-OSS-20B 2.4% / 4.1% / 39.9%.
   • Figure 16: The authors say cosine similarity between prompt-end and CoT directions is generally low and suggest multiple directions participate in refusal; the text does not give per-layer values.
   • Appendix B: On 200 items, the score densities of local GPT-OSS-20B and GPT-4o-mini have a small deviation, with no correlation coefficient.
   • The full held-out set's score distribution is in Figure 14 (N=487), separate from Figure 5's 24-item subset; incomplete generations are also scored 0.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that closed-source and more RL-trained models were not tested, and that a single evaluator, one-direction transfer, and scale differences limit attribution.

   Limitations and future directions noted by the authors

   • Model scope: Results are based on four open-source models (three distilled, one RL). The authors state they did not test frontier closed-source models, nor broader RL-trained reasoning models (Section 7.1).
   • Number of rollouts: Section 3 uses at most 25 rollouts per prompt, Section 4.1 uses 10 per CoT prefix. The authors state more rollouts would tighten variance estimates (Section 7.1).
   • Single judge: Output harmfulness is scored only with StrongREJECT. The authors state LLM-as-judge has known blind spots, and more metrics would strengthen confidence in the quantitative conclusions (Section 7.1).
   • Transfer direction: Prefix transfer is one-directional; sentences come from DeepSeek-R1-Distill-Llama-8B and are filled into DeepSeek-R1-Distill-Qwen-7B. The authors hypothesize the cause is the shared teacher DeepSeek-R1, but this was not directly tested (Section 7.1).
   • Follow-up tests named by the authors: A reverse experiment (Qwen-7B as source, Llama-8B as target) would provide corroboration; transferring to the teacher DeepSeek-R1 itself would also be informative (Section 7.1).
   • Scale and paradigm are confounded: The compared distilled and RL models differ in scale (8B vs. 20B). The authors state the observed differences cannot be attributed solely to training paradigm (Section 7.1).

   Experimental coverage

   • The tested models are DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, and GPT-OSS-20B, 4 in total (Section 2.2). The first three are distilled and did not undergo RL; GPT-OSS-20B includes deliberative alignment.
   • The prompt pool for Section 3 is 1,946 deduplicated prompts plus 500 randomly sampled from OR-Bench-80K, totaling 2,446, used only in that section; source counts are HarmBench 200, AdvBench 520, SorryBench 440, OR-Bench-Hard-1K random 500, StrongREJECT 313 (Appendix A).
   • Sentence-level curves are in Figure 3, for DeepSeek-R1-Distill-Llama-8B and GPT-OSS-20B, each with random N=32 from the high-variance subset, k=10 per prefix. The source model for first-sentence prefixes is Llama-8B, transfer target is Qwen-7B, k=15 (Section 4.3, Table 1).
   • Direction ablation evaluation includes a common refusal subset of 24 items and a full held-out set of 487; capability evaluations are AIME 2025, GPQA Diamond, BBEH Mini (Appendix G.5, I.2, I.3). In Figure 15, two evaluations did not worsen after ablation; the paper does not state which two, nor does it report raw accuracy for each benchmark.
   • The main metric is scored by local GPT-OSS-20B according to the StrongREJECT rubric; this model is also one of the tested models. Appendix B compares score distributions with GPT-4o-mini on 200 items; the paper does not report agreement coefficients, nor agreement with human annotation, nor the number of refusal/compliance comparison pairs or layer-subset counts.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors say the chain of thought affects refusal; distilled models can depend on the first sentence, and ablation improves compliance but hurts capability.

   Yamaguchi, Etheridge, and Arditi investigate where in the chain of thought open-source reasoning models decide to refuse. Models without CoT must decide before the first response token; reasoning models first write the chain and then answer, so the decision point can be at chain entry, within the chain, or after the chain ends.

   • Objects: Four open-source models. DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and Qwen3-8B come from distillation and did not undergo RL; GPT-OSS-20B uses RL from the same family as o3 and performs deliberative alignment in the CoT. The prompt pool for Section 3 is at most 2,446 prompts, each with at most 5 CoTs and 5 resampled responses each, scored by local GPT-OSS-20B according to StrongREJECT; the judge does not see the chain of thought.
   • Whether the chain participates in the decision: The authors report that after fixing a CoT, the standard deviation of compliance scores for all four models is lower than when conditioned only on the prompt (Table 2). GPT-OSS-20B goes from 0.045 to 0.003; the authors state that once the trajectory is fixed, the outcome is nearly determined by the trajectory; DeepSeek-R1-Distill-Qwen-7B goes from 0.230 to 0.118. The authors also state that low variance does not imply the decision occurs before the chain.
   • Which sentence it lands on: On the high-variance subset, the authors state that DeepSeek-R1-Distill-Llama-8B's variance decreases in the early part of the chain, while GPT-OSS-20B is slower (Figure 3; endpoint values are not given in the text). In Table 1, for different first sentences of the same prompt, the source model's mean at k=15 can be 0.00±0.00 or 0.98±0.06; after transferring to Qwen-7B with the same teacher, the authors state the direction is broadly similar, but one row is 0.18±0.38 versus 0.58±0.50.
   • Linear direction: Direction ablation increases harmful compliance. On the held-out set of 487 items, GPT-OSS-20B's refusal rate measured by mean score < 0.4 goes from 93.4% to 3.5% (CoT direction, Table 9). On the 24 common refusal items, Llama-8B's CoT ablation mean is 0.888±0.281 (Table 7). The authors state this is less stable than the same technique on non-reasoning models; benign reasoning benchmarks typically drop 5-30% relatively, with two unnamed exceptions.
   • Authors' conclusion: Chain of thought causally affects refusal; on a subset of distilled models, the decision can collapse to the first sentence and transfer between students of the same teacher; reasoning models' refusal is not separable by a single linear direction as in non-reasoning models. The authors propose a possibility: distillation imitates trajectories rather than discovering policies under reward, so safety conclusions relying on open-source distilled models may not transfer to models trained under different settings. This possibility was not directly tested, and the 8B vs. 20B scale difference prevents attributing the training paradigm alone.

   Collapse this answer

Related papers