Refusal without refusal: an analysis of safety fine-tuning response structure for reducing language model false refusals

Boilerplate refusal statements in safety fine-tuning responses cause models to also refuse superficially similar harmless requests.

• Where it occurs: Deployed models must both refuse harmful requests and answer harmless requests that merely sound harmful. The authors argue that false refusals reduce utility and user trust.
• Limitations of existing approaches: The authors list prompting, decoding, activation steering, and decision boundary interventions, and argue that they often incur computational overhead at inference time or rely on post-hoc modifications. This paper instead examines the training data that may cause this behavior.
• Core hypothesis: Safety data often first gives a boilerplate refusal and then an explanation. The authors argue that training binds the refusal statement to surface cues in harmful prompts; when these cues appear in harmless prompts, they may trigger false refusals.
• What this paper does: It decomposes safety responses into a refusal statement and an explanatory rationale, proposes Rationale-Only supervision that keeps only the rationale, and compares the position of the statement, whether the statement is kept, and whether the rationale names the specific request.

Safety alignment and post-hoc intervention

• Training-time alignment: Ouyang et al. (2022), Bai et al. (2022a,b), Rafailov et al. (2023), Dai et al. (2024), Zhang et al. (2025a) use supervised fine-tuning or reinforcement learning from human feedback to make models refuse harmful requests.
• Inference-time intervention: Bhardwaj et al. (2024), Xu et al. (2024a,b), Hazra et al. (2024), Banerjee et al. (2025) make post-hoc modifications to decoding, parameters, or internal representations.
• Reported side effects: Bianchi et al. (2024) report that safety fine-tuned models may refuse harmless requests and give generic, context-insensitive refusals. An et al. (2024) use a pseudo-harmful benchmark to probe alignment failures on superficially similar inputs. This paper uses the Safety-Tuned LLaMAs of Bianchi et al. (2024).

False refusals

• Phenomenon: Röttger et al. (2024), Shi et al. (2024) discuss erroneous refusals caused by overly cautious alignment. Cui et al. (2024), An et al. (2024) argue that such errors come from surface linguistic cues rather than deeper contextual understanding.
• Data-side work: Zhang et al. (2025b), Pan et al. (2025) directly target over-refusal with structured reasoning traces or safety boundary data.
• Inference-time mitigation: Zhao et al. (2024), Shi et al. (2024), Wang et al. (2025), Cao et al. (2025) mitigate false refusals through internal activations, prompt adjustments, or other inference-time interventions. The authors argue that these methods mostly treat symptoms rather than root causes.

Templates and shortcut learning

• Surface patterns: Geirhos et al. (2020) discuss how templated formats make models rely on surface patterns. Gururangan et al. (2018) show shallow cues such as word overlap and negation in natural language inference. Jia and Liang (2017) discuss fixed answer formats and position bias in question answering.
• The authors argue that the role of standardized refusal templates in safety data for false refusals remains understudied. The paper only describes these works and does not conduct per-paper controlled experiments.

Baselines and benchmarks

• Control conditions: Statement-Only, Statement and Rationale, and statement placed at Beginning, Middle, End; rationales are divided into Generic and Request-Specific. At inference time, SelfCD (Shi et al., 2024) and SCANS (Cao et al., 2025) are additionally applied.
• Benchmarks: AdvBench, MaliciousInstruct for harmful queries; XSTest-Safe, OKTest for pseudo-harmful; MMLU, OpenBookQA, HellaSwag, ARC, GSM8K, PIQA for capability. HarmBench, JailbreakBench, WILDJAILBREAK, SORRY-Bench for jailbreak-style inputs.

The authors place this paper on the training data side: separating refusal statements and rationales, rather than only modifying outputs at inference time.

The overall approach is data-side decomposition: split safety fine-tuning responses into formulaic refusal statements and explanatory rationales, then use Rationale-Only to remove the statement and supervise only the rationale. The authors fix the request and rationale content, changing only the target response structure.

Problem setup and signal decomposition

• Safety fine-tuning learns a mapping from prompts to target responses, where the response encodes whether and how to respond.
• The authors write the prompt as a conceptual decomposition ϕ(x)=ϕ_sem(x)+ϕ_surf(x): the former is semantic intent, the latter is surface lexical patterns. The authors note this is not explicit neural disentanglement, only used to discuss signal influence.
• The authors' hypothesis is that formulaic statements become repeatedly bound to surface cues in harmful prompts during training; the surface term strengthens safety when aligned with harmful intent, and may cause false refusals when it appears in harmless prompts.

Response decomposition and experimental conditions

• Refusal statement: a short, formulaic expression of noncompliance. Rationale: points out illegality, physical harm, or ethical risks relevant to the request. Figure 2 is a schematic of this decomposition.
• Base format: a short refusal is moved to the beginning, followed by at least two consecutive rationale sentences. The authors state that normalization preserves the user request and substantive content. Table 10: 1,922 of the full 2,163 entries, and 222 of the 256 sampled in the same way as the training set, have the statement before the rationale. The authors state that the setup reflects the original distribution rather than creating a new structure.
• Position: Beginning is the base format; Middle places the statement within the rationale, not as the first or last sentence; End places the statement as the last sentence. The remaining text is kept as unchanged as possible.
• Components and specificity: Statement-Only keeps only the refusal phrase and removes the explanation; Rationale-Only keeps only the explanation and removes the refusal marker; Statement and Rationale equals the base format. Generic replaces direct mentions of the requested behavior with general wording; Request-Specific names the behavior and states which aspects are considered harmful. The authors state that the changes are kept minimal to maintain semantic alignment with the original text. Variants are generated by Llama-3.3-70B-Instruct. Appendix A has data examples, and Appendix L gives the full prompts; the original harmful request text is not reproduced here.

Training configuration

• Four base models, not instruction-tuned variants: Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B, Qwen2.5-7B. The authors state this is to reduce confounding from prior instruction tuning and to directly observe the impact of safety data.
• Training mixes two types of data: 1,024 instruction–response pairs randomly sampled from Alpaca clean; 256 samples from Safety-Tuned LLaMAs after filtering out prompts unrelated to safety.
• All conditions share QLoRA: rank 64, alpha 16, dropout 0.1, applied to all linear layers, 4-bit NormalFloat; paged AdamW (32-bit), constant learning rate 1e-4, batch size 64, 10 epochs, max length 2,048. Inference uses vLLM greedy decoding. Appendix B gives these configurations.

Evaluation and success criteria

• Harmful compliance: AdvBench, MaliciousInstruct, lower CR is better. False refusal: XSTest-Safe, OKTest, these two sets contain queries that look harmful but are semantically harmless, higher CR is better.
• Compliance is the positive class. True positives are compliance on pseudo-harmful queries, true negatives are refusals on harmful prompts. Overall Recall, Precision, and F1 are computed by micro-averaging.
• The primary judge is Llama-3.3-70B-Instruct, and it is required to provide classification reasons. The safety judge achieves 98.33% accuracy on 120 cases (60 compliant, 60 refused). The false-refusal judge separately flags pseudo-compliance: superficially compliant but partially refusing, changing the subject, or overemphasizing ethical reservations. Appendix L gives the judge prompts.

Experimental setup

• Models evaluated: The main experiments use the base versions of Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B, and Qwen2.5-7B. Appendix H additionally fine-tunes Gemma-2-2B, Qwen2.5-3B, Llama-3.1-70B, and Qwen2.5-72B. The main tables in the main text expand only the first two; the component results for the latter two are in Appendix Table 19.
• Data and baselines: 256 safety supervision items plus 1,024 Alpaca clean items. Comparisons are Statement-Only, Rationale-Only, Statement and Rationale, plus position and rationale-specificity variants. The paper does not report the number of samples in the main evaluation sets such as AdvBench, nor the number of repetitions.
• Metrics: Compliance Rate (CR) on harmful and pseudo-harmful sets; Recall, Precision, and F1 with compliance as the positive class. Capability tasks use exact match; all are 0-shot except GSM8K, which is 8-shot chain-of-thought. Jailbreak-style benchmarks report harmful compliance rate, lower is better.
• Reviewer: The main reviewer is Llama-3.3-70B-Instruct. Agreement between the false-refusal criteria and humans was evaluated using 550 items from Llama-3.1-8B Rationale-Only on XSTest-Safe and OKTest: 5 graduate students each labeled 110 items with no overlap, Cohen's Kappa 0.81 (Table 14). On the same batch, this reviewer's CR was 0.71 and 0.60, while humans' was 0.74 and 0.58 (Table 15). Kappa between the safety criteria and WildGuard was 0.48 and 0.60.
• Decoding: greedy. Appendix C additionally uses GPT-5.1 to re-evaluate the four base models under the same protocol.

Main results

The table below compares components. Numbers come from Table 19; the corresponding rows for Llama-3.1-8B and Mistral-7B-v0.3 are the same as in Table 2. RO is Rationale-Only, SR is Statement and Rationale. Higher pseudo-harmful CR is better; lower harmful CR is better.

Swipe horizontally to read the table

Model	RO F1	SR F1	RO pseudo-harmful CR (XS/OK)	SR pseudo-harmful CR (XS/OK)	RO harmful CR (Adv/Mal)	SR harmful CR (Adv/Mal)
Llama-3.1-8B	0.77	0.56	0.71 / 0.60	0.39 / 0.41	0.02 / 0.06	0.02 / 0.02
Mistral-7B-v0.3	0.84	0.68	0.75 / 0.77	0.49 / 0.55	0.04 / 0.06	0.01 / 0.00
Gemma-2-9B	0.81	0.64	0.68 / 0.69	0.44 / 0.51	0.00 / 0.03	0.01 / 0.03
Qwen2.5-7B	0.81	0.63	0.65 / 0.76	0.37 / 0.55	0.03 / 0.03	0.02 / 0.02

• The authors state that adding the refusal statement increases false refusals, whether it appears alone or with the rationale. In Table 19, Statement-Only F1 is 0.62, 0.69, 0.69, 0.59, and pseudo-harmful CR is lower than Rationale-Only for the same model.
• The authors state that Rationale-Only reduces false refusals with largely comparable safety, and state that the safety compromise is small (Figure 1 gives qualitative examples). In Table 19, harmful CR is not lower in every cell: Llama-3.1-8B's MaliciousInstruct is 0.06 vs. 0.02, and Mistral-7B-v0.3 is 0.06 vs. 0.00. Mistral's Precision is SR 0.98, RO 0.94; the higher F1 corresponds to higher Recall.
• The caption of Figure 1 states: Statement and Rationale refuses both harmful and pseudo-harmful prompts, while Rationale-Only refuses harmful inputs but complies with harmless inputs. The figure contains specific prompts and responses, not repeated here.

Statement position and rationale specificity

• Position: The authors state that placing the statement at the beginning causes more false refusals than placing it in the middle or at the end. In Table 2, Llama-3.1-8B Beginning / Middle / End: XSTest-Safe CR 0.39 / 0.58 / 0.60, OKTest 0.41 / 0.52 / 0.54, F1 0.56 / 0.69 / 0.71; harmful CR ranges 0.02–0.04. In Table 18, Gemma-2-9B's OKTest CR is 0.51 for both Beginning and Middle, and 0.55 for End.
• Specificity: In Table 20, Request-Specific pseudo-harmful CR is higher than Generic. Llama-3.1-8B's XSTest-Safe / OKTest is 0.80 / 0.72 vs. 0.66 / 0.60, F1 0.84 vs. 0.75; Mistral-7B-v0.3's F1 is 0.88 vs. 0.84. The authors state that naming the behavior and its risk provides more information than a generic explanation.
• In the same table, Request-Specific also has higher harmful CR cells: Llama-3.1-8B's AdvBench / MaliciousInstruct is 0.04 / 0.05 vs. Generic 0.03 / 0.02; Mistral-7B-v0.3 is 0.06 / 0.08 vs. 0.04 / 0.03.

Capability, jailbreak inputs, and internal analysis

• Capability: Table 3 averages over six tasks: Llama-3.1-8B is Statement-Only 53.89, Rationale-Only 53.25, Statement and Rationale 54.08; Mistral-7B-v0.3 is 46.23, 47.70, 47.06. The authors state that no condition is consistently better or worse across tasks. Appendix Table 25 gives the other two models.
• Jailbreak-style harmful compliance: Table 4 averages: Llama-3.1-8B is 0.25 / 0.20 / 0.21, Mistral-7B-v0.3 is 0.27 / 0.15 / 0.16 (Statement-Only / Rationale-Only / Statement and Rationale). The authors state that Rationale-Only is comparable to training with the statement, with no consistent safety loss. In Table 26, Gemma-2-9B's HarmBench is Rationale-Only 0.19, Statement and Rationale 0.18.
• Entropy and attribution: The authors state that in Figure 3a, training with the statement yields lower first-token entropy on both harmful and pseudo-harmful prompts. In Table 27, Llama-3.1-8B means: AdvBench 0.04 vs. 1.11, XSTest-Safe 0.25 vs. 0.84, AlpacaEval 0.62 vs. 0.76 (Statement and Rationale vs. Rationale-Only). In the same table, Mistral-7B-v0.3 and Gemma-2-9B's 95% intervals on AlpacaEval cross 0. Table 5: among 79 XSTest-Safe items, Rationale-Only has Meaningful 78 (98.7%), training with the statement has 7 (8.8%) and Risky 48 (60.8%); among 74 OKTest items, the corresponding numbers are 72 (97.3%) vs. 10 (13.5%) and 40 (54.1%). The authors state that training with the statement relies on less content-related cues in about 90% of cases.

Structure replacement, in-context learning, and inference-time methods

• Structure: Table 6 (Llama-3.1-8B): after adding a neutral prefix before all rationales, XSTest-Safe / OKTest CR is 0.44 / 0.46, F1 0.61, still lower than Rationale-Only's 0.71 / 0.60 and 0.77. The 15 style variants give 0.55 / 0.55, F1 0.69, higher than the fixed statement but still lower than Rationale-Only. The authors state that the template regularity itself affects false refusals.
• Keywords: Table 7 makes one version each for drug, kill, money, steal, and shoot. The authors state that about 51 training responses containing the word had the statement added, and each word was evaluated with 50 pseudo-harmful queries. Llama-3.1-8B's Rationale-Only average CR is 0.75; Money-Statement on Money Queries is 0.36 vs. 0.60; Shoot-Statement on Shoot Queries is 0.72 vs. 0.88. The authors state that the statement causes lower compliance on the corresponding keyword.
• Applicability: In Table 8's in-context learning, Llama-3.1-8B Rationale-Only F1 is 0.87, XSTest-Safe / OKTest 0.81 / 0.84; Statement and Rationale is 0.73 and 0.52 / 0.65. Harmful CR is 0.06 / 0.06 vs. 0.04 / 0.03. The paper states that Qwen2.5-7B frequently generates EOS under URIAL, so its ICL is not reported. Table 9 applies SelfCD (authors' own implementation, α=1.5) and SCANS (steering multiplier 1) only to Llama-3.1-8B: Rationale-Only F1 is 0.83 and 0.82, Statement and Rationale is 0.66 and 0.68; Request-Specific F1 is 0.86 and 0.88, the bolded highest F1 among methods. Under SelfCD, this variant's harmful CR is 0.08 / 0.09. Surgical does not apply: no candidate vector has KL divergence below 0.2.

Other ablations and analyses

• Length: Average character counts: Statement-Only 88.89, Rationale-Only 361.21, Statement and Rationale 457.16, 1-Sentence Rationale-Only 97.65. In Table 37, Llama-3.1-8B's 1-Sentence F1 is 0.80, XSTest-Safe / OKTest 0.74 / 0.65. The authors state that the longest condition with the statement is closer to Statement-Only than to Rationale-Only.
• Reasoning prompt: Adding "Let's think step by step." at generation time without changing training. In Table 38, Llama-3.1-8B F1 is Statement-Only 0.63, Rationale-Only 0.80, Statement and Rationale 0.59. The authors state that this prompt does not close the gap.
• Safety data scale: Appendix K increases to 512, 1024, and 2048 items. In Table 40, Llama-3.1-8B at 2048 items has F1 of Rationale-Only 0.78, Statement and Rationale 0.51, Statement-Only 0.31. The authors state that the relative ordering is unchanged.
• Model scale: In Table 36, Llama-3.1-70B F1 is Rationale-Only 0.82, Statement and Rationale 0.67; Qwen2.5-72B is 0.85 and 0.65. The authors state the trend is the same.
• Reviewer change: Table 17 uses GPT-5.1; Llama-3.1-8B Rationale-Only F1 is 0.80, Statement and Rationale 0.56. The authors state that the harmful compliance rates of the two differ by at most 0.03, and the ordering is inconsistent across model families and harmful benchmarks; spot checks found no separate failure type for Rationale-Only.
• Official instruction models: In Table 39, Mistral-7B-Instruct-v0.3's AdvBench / MaliciousInstruct CR is 0.69 / 0.78. The authors state that refusal behavior varies greatly across instruction models and that alignment data and pipelines are not public, so the main analysis uses base models.

Limitations and future directions noted by the authors

• Fine-tuning scale: The Limitations section states that although a range of different models was evaluated, the fine-tuning scale was constrained by limited resources.
• Parameter scale: The same section states that experiments do not include models larger than 80B parameters.
• Future directions: The same section suggests future research on the impact of curated safety training data at larger scales and on data selection strategies that strengthen both safety and utility.
• Comparison with official instruction models: Appendix J states that direct comparison is limited because the alignment data and training pipelines of these models are not public.

Experimental coverage

• The main experiments in the main text fine-tune and evaluate Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B, and Qwen2.5-7B; Appendix H and Table 36 additionally cover Gemma-2-2B, Qwen2.5-3B, Llama-3.1-70B, and Qwen2.5-72B.
• The safety supervision in the main setting is the 256 items from Safety-Tuned LLaMAs, mixed with 1,024 items from Alpaca clean; Appendix K and Table 40 additionally report 512, 1024, and 2048 safety data items.
• Main metrics cover AdvBench, MaliciousInstruct, XSTest-Safe, and OKTest; capability is measured on six standard benchmarks, and jailbreak-style inputs are HarmBench, JailbreakBench, WILDJAILBREAK, and SORRY-Bench. The paper does not report the number of samples in these main evaluation sets, nor the number of repetitions.
• Agreement between the false-refusal reviewer and humans is reported on 550 items from Llama-3.1-8B Rationale-Only, with Cohen's Kappa of 0.81; GPT-5.1 was also used to re-evaluate the four base models (Table 17).
• SelfCD and SCANS results are reported only for Llama-3.1-8B (Section 7, Table 9). ICL is reported for Llama-3.1-8B, Mistral-7B-v0.3, and Gemma-2-9B; the paper states that Qwen2.5-7B frequently generates EOS and is therefore not included in that setting.

This is a structural analysis of safety fine-tuning data: false refusals are traced by the authors to formulaic refusal statements in responses, rather than relying only on inference-time remedies.

• Problem: models need to refuse harmful requests, yet they refuse requests that look harmful but are semantically harmless. The authors argue this reduces utility and user trust.
• Approach: decompose safety responses into refusal statements and explanatory rationales. Fix the request and rationale, compare the statement placed at the beginning, middle, or end, as well as Statement-Only, Rationale-Only, and Statement and Rationale; also compare whether the rationale names the requested behavior. The main setup uses QLoRA fine-tuning on four base models, mixing 1,024 Alpaca clean samples with 256 safety samples.
• Main results: In Table 19 (first two rows identical to Table 2), Llama-3.1-8B Rationale-Only Overall F1 is 0.77 versus 0.56 with statements; XSTest-Safe CR is 0.71 versus 0.39. Mistral-7B-v0.3 F1 is 0.84 versus 0.68. Gemma-2-9B and Qwen2.5-7B F1 are 0.81 versus 0.64 and 0.81 versus 0.63.
• Safety: harmful CR remains low, but not lower in every cell. Llama-3.1-8B on MaliciousInstruct is 0.06 versus 0.02. The authors state that on jailbreak-style benchmarks, Rationale-Only is comparable to training with statements (Table 4 average 0.20 versus 0.21 for Llama-3.1-8B).
• Mechanism and extrapolation: the authors argue that formulaic statements make the model rely more on surface cues. In Table 5, Llama-3.1-8B Rationale-Only has 98.7% Meaningful among 79 XSTest-Safe items, versus 8.8% with statement training. This gap remains visible after ICL (Table 8, Llama F1 0.87 versus 0.73) and after SelfCD and SCANS (Table 9).
• Authors' conclusion: structurally finer safety supervision is needed to serve both usefulness and safety simultaneously. Limitations state that experiments do not include models larger than 80B, and suggest future research on large-scale safety data filtering.