Skip to content

SAVeR²: Safety Alignment of Large Reasoning Models via Reasoning and Verifiable Rewards

SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2611.07482· NeurIPS 2026· Yichen Sun, Qirui Liu, Yan Wang, Jianan Lin, Linbo Jiang, Shiyu Wang, Zhibo Wang, Zhixuan ChuThe State Key Laboratory of Blockchain and Data Security, Zhejiang University; Alibaba Group; Chongqing Ant Consumer Finance Co., Ltd.Indexed by this site · 11 days ago· Originally publishedAlignment

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summarySAVeR² is a safety alignment method for large reasoning models (LRMs) that decomposes safety objectives into four types of fine-grained verifiable reward signals and uses difficulty-aware data selection to stabilize training. It introduces a gradient projection mechanism that preserves reasoning capability, resolving conflicts between safety and reasoning gradients and effectively eliminating the "safety tax." Experiments on models from 1.5B to 671B show that the method achieves better safety performance and lower over-refusal rates on multiple benchmarks without harming general reasoning capability.

Questions
The explicit long CoT of large reasoning models (LRMs) brings new safety risks: even if the final answer refuses, the intermediate reasoning may leak harmful details. The authors argue that existing LRM safety alignment methods have three problems: poor adversarial generalization, a "safety tax" that degrades reasoning ability, and single coarse-grained rewards that are not realistic.
Methods
Proposes SAVeR2: first an SFT warm-up teaches the model structured safety reasoning, then training data of medium difficulty is selected by difficulty, and finally GRPO jointly trains four fine-grained verifiable safety rewards and a general reasoning reward, while gradient projection analytically eliminates the conflict between safety gradients and reasoning gradients; a benign data reward is added to mitigate over-refusal.
Experiments and results
Evaluated on models from 1.5B to 671B: on DS-14B, safety average 94.28 and BSA F-score 81.24, both higher than the listed baselines (Table 1); general capability is largely maintained, DS-14B H-mean 82.82 (Table 2); over-refusal FRR average 22.33 (Table 5); average DSR 97.33% under AdvBench jailbreaks (Table 9).
Limitations and future work
In the Limitations section, the authors note that gradient projection introduces additional computational overhead compared with standard GRPO, and that more efficient projection matrix approximation methods could be explored in the future. In terms of experimental coverage, the evaluated models are the DeepSeek-R1 distillation series (1.5B/7B/14B), R1-671B, and Qwen3-4B-Thinking, and the judging relies mainly on GPT-4o and Llama-Guard.

Related events and news

1 events · 1 news mentions

News that mentions this paper

模安局盘点NeurIPS 2026 AI安全七大研究重点

Related papers

Show 3 more papers
Read originalNeurIPS 2026