SAVeR²: Safety Alignment of Large Reasoning Models via Reasoning and Verifiable Rewards
SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Translation in progress — untranslated text is omitted.
SAVeR² is a safety alignment method for large reasoning models (LRMs) that decomposes safety objectives into four types of fine-grained verifiable reward signals and uses difficulty-aware data selection to stabilize training. It introduces a gradient projection mechanism that preserves reasoning capability, resolving conflicts between safety and reasoning gradients and effectively eliminating the "safety tax." Experiments on models from 1.5B to 671B show that the method achieves better safety performance and lower over-refusal rates on multiple benchmarks without harming general reasoning capability.
- The explicit long CoT of large reasoning models (LRMs) brings new safety risks: even if the final answer refuses, the intermediate reasoning may leak harmful details. The authors argue that existing LRM safety alignment methods have three problems: poor adversarial generalization, a "safety tax" that degrades reasoning ability, and single coarse-grained rewards that are not realistic.
- Proposes SAVeR2: first an SFT warm-up teaches the model structured safety reasoning, then training data of medium difficulty is selected by difficulty, and finally GRPO jointly trains four fine-grained verifiable safety rewards and a general reasoning reward, while gradient projection analytically eliminates the conflict between safety gradients and reasoning gradients; a benign data reward is added to mitigate over-refusal.
- Evaluated on models from 1.5B to 671B: on DS-14B, safety average 94.28 and BSA F-score 81.24, both higher than the listed baselines (Table 1); general capability is largely maintained, DS-14B H-mean 82.82 (Table 2); over-refusal FRR average 22.33 (Table 5); average DSR 97.33% under AdvBench jailbreaks (Table 9).
- In the Limitations section, the authors note that gradient projection introduces additional computational overhead compared with standard GRPO, and that more efficient projection matrix approximation methods could be explored in the future. In terms of experimental coverage, the evaluated models are the DeepSeek-R1 distillation series (1.5B/7B/14B), R1-671B, and Qwen3-4B-Thinking, and the judging relies mainly on GPT-4o and Llama-Guard.
Related events and news
1 events · 1 news mentions
News that mentions this paper
模安局盘点NeurIPS 2026 AI安全七大研究重点
- 公众号 模安局gives the paper's titleNeurIPS 2026 AI安全的七个研究重点
SAVeR²: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Related papers
- Adversarial Self-DistillationDefensesarXiv