Constrained Language Model Policy Optimization Based on Risk-Aware Stepwise Alignment

The researchers propose risk-aware stepwise alignment (RSA), which explicitly introduces risk awareness into language model safety alignment, addressing the shortcomings of risk-neutral methods such as Safe RLHF and SACPO. The method formulates safety alignment as token-level risk-aware constrained policy optimization, updates the policy stepwise based on a nested risk measure, mitigates the risk of the model deviating excessively from the reference policy, and suppresses low-probability high-impact harmful behaviors. Experiments show that RSA ensures strong safety while maintaining high helpfulness, significantly suppressing tail risk.