One-Shot Safety Alignment of Large Language Models via Optimal Dualization
One-Shot Safety Alignment for Large Language Models via Optimal Dualization
AI summary
The researchers propose a method for safety alignment of large language models via optimal dualization, transforming safety-constrained RLHF into an equivalent unconstrained alignment problem. The method pre-optimizes a smooth convex dual function with a closed form, eliminating cumbersome primal-dual policy iteration, thereby greatly reducing computational burden and improving training stability. The paper provides practical algorithms MoCAN and PeCAN under model-based and preference-based settings respectively, and experiments over a broad range validate the effectiveness of both algorithms.