Unifying Stable Optimization and Reference Regularization in RLHF

RLHF has long been troubled by two problems, reward hacking and stable optimization, and the paper proposes a unified regularization method that balances both simultaneously. The method unifies the KL penalty toward the supervised fine-tuning model π0 and the ratio clipping toward the current policy πt into a weighted supervised fine-tuning loss, which the authors claim balances alignment performance and implementation complexity. Experiments on multiple benchmarks show that its alignment performance and stability are better than RLHF and online preference learning methods.