Fine-tuning Robust Policy Optimization (FRPO) for preventing catastrophic forgetting in large language models

Large language models are typically first trained with RLHF and then fine-tuned for downstream objectives; even small downstream updates may damage learned behaviors such as safety, and this vulnerability is called catastrophic forgetting. The authors argue that the standard RLHF objective does not guarantee robustness to future adaptation, and that preventing forgetting requires making the base policy, before fine-tuning, avoid fragile high-reward solutions whose reward drops sharply after fine-tuning. To this end, the authors propose Fine-tuning Robust Policy Optimization (FRPO), which not only optimizes the current policy but also guarantees reward stability in a max-min form within the KL-constrained policy neighborhood reachable by downstream adaptation, and they adapt GRPO into an algorithm with no additional computational cost. The authors state that experiments show the method significantly reduces safety degradation across multiple base models and both SFT and RL downstream fine-tuning settings while maintaining downstream task performance; in a mathematical RL setting it also maintains accuracy after subsequent fine-tuning.