Self-Play Preference Optimization (SPPO) for language model alignment

Researchers propose a self-play preference optimization method, SPPO, which models language model alignment as a constant-sum two-player game and provably approaches the Nash equilibrium through iterative policy updates. The method addresses the limitation that standard RLHF relies on parametric models such as Bradley-Terry and struggles to capture the intransitivity of human preferences, aligning directly using preference probabilities. In experiments, using only the 60,000 prompts (without responses) from the UltraFeedback dataset and no prompt augmentation, it fine-tuned Mistral-7B-Instruct-v0.2 with the help of the 0.4B-parameter pretrained preference model PairRM, achieving a 28.53% length-controlled win rate against GPT-4-Turbo on AlpacaEval 2.0, which the authors call state of the art, and surpassing (iterative) DPO and IPO on MT-Bench, Arena-Hard and the Open LLM Leaderboard; starting from the stronger base model Llama-3-8B-Instruct, it reaches 38.77%.