Achieving safe alignment of language models through non-cooperative games

The researchers propose AdvGame, which models safety alignment as a non-zero-sum game between attacker and defender language models, jointly training both sides via online reinforcement learning so that each continuously adapts to the other's evolving strategies. The authors state that this method makes the defense model more helpful while more robust to adversarial attacks, and pushes outward the Pareto frontier of safety and usability. The method uses a preference reward signal based on pairwise comparisons rather than pointwise scoring, which the authors believe provides more robust supervision and may reduce reward hacking. The trained attacker model converges into a general red-teaming agent that can probe any target model, and the code is open-sourced at github.com/facebookresearch/advgame.