SALMON: Self-Alignment Based on an Instructable Reward Model

The researchers propose the SALMON self-alignment method, which trains an instructable reward model on synthetic preference data so that it can generate reward scores according to arbitrary human-defined principles, thereby aligning a base language model with minimal human supervision. In the RL training stage, only these principles need to be adjusted to control the reward model and thereby influence the policy model's behavior, reducing reliance on online human preference annotation. The authors applied this method to the LLaMA-2-70b base model and developed the AI assistant Dromedary-2, using only 6 in-context learning examples and 31 human-defined principles, and claim that it significantly surpasses advanced AI systems such as LLaMA-2-Chat-70b on multiple benchmark datasets. The code and model weights have been open-sourced.