Improving RLHF with efficient reward model ensembles
Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
AI summary
Researchers propose an efficient reward model ensemble method that makes RLHF reward model predictions more accurate and mitigates the problem of outputs deviating from human values due to limited human preference data. To avoid the high computational and resource costs of ensembling multiple large language model reward models, the method includes two schemes: linear layer ensembling and LoRA ensembling. The authors verified in Best-of-n and PPO experiments that ensemble reward models can improve the alignment performance of RLHF outputs.