Rule Based Rewards: A Method for Large Language Model Safety

The researchers propose Rule Based Rewards (RBR), a preference modeling method that uses AI feedback and requires only a small amount of human data, for improving large language model safety training. The method uses a set of rules (e.g., refusals should not carry a judgmental tone) together with an LLM grader, using fine-grained, composable few-shot scores directly as RL training rewards. RBR achieves an F1 of 97.1, higher than the human feedback baseline of 91.7, better balancing helpfulness and safety.