Automatically Discovering Reward Model Biases

The paper proposes and studies the problem of automatically discovering reward model biases, using an LLM to iteratively propose and refine candidate biases. The method can reproduce known biases and also discovers new biases, for example, the open-weight reward model Skywork-V2-8B often prefers responses containing redundant spaces and hallucinated content. The paper states that evolutionary iteration outperforms flat best-of-N search, and validates the pipeline's recall through synthetically injected biases. The authors hope this work advances the use of automated interpretability methods to improve reward models.