Generalizing from easy to hard: scalable alignment beyond human supervision

Researchers propose a scalable alignment method that generalizes from easy to hard: a process supervision reward model is trained on human annotations of simple problems at MATH levels 1-3, and then used to evaluate the policy model's solutions to hard problems at levels 4-5. This evaluator can enable generative models to likewise generalize from easy to hard through reranking or reinforcement learning. Using only human supervision on simple problems, the process-supervised 7B reinforcement learning model and the 34B model (reranking@1024) achieve 34.0% and 52.5% accuracy on MATH500, respectively. The authors describe the method as pointing to a promising path for building AI systems beyond the frontier of human supervision.