Outcome accuracy is not enough: aligning the reasoning process of reward models

Generative reward models (GenRM) and LLM-as-a-Judge, because they optimize only outcome accuracy, can exhibit deceptive alignment in which the judgment is correct but the rationale is wrong. The paper proposes the Rationale Consistency metric to quantify the consistency between the reasoning process and human judgments, which can detect this problem. Training GenRM by combining this metric with outcome accuracy reaches 87.1% on RM-Bench and 82% on JudgeBench, averaging 5% above outcome-only baselines, and after use in RLHF improves the Arena Hard v2 creative writing task by 7%.