Does large language model alignment really need diversity? An empirical study adapting RLVR methods to moral reasoning

The authors say this is the first empirical study to comprehensively compare the two paradigms of distribution matching and reward maximization on MoReBench, and the results show that distribution matching methods did not, as hypothesized, significantly outperform reward maximization methods on alignment tasks. The study's hypothesis was that moral reasoning tolerates multiple valid answers, and alignment tasks may inherently require diversity-seeking distribution matching algorithms; to support stable verifiable-reward reinforcement learning (RLVR) training, the authors trained a Qwen3-1.7B judge model and built a rubric-based reward pipeline. Semantic visualization shows that the high-reward distribution for moral reasoning is more concentrated than for mathematical reasoning, whereas in mathematical reasoning diverse solution strategies can all obtain similarly high rewards; the authors say this counterintuitive finding explains why mode-seeking optimization is equally or even more effective on alignment tasks. The authors therefore conclude that alignment tasks do not inherently require diversity-preserving algorithms, and that standard reward-maximization RLVR methods can effectively transfer to moral reasoning without explicit diversity mechanisms.