RLHF May Not Reflect True Preferences

The paper points out that RLHF assumes annotated responses reflect true human preferences, but six decades of behavioral science record shows that people often give answers when they have no real opinion, construct preferences on the spot based on contextual cues, and understand the same question inconsistently; the authors argue that such problems are especially common in the value judgments that AI alignment cares about most. The authors argue that measurement validity is logically prior to preference aggregation, and arrange annotated responses on a spectrum from no attitude (no signal) to true preference (full signal), developing diagnostic methods accordingly. On two RLHF datasets, the authors find that inconsistency is systematic and directionally biased: after filtering out annotators with high inconsistency, the majority harm classification flipped on 18.6% of prompts, and average ratings shifted by more than 13 points on a 100-point scale. The authors therefore argue that much current RLHF practice treats noise as signal and biases produced by the elicitation process as human values.