Revisiting AI alignment distortion: RLHF is a decent utilitarian aligner

Researchers conduct a fine-grained analysis of RLHF distortion with reward clipping, showing that the exponential growth of distortion with the Bradley-Terry temperature parameter β is not an inherent property of the algorithm but a result of the mismatch between the preference data distribution μ and the KL reference policy π_ref distribution. The paper gives tight upper and lower bounds on distortion across multiple KL regularization strength regimes; in a representative regime of the Bradley-Terry model, the distortion is Θ̃(βB + β), where B is an upper bound on the log-density ratio between μ and π_ref; when μ = π_ref, i.e., there is no distribution mismatch, RLHF achieves optimal distortion O(β) up to a constant factor. Based on this, the authors suggest that when using RLHF to reasonably maximize average utility, preference data sampled on-policy should be used, or the model should first be fine-tuned on a data source close to μ before RLHF. This work responds to the conclusion of Gölz et al. (2025) that distortion can grow exponentially with β when user preferences are heterogeneous.