InfAlign: Inference-aware language model alignment

The researchers propose the InfAlign framework, which makes language model alignment directly optimize the win rate of the aligned model relative to the base model at inference time. The research proves that for any inference-time decoding procedure, the optimal alignment policy is equivalent to a standard RLHF problem after a transformation of the reward, and on this basis proposes the InfAlign-CTRL algorithm, which includes reward calibration and KL-regularized reward maximization. For best-of-N sampling and best-of-N jailbreak, the authors give specific reward transformations that bring up to 3-8% inference-time win rate improvements; the authors also claim that their reward calibration method can serve as a strong baseline for optimizing standard win rate.