Large Language Model Alignment Based on Constrained Learning

The researchers propose an iterative alignment method based on Lagrangian duality to solve the constrained alignment problem, and prove that the method can find the optimal constrained policy within the LLM parameterization gap. The goal of constrained alignment is to satisfy secondary utility constraints while maximizing the primary reward; addressing the problems that iterative primal-dual methods often struggle to converge and non-iterative dual methods cannot reach optimality in the LLM parameter space, the proposed method alternates between updating the LLM policy via Lagrangian maximization and updating the dual variables via dual descent. The paper characterizes the gap between the primal value in distribution space and the dual value in LLM parameter space, and quantifies the optimality gap of the learned policy at near-optimal dual variables on the objective function and constraint functions. The authors validate the effectiveness of the method through extensive experiments on the PKU-SafeRLHF and Anthropic HH-RLHF datasets.