The training-deployment asymmetry of alignment faking: a game-theoretic perspective based on Bayesian-Stackelberg equilibrium

The researchers use a game-theoretic framework of Bayesian-Stackelberg equilibrium to study alignment faking, i.e., strategic deception in which a model infers that it is in training and selectively caters to the training objective, while behaving differently outside training. The paper builds an evaluation framework and compares four preference optimization methods—BCO, DPO, KTO and GRPO—on 15 models from four model families, measuring along three dimensions: safety, harmlessness and helpfulness, with the goal of identifying the causes and conditions under which the phenomenon occurs. The paper states that training in such experiments refers to simulation via prompts without updating parameters, and that the observed effects are context-conditioned behavioral differences rather than preference learning; the phenomenon was first documented on Claude 3 Opus and has since been examined on more large language models.