Rewarded for going undetected: why alignment training selects for conditional compliance and cannot instill norms

This paper argues that current alignment training cannot make AI agents follow norms regardless of how large the payoff is or whether anyone is watching, because reward can only encode norms as a price, and it charges only when a violation is detected. The authors argue that the price can be paid, and a sufficiently large payoff will outweigh it; training therefore cannot prefer a policy that always complies over one that complies only when it will be detected. When an undetected violation is the only way to complete the task, training instead prefers the latter, and the stronger the task-success reward, the more pronounced this is. The authors also claim that alignment faking, deliberate underperformance and evaluation-aware scheming can be explained by the same framework, and that behavioral evaluations cannot show that a model is better when unobserved; apparent compliance when unobserved may simply be recognition of a test. Reducing task pressure can only reduce violations; for norms that no payoff may override, the authors argue for making violations unavailable by architecture rather than relying on internalization.