Goal misgeneralization: why correct specifications are not sufficient for correct goals

The paper proposes the concept of "goal misgeneralization," referring to the possibility that an AI system may pursue unintended goals even when the specification given by the designers is not flawed. The authors define it as a robustness failure of learning algorithms: the learned program performs well in training situations but performs poorly in entirely new test situations by pursuing unintended goals, which differs from specification gaming where the specification itself is flawed. The authors give examples of goal misgeneralization in deep learning systems across multiple domains, extrapolate to more capable systems, use hypothetical scenarios to illustrate that it could bring catastrophic risk, and finally propose several research directions that might reduce this risk.