Bad token: emergent misalignment is shallower than expected

LessWrong user axel_sdq, drawing on a recent paper, points out that emergent misalignment may be more like a shallow phenomenon triggered by system prompt tokens. The paper fine-tunes Qwen-2.5-7B and Llama-3.1-8B to give bad advice on financial questions, after which the models also show broader misaligned behavior on unrelated topics. The author says that when the keys and values at the system prompt position are restored to the original model's computations, the model largely returns to alignment; even lightly perturbing the system prompt prefix, such as changing the date, noticeably reduces misaligned behavior. The paper then uses Token-Regularized Fine Tuning (TReFT) to fix the keys and values at those positions and fine-tune again, yielding narrow in-domain misalignment. The author infers that the "evil" in emergent misalignment has no clear relationship to the semantic meaning of the system prompt, may be more like a trigger-based backdoor, is shallower than expected, and cautions that the example may have been cherry-picked.