Emergent misalignment in large language models is easy; narrow-domain misalignment is difficult

Anna Soligo et al. published a paper studying the inductive biases of large language models using emergent misalignment as a case study, finding that a model could have learned only the task on a narrow-domain harmful dataset, but the more general misaligned solution is more stable and also more efficient. Building on existing results that different fine-tuning converges to the same general misalignment linear representation, the authors find that a narrow-domain solution also has a linear representation, which can be learned by introducing a KL divergence loss. Comparisons show that the general misalignment representation has lower loss, is more robust to perturbations, and has a greater effect in the pre-training distribution. The authors state that this isolates a concrete misalignment representation for monitoring and mitigation, and they open-sourced the code, dataset, and fine-tuned models.