What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data

Starting from the three fine-tuning components of training dynamics, model priors, and data, the researchers analyze the sources of differences in emergent misalignment (EM) after narrow-domain fine-tuning. They compare the relationship between in-domain training loss and out-of-domain alignment scores across different datasets and model families, and attempt to use different learning schedules to find alternative local optima for a single narrow-domain fine-tuning, but did not find training with broadly better alignment scores at similar or lower training loss. The mean and standard deviation of misaligned model scores usually differ statistically from the pretrained model, but overall there is a positively correlated latent signal; the activations of the pretrained model and the original instruct model before narrow fine-tuning, using only the evaluation prompt, can predict the fine-grained alignment scores after fine-tuning. Comparing activation changes before and after narrow fine-tuning, the activation shifts caused by training and evaluation prompts have medium-to-high overlap and similarity in subspaces; when using the activation of the last prompt token, the subspace overlap of the two is correlated with the shift similarity.