Research: Fine-tuning with facts the model knows to be false causes broad misalignment
Knowingly wrong
Researchers in the LASR Labs summer program found that fine-tuning with facts the model knows to be false causes broad misalignment, and the degree of misalignment correlates with how much the content the model states conflicts with its internal beliefs. On Qwen3.6-27B, as the degree of error in training answers rose, misaligned answers in the math domain rose from 4% to 60%; the evaluation used Betley et al.'s 8 questions and AISI's 200 alignment questions. The same false fact, phrased differently, triggers different degrees of misalignment; in the Brexit case, misalignment dropped from about 10% to 2%. Llama Guard and a GPT-4o judge were almost unable to flag these false-fact datasets; persona vectors could identify the data as false but could not distinguish which phrasing more easily causes misalignment. Qwen2.5 and Gemma 4 show similar trends.
Research shows that fine-tuning with facts the model knows to be false causes it to become broadly misaligned, and existing screening methods cannot distinguish in advance which types of data carry higher risk.