Guiding out-of-distribution generalization of large language models with concept ablation fine-tuning

AI summaryThe researchers propose concept ablation fine-tuning (CAFT), a method that uses interpretability tools to control out-of-distribution generalization after fine-tuning large language models, without modifying training data or requiring data from the target distribution. CAFT first locates directions representing undesirable concepts in the model's latent space using principal component analysis (PCA) or sparse autoencoders (SAEs), then ablates these directions with linear projections during fine-tuning, with no intervention at inference time. On emergent misalignment tasks, CAFT reduces misaligned responses by about 10 times on Qwen2.5-Coder-32B-Instruct and Mistral-Small-24B-Instruct-2501 while largely maintaining performance on the training distribution, with PCA outperforming SAE. On two multiple-choice tasks with spurious correlations, CAFT can often completely reverse the model's default out-of-distribution generalization, where SAE outperforms PCA. The authors note that the method still requires human interpretation of directions, and PCA and SAE each have applicable scenarios.