Following the Gradient: Tracing Large Model Safety Degradation to Training Data
Follow the Gradient: Tracing Safety Degradation Back to Training Data
Quick read
Based on the paper's full text (AI-generated); refer to the original
Researchers from DatologyAI and UIUC propose SimpleLoRIF, a scalable gradient attribution method for identifying seemingly harmless samples in fine-tuning data that weaken safety alignment. By simplifying LoRIF and backpropagating only through the last four transformer blocks, the method can embed all 939,343 Tulu 3 samples in about 1 hour on 8 H100s. In emergent misalignment experiments, the gradients of unsafe code training samples align with the broad harmful behaviors that emerge after fine-tuning, reaching an AUC of 0.865 on the fine-tuned Mistral model and 0.604 on Qwen. The resulting SafeTulu removes the 20% of samples in Tulu 3 most aligned with the gradients of unsafe responses, reducing the GCG attack success rate by up to 22% on Qwen, Llama and Olmo, outperforming LLM-as-a-judge filtering and existing gradient methods while largely preserving general capabilities.
- Fine-tuning an LLM on seemingly harmless instruction data quietly weakens its safety alignment, and such data shows no harmful content on the surface, making it hard for conventional content filters and LLM safety judges to identify. The authors turn the safety preservation problem into a training data selection problem: identify and remove the samples that cause safety degradation.
- 作者提出 SimpleLoRIF，一种可扩展的梯度嵌入方法：简化 LoRIF、去掉曲率修正，并把反向传播限制在最后四个 transformer block，从而能对约百万级样本的 SFT 数据集做逐样本梯度归因。用该方法把 Tulu 3 中与不安全 SafeRLHF 响应梯度最对齐的 20% 样本移除，构建 SafeTulu。
- In the emergent misalignment setting, the gradients of insecure-code samples align with the gradients of harmful outputs that appear after fine-tuning, with an AUC of 0.604 on Qwen and as high as 0.865 on Mistral. Training SafeTulu on Qwen, Llama, and Olmo reduces the GCG attack success rate by up to 22%, outperforming LLM-as-a-judge and prior gradient filtering methods, while capabilities are largely maintained or improved; ASR on ActorBreaker is also the lowest. Analysis of the removed data finds sources of safety degradation that may be overlooked, such as tool-execution failures and narrative arithmetic.
- The authors tested only general safety degradation, not specific harms or behaviors (such as sycophancy); they used only SafeRLHF as the target set and did not test other target sets. The appendix also shows that gradient representations do not necessarily transfer across models, and that the gradient signal does not increase monotonically with model scale.
论文将梯度归因扩展到百万级后训练数据集，给出在训练前剔除安全退化样本的可复现流程，适合关注微调安全的团队参考。