Identification and intervention of safety-critical parameters in large language models

The researchers propose the Expected Safety Impact (ESI) framework to quantify the impact of different parameters on large language model safety, and use it to identify differences in the distribution of safety-critical parameters between dense models and MoE models. The study finds that many safety-critical parameters in dense LLMs are located in the middle-layer value matrices and MLP, whereas in MoE models they shift to the later-layer MLP. Based on ESI, the authors propose two intervention methods, Safety Enhancement Tuning (SET) and Safety Preserving Adaptation (SPA): SET updates only 1% of the model weights and iterates 100 times, reducing the attack success rate of unaligned LLMs by more than 50%; SPA, after instruction fine-tuning for 1,000 iterations on different tasks, limits the decline in safety capability of aligned LLMs to within 1%. Code at https://github.com/ZJU-LLM-Safety/SafeWeights-ACL。