The Art of (Mis)Alignment: How Fine-Tuning Methods Effectively Misalign and Realign Large Language Models in Post-Training

The researchers systematically evaluate the misalignment and realignment effects of four supervised fine-tuning (SFT) and two preference fine-tuning (PFT) methods on four safety-aligned large language models, and find a mechanistic asymmetry between attack and defense. Results show that Odds Ratio Preference Optimization (ORPO) is the most effective for misalignment, and Direct Preference Optimization (DPO) performs best for realignment, but at the cost of model utility. The study also finds model-specific resistance phenomena and residual effects of multi-turn adversarial dynamics. The authors point out that additional safety realignment is needed before deploying untrusted third-party models, and call for stronger defenses and customized safety alignment strategies. Code is open-sourced at https://github.com/zhangrui4041/The-Art-of-Mis-alignment。