COPSD: Constitution-Guided Online Policy Safety Distillation

The researchers propose a constitutional online policy safety distillation method, COPSD, which first uses Cross-SFT cold-start to calibrate the teacher model, and then performs constitution-conditioned online policy distillation. This method targets the collapse of safety online policy self-distillation (OPSD): the authors find that constitutional conditioning causes the teacher distribution to shrink into brief and overly conservative responses. On 3 multimodal large language models and 12 safety and general benchmarks, the paper claims that COPSD improves both safety and helpfulness over baselines, and reduces the safety tax on general reasoning.