Backtracking technique improves language model generation safety

Researchers propose a backtracking technique that uses a special [RESET] token to let a language model retract and recover from already generated unsafe content, addressing the limitation that generated tokens cannot be retracted and that the model tends to continue generating unsafe text along the original trajectory. The method can be incorporated into SFT or DPO training to simultaneously optimize helpfulness and harmlessness. In the authors' evaluation, Llama-3-8B trained with backtracking is four times safer than the baseline model (6.1% → 1.5%), with no regression in helpfulness; although not trained against attacks, the method also provides protection against four adversarial attacks, including adaptive attacks.