Mitigating many-shot jailbreaking attacks with a single safety example

Many-shot Jailbreaking (MSJ) makes safety-aligned language models answer harmful questions by prepending a large number of harmful question-answer examples before a harmful query. The paper empirically finds that as harmful examples increase, the representation of a fixed harmful query gradually deviates from the safety-aligned region, exhibiting progressive activation drift; theoretically, this drift can be explained as implicit malicious fine-tuning, i.e., conditioning on N harmful examples is equivalent to performing an SGD-style update on the corresponding N harmful samples. Accordingly, the authors propose appending a fixed safety example at inference time to induce a counteracting safety-direction update and restore refusal behavior, improving model robustness to MSJ without modifying model parameters and without requiring white-box access at deployment. The code is open-sourced at https://github.com/Thecommonirin/SafeEnd。