SecAlign: defending against prompt injection with preference optimization

AI summaryResearchers from Meta and UC Berkeley proposed SecAlign in a CCS 2025 paper, using preference optimization to train large language models to defend against prompt injection. The method forms preference pairs from injected inputs, safe outputs that respond to legitimate instructions, and unsafe outputs that respond to injections, and uses DPO to train the model to prefer safe outputs; the preference data can be automatically constructed from public instruction fine-tuning datasets without manual annotation. On five open-source models including Mistral-7B-Instruct and Llama3-8B-Instruct, SecAlign consistently achieved a 0% attack success rate against non-optimized injection attacks such as Ignore and Completion, and suppressed ASR to below 10% against optimization-based attacks unseen in training such as GCG, AdvPrompter, and NeuralExec (the strongest attack on Llama3-8B-Instruct was 8%), more than 4 times lower than the then SOTA defense StruQ, with AlpacaEval2 utility essentially on par with that before defense.