SecPO: Principled adversarial training for prompt injection security
SecPO: Principled Adversarial Training for Prompt Injection Security
Quick read
Based on the paper's full text (AI-generated); refer to the original
The researchers propose the SecPO loss and combine it with adversarial training for model-level indirect prompt injection defense: on Qwen3.6-27B, they reduce the ASR@800 of AgentDojo's AttackerMovesSecond adaptive attack from 78.8% for Meta-SecAlign to 8.8%. SecPO replaces the KL constraint in DPO toward an undefended reference model with an oracle distribution obtained by querying that model on clean inputs, theoretically bypassing the problem of anchoring to an unsafe reference model, and in practice also gives injection-sensitive samples larger gradients. On the training side, PAIR search is used to synthesize strong injections (best@240) for each sample for adversarial training. On SEP, the ASR of three adaptive attacks, Genetic, TAP, and PISmith, drops from 93.7%, 90.1%, and 94.0% to 13.3%, 12.5%, and 1.8%, respectively; on DTaP it is 7.6%; and benign utility is essentially unchanged on nine benchmarks involving tool calling, instruction following, science, and mathematics.
- Prompt injection lets instructions in untrusted data hijack AI agents and is the primary threat to agents. The strongest existing defense, Meta-SecAlign, still has an ASR close to 100% under adaptive attacks; the authors believe its DPO loss anchors the model to an unsafe undefended reference model and that it is trained only with weak simulated attacks.
- The authors propose the SecPO loss: replacing the undefended reference model in DPO with an ideal oracle, leveraging the observation that the oracle's output distribution on injected inputs equals the undefended model's distribution on clean inputs, so it suffices to feed the reference model clean inputs. Combined with adversarial training (AT) based on PAIR search, strong injection samples are synthesized concurrently via attacker/judge APIs for training.
- On Llama-3.1-8B-Instruct and Qwen3.6-27B, AT reduces the ASR of three adaptive attacks by 1.5–14.5 times compared with Meta-SecAlign, and SecPO further reduces it by 2.4–4.6 times; Qwen3.6-27B under AgentDojo AttackerMovesSecond has an ASR@800 of 8.8% versus 78.8%. On nine benign benchmarks, utility shows no significant decline, and there is also transfer robustness against visual injection.
- The method assumes the agent can clearly distinguish trusted instructions from untrusted data, and does not cover cases where labels are unknown or the user is themselves malicious; the main experiments are in non-reasoning mode, and extension to reasoning models is left for future work; it does not fix pre-existing errors such as hallucination; adaptive attack evaluation is costly, making it difficult to systematically evaluate hyperparameter sensitivity; stronger new attacks may still break it.
The paper provides a reproducible adversarial training recipe and an open-source training set, and quantifies the reduction in attack success rate relative to Meta-SecAlign on agent tasks such as AgentDojo.