PIPETTE: prompt injection purple teaming for robust LLM agents
PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
The researchers propose PIPETTE, a purple teaming framework that turns prompt injection red team findings into improvements in defense robustness. The paper notes that two RL-based red teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly finding similar attack strategies whose additional successes no longer bring robustness gains to adversarial training, so a high attack success rate in the red teaming phase does not guarantee a stronger defense. PIPETTE uses a dictionary-learning-based reward that encourages the red team to cover a broader range of attack strategies, and integrates an on-policy adversarial training pipeline for indirect prompt injection defense, enabling agents to complete legitimate user tasks while resisting attacks. Evaluations on the AgentDojo and ASB agent benchmarks show that defenses trained with PIPETTE achieve near-zero attack success rates against static attacks, can resist strong adaptive attacks, and maintain benign task utility close to the undefended baseline.