PISmith: A Reinforcement Learning-Based Red-Teaming Framework for Prompt Injection Defense

Core problem: the robustness of existing prompt injection defenses under adaptive attacks has not been systematically evaluated, which the authors believe may create a false sense of security.

• Problem setting: LLM applications (especially autonomous agents interacting with external tools) are vulnerable to prompt injection attacks—attackers embed malicious instructions into the input context (e.g., web pages, documents) to manipulate outputs. Many defenses exist (preventive and filtering), but the authors cite Nasr et al. (2025) and Shi et al. (2025a) to argue that defenses reporting near-zero ASR may be substantially or even completely broken by adaptive attackers.
• Shortcomings of existing red-teaming methods: the authors state that static attacks (template-based) are not adaptive; search-based attacks (TAP, PAIR) require per-instance optimization and incur high inference-time compute; the RL-based RL-Hammer mitigates reward sparsity by joint training on undefended and defended targets, but increases training cost, and the authors' experiments show it remains suboptimal on difficult benchmarks.
• Core observation: applying standard GRPO directly to attack strong defenses suffers from two failure modes due to reward sparsity—the policy overfits to a few successful samples, causing entropy collapse and termination of exploration; rare successes are diluted by many failures in the gradient and are hard to learn effectively (Figure 1).
• Paper proposes: PISmith, an RL-based red-teaming framework introducing two mechanisms, adaptive entropy regularization and dynamic advantage weighting; and systematically evaluates on 13 benchmarks, 8 defenses, 7 baselines, and two agent benchmarks, InjecAgent and AgentDojo.
• Threat model:
  • Goal: make the output of a defended target LLM complete the injection task g (e.g., embedding phishing links, content promotion, denial of access, simulating infrastructure failure; Appendix C).
  • Knowledge: black-box—the attacker can only query the defended LLM and observe its outputs.
  • Capability: insert an injection prompt p into the data context xctx; train an attack LLM πϕ with on-policy RL to generate p.
  • Victim system: target LLMs equipped with prompt injection defenses (e.g., Meta-SecAlign-8B, GPT-4o-mini with instruction hierarchy, GPT-5-nano) and tool-based agents.

Reinforcement learning for LLMs

• PPO (Schulman et al., 2017) is a foundational method for aligning LLMs with human feedback, but relies on a learned value model and is resource-intensive.
• GRPO (Shao et al., 2024) computes advantages using group-relative rewards and removes the value model; it performs well on mathematical reasoning and code generation and is the base of this paper's method.
• Appendix A.2 discusses entropy-aware GRPO variants: DAPO (Yu et al., 2025)'s Clip-Higher and Dynamic Sampling, AEPO (Wang et al., 2025a)'s fixed entropy target, etc.; the authors argue these target reasoning tasks and differ from the reward-sparsity form of prompt injection red-teaming—Dynamic Sampling discards almost the entire training batch under strong defenses, and a fixed entropy target cannot match red-teaming's need to "explore first, then exploit."

Prompt injection attacks (the paper divides them into three categories)

• Static attacks (Willison, 2022; Perez & Ribeiro, 2022; Liu et al., 2024, etc.): rely on predefined templates (e.g., context ignoring, fake completion) and are inherently non-adaptive; Liu et al. (2024)'s Combined Attack combines multiple static strategies.
• Search-based attacks: TAP (Mehrotra et al., 2024) organizes optimization as tree search, PAIR (Chao et al., 2025) as multi-turn dialogue, and PIArena (Geng et al., 2026)'s Strategy rewrites injection prompts using a strategy library; the authors note such methods optimize each instance independently and incur high inference-time compute.
• RL-based attacks: RL-Hammer (Wen et al., 2025) fine-tunes an attack LLM with GRPO, mitigating reward sparsity by joint training on weak (undefended) and strong (defended) targets; the authors argue this increases training cost and does not fundamentally solve the RL difficulty under sparse rewards. Nasr et al. (2025) showed attackers can bypass defenses using general optimization techniques such as RL; the authors note that the technical details and code of its RL attack component are not fully public.

Prompt injection defenses (the paper divides them into two categories)

• Filtering-based: PromptGuard (Meta, 2024) fine-tunes DeBERTa to classify injections; DataSentinel (Liu et al., 2025a) models detection as a minimax game; others include Attention Tracker, PIGuard, PIShield, etc.
• Preventive: Instruction Hierarchy (Wallace et al., 2024) trains models to prioritize privileged instructions; SecAlign/Meta-SecAlign (Chen et al., 2025b;c) uses DPO for preference optimization; others include PromptArmor, DataFilter, PISanitizer, SecInfer, etc. Appendix A.4 also discusses defenses based on security policies (CaMeL, Progent, DRIFT, etc.); the authors note these mainly target agent scenarios with explicit tool-calling interfaces and are hard to generalize to tasks such as QA.

Baselines and benchmarks: attack baselines are Direct, Combined (static), TAP, PAIR, Strategy (search-based), Vanilla GRPO, RL-Hammer (RL-based); evaluated defenses are Meta-SecAlign-8B and Sandwich, Instructional, PromptArmor, DataFilter, PIGuard, PromptGuard, DataSentinel; benchmarks are PIArena's 13 datasets and InjecAgent, AgentDojo, AgentDyn.

Authors' positioning: PISmith directly handles reward sparsity with adaptive entropy regularization and dynamic advantage weighting, without training on multiple objectives as RL-Hammer does; the authors call this a targeted improvement over the limitations of existing RL red-teaming methods.

整体思路：把注入攻击形式化为策略优化问题，用 GRPO 变体在黑盒下训练攻击 LLM PISmith，针对强防御导致的奖励稀疏设计两个按当前平均奖励 r̄ 调节的机制。

问题形式化

• 训练样本由目标任务 (xinst, xctx) 和注入任务 g 组成；攻击者把注入提示词 p 插入数据上下文 xctx。
• 优化目标（Eq. 1）：max_πϕ 1/(|D|)∑_{(xinst,xctx,g)∈ D} 𝔼_{p∼πϕ(·|xinst,xctx,g)}  r(M_θ(x_inst, x_ctx⊕ p), g)，其中 r 为二元奖励：目标 LLM 的响应完成注入任务 g 则为 1，否则为 0。
• 标准 GRPO：每个样本采样 K 个 rollout，优势按组内均值和标准差归一化（Eq. 2），用带裁剪的代理损失加对参考策略的 KL 惩罚更新（Eq. 3）。

标准 GRPO 的两个失效模式（§3.2）

• 熵坍缩：奖励极稀疏时，GRPO 强化少数偶然成功中的特定模式，策略熵持续下降、rollout 趋同，探索在找到有效策略前终止，梯度随之消失。
• 成功信号被稀释：即使维持探索，稀有成功在梯度中被大量失败淹没；失败只说明某种措辞无效，成功才暴露防御弱点。
• 附录 B 论证标准 GRPO 的 KL 项为何不够：KL 分解为 −H(πϕ) + H(πϕ, πref)，其中交叉熵项把策略锚定在不懂攻击的参考策略上，阻碍发现新注入策略，因此作者用纯熵奖励替换 KL 项。

自适应熵正则（§3.3.1）

• 只保留熵项并加上限：L_entropy = −β(r̄) · H(π_ϕ) 当 H(π_ϕ) < H_cap，否则为 0（Eq. 4）；Hcap = 0.5，防止早期熵项主导优化、生成退化为无语义序列。
• β 随当前平均奖励 r̄ 自适应（Eq. 5）：r̄ 低于阈值 τ 时取 βmax 附近的强探索，r̄ 升到 τ 以上后线性衰减到 βbase，转向利用已发现的策略。

动态优势加权（§3.3.2）

• 对成功 rollout（ri = 1）的优势乘以 γ(r̄)，失败不变（Eq. 6）。
• γ(r̄) 在 r̄ < τ 时为 1 + (γmax − 1)·(τ − r̄)/τ，r̄ ≥ τ 时为 1（Eq. 7）；γmax = 5.0。r̄ 低时放大稀有成功的梯度贡献，r̄ 升高后恢复标准 GRPO。

完整目标与关键参数

• 总损失（Eq. 8）为带裁剪的代理损失（用加权优势 Ãi）加 Lentropy；作者称两机制互补：熵正则保证产生稀有成功，优势加权保证从中学习，r̄ 升高后两者同时放松、平滑回到标准 GRPO。
• 关键参数（附录 D.1）：攻击基座 Qwen3-4B-Instruct-2507；Hcap = 0.5，βmax = 0.01，βbase = 0.001，γmax = 5.0，τ = 0.5；GRPO 组大小 K = 15，学习率 1×10⁻⁵，裁剪阈值 ϵc = 0.2，训练 10 epoch。
• 训练数据：非智能体基准只用 Dolly Closed QA 前 100 条；InjecAgent 沿用 RL-Hammer 的数据划分；AgentDojo 用 workspace 域前 7 个（共 14 个）注入任务配全部用户任务。
• 成功判定：GPT-4o-mini 作为 LLM-as-judge，输入注入任务和目标响应，输出 YES/NO（附录 D.2 给出评审提示词）；附录 D.3 给出三类基准的攻击 LLM 训练提示词模板。

Experimental setup

• Attacked targets: Meta-SecAlign-8B (RQ1, ablation); Qwen3-4B-Instruct-2507 as backend with 7 defenses (RQ2); on InjecAgent, Meta-SecAlign-8B, GPT-4o-mini, GPT-4.1-nano, GPT-5-nano; on AgentDojo, three closed-source models (Meta-SecAlign-8B excluded because its agent-task utility is too low).
• Benchmarks: PIArena's 13 datasets (4 QA, 3 RAG, 6 long-context), totaling 200×4 + 100×9 = 1,700 evaluation cases (Appendix D.4); agent benchmarks InjecAgent, AgentDojo, with AgentDyn's GitHub subset for extended experiments.
• Attacker: PISmith's base is Qwen3-4B-Instruct-2507, trained only on 100 Dolly Closed QA examples; RL baselines (Vanilla GRPO, RL-Hammer) share the same base, training data, and evaluation protocol; RL-Hammer additionally uses Llama-3.1-8B-Instruct as the undefended joint-training target; search-based baselines (TAP, PAIR, Strategy) optimize per instance, with the same attack LLM, Qwen3-4B-Instruct-2507.
• Metrics: N = 10 candidate injection prompts generated per instance; ASR@10 (at least one success), ASR@1 (single-attempt success rate, averaged over 10); Utility is task completion accuracy without attack.
• Judging: GPT-4o-mini as LLM-as-judge outputs YES/NO; human verification in Appendix H.

Main results

Table 1 (against Meta-SecAlign-8B; static/search-based report ASR@1, RL-based report ASR@10/ASR@1; excerpt of average row and representative rows):

Swipe horizontally to read the table

Benchmark	Direct	Combined	TAP	PAIR	Strategy	Vanilla GRPO	RL-Hammer	PISmith
SQuAD v2	0.01	0.01	0.09	0.10	0.09	0.01/0.01	0.94/0.64	1.0/0.94
Dolly Closed QA	0.03	0.06	0.18	0.30	0.30	0.26/0.10	0.95/0.78	1.0/0.94
HotpotQA RAG	0.02	0.04	0.14	0.15	0.09	0.22/0.10	0.89/0.59	1.0/0.91
HotpotQA Long	0	0	0	0	0.02	0/0	0.04/0.01	0.99/0.61
Qasper	0.01	0.01	0.07	0.1	0.24	0.05/0.02	0.21/0.10	0.99/0.69
LCC	0.06	0.11	0.23	0.37	0.29	0.31/0.18	0.74/0.48	0.99/0.89
Average (13 benchmarks)	0.04	0.07	0.11	0.16	0.21	0.13/0.05	0.70/0.48	1.0/0.87

(The full 13 rows include Dolly Info, Dolly Sum, NQ, MSMARCO, GovReport, MultiNews, Passage Retr.; PISmith's rows are 0.99–1.0/0.61–0.97.)

• The authors state that static attacks have very low ASR on all benchmarks, and search-based attacks outperform static but average ASR@1 of only 0.11–0.21.
• The authors state that although PISmith is trained only on 100 Dolly Closed QA examples, it generalizes to 12 unseen benchmarks; on difficult tasks that restrict output format such as HotpotQA-Long, its improvement over RL-Hammer is largest.
• Query cost in Appendix D.4: PISmith total queries 32,000 (training 15,000 + inference 17,000), RL-Hammer 47,000, TAP 82,110, PAIR 74,460, Strategy 194,140.

Utility–robustness trade-off (RQ2, Table 7, Figure 2)

• Backend is Qwen3-4B-Instruct-2507, 13-benchmark average: No Defense Utility 0.74, PISmith ASR@1 0.92; Sandwich, Instructional, PromptArmor Utility 0.73–0.74, ASR@1 0.91–0.92; DataFilter Utility 0.63 (−0.11), ASR@1 0.49; PIGuard 0.72 (−0.02), 0.82; PromptGuard 0.66 (−0.08), 0.89; DataSentinel 0.55 (−0.19), 0.52.
• The authors state that no defense falls in the lower-right region of Figure 2 (high utility, low ASR): defenses that preserve utility are more easily broken by PISmith, and defenses with lower ASR incur large utility loss; the authors' explanation is that more aggressive defenses indiscriminately block both malicious and normal context.

Agent scenarios (RQ3, Tables 2, 3, 5)

• InjecAgent: PISmith's ASR@10/ASR@1 against the four targets are Meta-SecAlign-8B 1.0/0.99, GPT-4o-mini 1.0/0.99, GPT-4.1-nano 1.0/1.0, GPT-5-nano 1.0/0.95; on GPT-5-nano static attacks (Direct 0.00, Enhanced 0.00) fail completely, and RL-Hammer is 0.96/0.86.
• AgentDojo (trained on the workspace subset): PISmith against GPT-4o-mini 0.78/0.62, GPT-4.1-nano 0.81/0.64, GPT-5-nano 0.38/0.24; the highest among 6 template baselines is Imp. Instr. at 0.23 (GPT-4o-mini).
• Extended training data (Table 5): after retraining with the AgentDyn GitHub subset added, on GPT-5-nano AgentDojo rises to 0.75/0.52, and on AgentDyn it is 0.88/0.60 (Direct baseline 0.01); the authors state that attack performance is not fundamentally limited by stronger targets but expands with training coverage.

Other ablations and analyses

• Component ablation (Figure 3, Meta-SecAlign-8B, 13-benchmark average ASR@1): Vanilla GRPO 0.05, w/o Entropy 0.09, w/o Boost 0.57, full PISmith 0.87.
• Training data diversity (Table 4, target Meta-SecAlign-8B): expanding the training set from 100 Dolly examples to 300 (100 each of Dolly, HotpotQA-RAG, HotpotQA-Long) raises average ASR@1 from 0.87 to 0.99, with ASR@10 remaining 1.00.
• Hyperparameter sensitivity (Table 19, same ablation setting): with γmax at 1.0/2.0/5.0/10.0, ASR@1 is 0.57/0.67/0.87/0.20; with Hcap at 0.2/0.5/1.0, 0.75/0.87/0.78; with τ at 0.2/0.5/0.8, 0.78/0.87/0.81.
• Comparison with general entropy-aware GRPO variants (Table 20, ported to the same attack pipeline, target Meta-SecAlign-8B): Vanilla GRPO 0.05/0.13, EPO 0.64/0.50, DAPO 0.28/0.14, PISmith 1.00/0.87 (ASR@10/ASR@1; the two Vanilla GRPO values in the table are in the reverse order of 0.13/0.05 in Table 1, copied as in the original source).
• Human verification of judging (Appendix H): 200 random samples (covering 4 injection task types, 3 task families), two annotators with 100% agreement, human consensus agreeing with GPT-4o-mini judging on 196/200 (98%); all 4 disagreements were cases where the judge accepted a response humans considered partially complete, from which the authors give an upper bound of 2% on the judge's false positive rate.

Limitations and future directions noted by the authors

• Extending to stronger attack models (Appendix A.1): the authors state that PISmith uses Qwen3-4B-Instruct-2507 by default, expect that switching to stronger general-purpose LLMs (the text cites DeepSeek V4 Pro and GLM-5.2) would further improve attack performance, and leave a systematic study of this scaling trend to future work.
• From red-teaming to defense (Appendix A.1, §5): the authors state that PISmith can serve as an adaptive attacker in defense training, co-training with the target model (citing the concurrent work OpenAI GPT-Red), rather than only evaluating fixed defenses post hoc.
• Developing defenses that balance robustness and utility (§5): the authors state that current LLMs and their prompt injection defenses are still insufficient to support secure real-world deployment, and that more resilient and utility-preserving defenses are needed.
• The paper has no dedicated Limitations section; the above are future directions explicitly stated in the conclusion and appendices.

Scope of experiments

• The attacked target models are Meta-SecAlign-8B, Qwen3-4B-Instruct-2507 (backbone), GPT-4o-mini, GPT-4.1-nano, and GPT-5-nano, 5 in total (§4.2–4.4).
• The attack LLM is fixed as Qwen3-4B-Instruct-2507; non-agent benchmark training data is only 100 Dolly Closed QA examples, and AgentDojo uses only the first 7 injection tasks in the workspace domain (Appendix D.1).
• 8 defenses are evaluated: Meta-SecAlign-8B and Sandwich, Instructional, PromptArmor, DataFilter, PIGuard, PromptGuard, DataSentinel; the security-policy-based defenses discussed in Appendix A.4 (CaMeL, Progent, DRIFT, etc.) do not appear in the experiments.
• Attack success is uniformly judged by GPT-4o-mini; human validation uses 200 samples and two annotators (Appendix H).
• The paper does not report training random seeds, variance across repeated training runs, or experiments on adaptive defenses against PISmith (i.e., the effect after the defender retrains against PISmith).

A prompt injection red-teaming attack paper: the authors argue that systematic evaluation of existing defenses' robustness to adaptive attacks is lacking, and propose an RL-based red-teaming framework, PISmith.

• Problem: static template attacks are non-adaptive; search-based attacks are expensive due to per-instance optimization; the RL-style RL-Hammer alleviates reward sparsity via multi-objective joint training, but the authors claim it is still suboptimal; directly applying standard GRPO to attack strong defenses leads to entropy collapse and dilution of success signals due to reward sparsity (Figure 1).
• Method: in a black-box setting, train an attack LLM (Qwen3-4B-Instruct-2507) with a GRPO variant; both mechanisms are adjusted by the current average reward r̄: adaptive entropy regularization with an entropy cap maintains exploration, and dynamic advantage weighting amplifies the gradients of successful rollouts according to their scarcity; once r̄ rises, it reverts to standard GRPO.
• Main results: against Meta-SecAlign-8B, PISmith achieves an average ASR@10/ASR@1 of 1.0/0.87 across 13 benchmarks, versus 0.70/0.48 for RL-Hammer, and the highest average ASR@1 among search-based baselines is 0.21 (Table 1); training uses only 100 Dolly Closed QA examples.
• Utility–robustness: among 7 defenses (backbone Qwen3-4B-Instruct-2507), the utility-preserving ones (Sandwich et al., Utility 0.73–0.74) reach ASR@1 of 0.91–0.92, while DataSentinel, which has lower ASR (0.52), drops to Utility 0.55; the authors state that no defense achieves both (Table 7, Figure 2).
• Agent scenarios: on InjecAgent, 1.0/0.95 against GPT-5-nano (static attacks: 0); on AgentDojo, 0.78/0.62 against GPT-4o-mini and 0.38/0.24 against GPT-5-nano, rising to 0.75/0.52 after retraining with the AgentDyn GitHub subset added (Tables 2, 3, 5). Ablations show that removing entropy regularization or advantage weighting lowers average ASR@1 to 0.09 and 0.57, respectively (Figure 3).
• Authors' conclusions: existing defenses remain fragile under adaptive attacks, and defenses that balance robustness and utility are urgently needed; the authors plan to extend PISmith to stronger attack backbones and to use it for defense co-training (Appendix A.1). The judge is GPT-4o-mini, with 98% agreement with human consensus (Appendix H).