RL-Hammer: training an attack model from scratch with GRPO, achieving 72% attack success rate against defended GPT-5

Evaluating the real-world robustness of commercial-grade LLM agents against automated reinforcement learning prompt injection attacks.

• Application context and security risks: LLMs are being widely deployed as autonomous agents for OS control, code assistants, and automated workflows, but face prompt injection risks where malicious instructions embedded in external tool outputs can induce the model to perform unintended actions.
• Limitations of existing defenses: Model-level defenses such as Instruction Hierarchy and SecAlign show robustness under static or existing attacks, but the authors argue that previous attack methods were weak, so existing benchmarks may underestimate the actual risk of prompt injection.
• Challenges in reinforcement learning training: In black-box settings there is no gradient signal, and positive feedback is sparse against defended models, causing naive RL (e.g., GRPO) to drive attack success rates on commercial-grade targets down to near 0% (Figure 2b).
• Core idea and proposed approach: The authors propose RL-Hammer, an RL-based attack that requires no warm-start data, combining practical techniques such as removing KL regularization, joint training across multiple target models, and format constraints, so the attack model learns automated prompt injection from scratch.
• Threat model:
  • Attack goal: rewrite and amplify malicious injected instructions to induce the victim agent to execute attacker-specified tools and parameters when processing tool outputs.
  • Attacker knowledge: only black-box access to the target model, no gradients or logits, and only string parsing to determine whether an action was executed successfully (obtaining a 0/1 reward).
  • Attacker capability: controls the text content embedded in tool outputs, samples a policy through multi-turn interaction with the target model, and requires no initial labeled attack dataset.
  • Victim system: open-source and commercial-grade LLM agents with tool-calling capabilities, including systems deployed with defenses such as Instruction Hierarchy and Meta-SecAlign.

The paper reviews three main research directions related to automated prompt injection:

• Jailbreak attacks and automated red teaming
  • Discrete gradient optimization: Shin et al. (2020), Guo et al. (2021), Wen et al. (2023), Zou et al. (2023) (GCG), Zhu et al. (2023) (AutoDAN) demonstrate gradient-based adversarial suffixes and discrete optimization jailbreak methods in white-box scenarios.
  • Reinforcement learning automated red teaming: Perez et al. (2022), Paulus et al. (2024) (AdvPrompter), Guo et al. (2025) (Jailbreak-R1) use reinforcement learning to explore the safety boundaries of large models; the authors note that although Jailbreak-R1 applies GRPO, it relies on carefully constructed warm-start datasets.
• Prompt injection attacks
  • Discrete optimization and white-box injection: Liu et al. (2024a) propose a discrete optimizer for prompt injection; Pandya et al. (2025) develop an attention-based white-box attack capable of bypassing specific defenses.
  • Black-box preference optimization: Xu et al. (2024) (AdvAgent) propose a DPO-based black-box prompt injection training method; the authors note that this pipeline requires multiple training stages.
• Prompt injection defenses and detection
  • Model-level instruction defenses: Wallace et al. (2024) propose Instruction Hierarchy, which defends by distinguishing system-privileged instructions from tool inputs; Chen et al. (2024, 2025b) propose SecAlign and Meta-SecAlign, using preference optimization to improve defense robustness; Wu et al. (2024) propose instruction segment embeddings; Chen et al. (2025a) propose StruQ structured query defense.
  • Prompt injection detectors: Jain et al. (2023) propose a perplexity-based filtering benchmark; Meta (2025) open-sources Llama-Prompt-Guard-2-86M; ProtectAI.com (2024) develops the deberta-v3-base detector.
• Baseline methods and evaluation benchmarks: The experiments compare Default Prompt, Enhanced, Tool Knowledge, direct transfer based on Llama-3.1-8B-Instruct, the white-box attack GCG, and the reinforcement-learning-based jailbreak baseline Jailbreak-R1; benchmarks include InjecAgent, AgentDojo, and AdvBench.
• Positioning of this paper: The authors position RL-Hammer as a reinforcement learning attack framework trained from scratch without any warm-up data, without a value model or paired preference labels, relying only on black-box responses from the target model, used to examine the true boundaries of industrial-grade defense models.

作者提出 RL-Hammer，基于组相对策略优化（GRPO）算法从零训练攻击者模型，通过重写恶意目标生成对抗提示词注入工具输出中，并引入三项实用技术解决稀疏奖励与生成退化问题。

基础设定与朴素 GRPO 攻击

• 黑盒攻击形式化：攻击者模型输入恶意任务目标 x，生成对抗提示词注入工具输出并传入目标模型；系统自动通过字符串解析检测目标模型是否以正确参数执行指定动作，成功则返回奖励 1，否则返回 0。
• 组相对奖励计算：对每个目标采样 G 个候选输出，根据组内相对优势计算更新方向： Â_i,t = (r_i − mean(r))/(std(r)) 无需训练价值模型或构建成对偏好数据即可优化策略。
• 稀疏奖励瓶颈：作者称朴素 GRPO 在开源模型上可达到约 60% ASR（附录 Figure 2a），但面对具备防御的商业级模型时，正反馈稀疏导致 ASR 降至接近零（附录 Figure 2b）。

去除目标中的悲观约束

• 移除 KL 正则项：标准的 GRPO 优化目标包含约束策略偏离参考策略的 KL 散度项；作者认为该正则项限制了攻击者模型专门化适应提示注入任务的能力，因此直接设定权重系数 beta = 0。
• 降低计算开销：去除 KL 项后无需在前向传播中计算参考模型的 logits，节省了显存并加快了训练速度。

多目标模型联合软奖励训练

• 跨模型信号迁移：直接在鲁棒模型上训练无法获得学习信号，而仅在容易模型上预热会导致过拟合且迁移性差；作者在附录 Figure 3 中观察到攻击者在容易模型训练中期能偶然发现可迁移至鲁棒模型的策略。
• 联合软奖励函数：将鲁棒模型 f1 与容易模型 f2 联合训练，定义软奖励函数为攻击成功的目标模型比例： r(f_1, f_2, x, y) = \begin{cases} 1 & \text{若 } f_1(y) \text{ 与 } f_2(y) \text{ 均成功} \\ \alpha & \text{若仅 } f_1(y) \text{ 成功} \\ 1 - \alpha & \text{若仅 } f_2(y) \text{ 成功} \\ 0 & \text{若两模型均未成功} \end{cases} 其中 0 < alpha < 1 为权重因子；作者称细粒度软反馈能引导攻击者在两类模型上均获得高奖励。

强制限制输出格式

• 特殊标记约束：强制要求攻击者生成的提示词包裹在特殊标记内，以防模型坍缩为过长或无休止的重复文本。
• 联合判奖规则：仅当攻击同时成功且符合格式规范时才给予奖励 1；作者发现若为格式单独提供辅助奖励，模型会快速过拟合到格式任务而导致 ASR 明显下降。

训练实现与参数配置

• 实现细节：攻击者基础模型为 Llama-3.1-8B-Instruct，采用 LoRA 进行微调；每个注入目标执行 8 次采样，批大小为 8 个注入目标，学习率为 1e-5，训练 40 个 epoch。
• 软硬件平台：在单台 NVIDIA H200 节点上完成训练，代码基于 Hugging Face TRL 库构建；附录 A.1 给出了生成提示词所用的提示模板。

Experimental Setup

• Target Models Under Test: Includes Llama-3.1-8B-Instruct, Meta-SecAlign-8B, Meta-SecAlign-70B, GPT-4o-mini, GPT-4o, GPT-5-mini, GPT-5, Gemini-2.5-Flash, Claude-3.5-Sonnet, and Claude-4-Sonnet. White-box models use prompting to enable tool calling, black-box models use native tool-calling functionality.
• Evaluation Datasets and Scale: The main experiment uses the InjecAgent benchmark (training set 310 samples, validation set 100 samples, test set 100 samples); extended tasks use AgentDojo (test set 100 samples, validation set 100 samples) and the AdvBench jailbreak dataset.
• Baseline Methods: Default Prompt, Enhanced, Tool Knowledge, direct transfer based on Llama-3.1-8B-Instruct, white-box attack GCG, and the reinforcement learning-based jailbreak baseline Jailbreak-R1.
• Metric Definitions: Attack Success Rate (ASR, string parsing determines whether actions and parameters match), Detection Rate, diversity distance metrics (BLEU, BERTScore, Embedding Distance, LLM Judge).
• Review and Detection Tools: AdvBench tasks use HarmBench-Llama-2-13b-cls for judgment; detectors include a perplexity filter, Llama-Prompt-Guard-2-86M, deberta-v3-base-prompt-injection-v2 (ProtectAI-Guard), and an LLM Judge based on GPT-4o-mini.
• Test Sample Size: InjecAgent and AgentDojo test sets are both 100 samples; the paper does not report the number of experimental repetitions.

Main Results

Swipe horizontally to read the table

Target Model	Default Prompt	Enhanced	GCG	RL-Hammer (jointly trained on target)
Llama-3.1-8B-Instruct*	31.00%	46.00%	32.00%	100.00%
Meta-SecAlign-8B	0.00%	3.00%	21.00%	98.00%
Meta-SecAlign-70B	0.00%	1.00%	5.00%	99.00%
GPT-4o	0.00%	3.00%	1.00%	98.00%
GPT-5	0.00%	0.00%	Not reported	72.00%
Gemini-2.5-Flash	1.00%	5.00%	Not reported	92.00%
Claude-3.5-Sonnet	2.00%	1.00%	Not reported	83.00%
Claude-4-Sonnet	0.00%	0.00%	Not reported	77.00%

Note: According to Table 1. Llama-3.1-8B-Instruct marked with * is also the easy target model during joint training; the Meta-SecAlign-70B result is the transfer test performance of the model jointly trained with Meta-SecAlign-8B; due to table row limits, GPT-4o-mini (RL-Hammer reached 94.00%) and GPT-5-mini (RL-Hammer reached 92.00%) are omitted.

• Joint Training Performance: The authors claim RL-Hammer maintains high success rates across each commercial-grade target model tested, reaching 98.00% ASR on GPT-4o and 72.00% ASR on the defended GPT-5 (Table 1).
• Same-Family Model Transferability: An attacker jointly trained on Meta-SecAlign-8B reaches 99.00% ASR on the same-family Meta-SecAlign-70B, but transfers poorly to GPT-4o (11.00%) and Gemini-2.5-Flash (4.00%) (Table 1).
• Cross-Family Transfer Performance: An attacker jointly trained on GPT-4o reaches 70.00% ASR on the untrained Meta-SecAlign-70B and 56.00% ASR on Gemini-2.5-Flash, but 0.00% ASR on Claude-3.5-Sonnet (Table 1).

Extended Task Evaluation: AgentDojo and AdvBench

• Complex Agent Environment AgentDojo: In AgentDojo tests with a full tool environment, RL-Hammer achieves 51.00% ASR on GPT-4o and 26.00% ASR on Claude-3.5-Sonnet, higher than the Tool Knowledge baseline's 21.00% and 12.00% (Table 3).
• Jailbreak Benchmark AdvBench: On AdvBench jailbreak tasks, RL-Hammer reaches 99.04% ASR on GPT-4o and 97.11% ASR on Claude-3.5-Sonnet; the baseline Jailbreak-R1 under 10 attempts achieves 47.12% and 2.88% ASR on the two models respectively (Table 4).
• Authors' Interpretation: The authors believe RL-Hammer demonstrates effectiveness in complex multi-tool scenarios and cross-task jailbreaks, outperforming baseline methods that rely on multi-stage training and data warm-up.

Diversity Analysis and Metric Hacking

• Numerical Performance of Diversity Rewards: Without diversity rewards, model outputs have high similarity; after introducing diversity rewards such as BLEU, BERTScore, Embedding, or LLM Judge, the distance on the corresponding metric expands significantly and ASR remains at 97.00%–100.00% (Table 5).
• Metric Hacking Behavior: Manual inspection shows that with BLEU rewards, the model scores higher by changing case or replacing individual words; with Embedding rewards, the model changes vector representations by prepending unrelated random questions (e.g., ancient sock-wearing habits) before the core injection statement (Table 6); the authors state the model did not produce substantially new strategies.

Detection Rate and Detector Evasion

• Natural Evasion Capability: Prompts generated by RL-Hammer have a 0.00% detection rate on the perplexity filter, 16.00% on Llama-Prompt-Guard, and 17.00% on ProtectAI-Guard, while GCG has detection rates of 85.00%, 80.00%, and 100.00% on the three respectively (Table 7).
• Comprehensive Evasion with Stealth Reward: For the GPT-4o-mini LLM Judge, the only detector able to identify attacks at 85.00%, after introducing the LLM Judge reward in training, the attacker's detection rate drops to 0.00% on all 4 detectors while maintaining 97.00% ASR (Table 7).
• Authors' Interpretation: The authors believe current prompt injection detectors based on rules and lightweight classifiers have limitations, and more robust detection strategies need to be explored.

Other Ablations and Analyses

• KL regularization coefficient ablation: when the coefficient drops from 0.05 to 0, training reward keeps rising; with the coefficient set to 0, the final reward approaches 1.0 (Figure 1a).
• Joint training ablation: training directly only on the robust model keeps training reward near 0; after adding joint training with the easy model, both models' rewards rise to near 1.0 (Figure 1b).
• Reward form ablation: soft reward training converges significantly faster than hard reward (Figure 1c).
• Format reward ablation: without format constraints, generation length surges to hundreds or even thousands of tokens with heavy repetition; after adding the format reward, generation length stabilizes around 200 tokens (Figure 1d and Table 9).

Limitations and future directions noted by the authors

• Training cost and high query volume: The authors note in Section 6 that RL-Hammer training is costly, mainly because interacting with the target model requires a large number of queries, and in practice excessive adversarial queries may trigger detection and banning mechanisms from model providers.
• Training strategies to improve query efficiency: The authors suggest in Section 6 that future work could explore more query-efficient training schemes, such as using a local proxy model to prioritize promising queries, or other efficiency-improving techniques.
• Difficulty of generating diverse attacks: The authors note in Section 5.2 that the attack model tends to game the diversity reward by changing case or adding irrelevant text; how to develop robust methods that generate truly diverse attack strategies is an important future direction.
• Limitations of detection defenses: The authors note in Section 5.3 that existing detection methods see detection rates drop to 0% when facing attacks with added stealth rewards; future work needs more robust detection defense strategies against adaptive attacks.

Experimental coverage

• Models tested: Covers Llama-3.1-8B-Instruct, Meta-SecAlign-8B, Meta-SecAlign-70B, GPT-4o-mini, GPT-4o, GPT-5-mini, GPT-5, Gemini-2.5-Flash, Claude-3.5-Sonnet, and Claude-4-Sonnet, 10 models in total (Table 1).
• Tasks and benchmarks: Experiments include the prompt injection benchmark InjecAgent, the agent environment AgentDojo, and the jailbreak dataset AdvBench, 3 benchmarks in total (Section 4).
• Detector evaluation coverage: Covers 4 detection methods: a perplexity filter, Llama-Prompt-Guard-2-86M, ProtectAI-Guard, and an LLM Judge based on GPT-4o-mini (Table 7).
• Hardware and training scale: The attack model is fine-tuned from Llama-3.1-8B-Instruct with LoRA, and training runs for 40 epochs on a single NVIDIA H200 node (Section 4.1).
• Unreported information: The paper does not report the total number of queries or time cost during attack training, does not report standard deviations from repeated experiments with different random seeds, and does not report agreement checks between LLM judgments and human evaluation.

RL-Hammer is a black-box prompt injection adversarial attack framework trained from scratch using the GRPO algorithm, requiring no warm-start data.

Existing prompt injection defenses for LLM agents (such as Instruction Hierarchy and SecAlign) are mainly tested under static or weak attacks, and their true defensive capability against automated strong attacks has not been sufficiently examined.

The core mechanisms of the method include:

1. Using the GRPO algorithm to directly optimize via group-relative rewards, eliminating the need for a value model and paired preference labels;
2. Setting the KL regularization coefficient to 0, removing the pessimistic constraint that keeps the attack policy close to the initial distribution, accelerating convergence and reducing memory consumption;
3. Jointly training on robust and easy target models, with a soft reward mechanism designed to enable cross-model attack policy transfer;
4. Forcing outputs to be wrapped in special markers to avoid degenerate and repetitive verbose text.

Main experimental results include:

• On the InjecAgent benchmark, after joint training, ASR reaches 98.00% against GPT-4o, 72.00% against GPT-5, and 99.00% against Meta-SecAlign-70B (Table 1);
• In the AgentDojo agent environment, ASR is 51.00% against GPT-4o and 26.00% against Claude-3.5-Sonnet (Table 3);
• On the AdvBench jailbreak task, ASR reaches 99.04% against GPT-4o and 97.11% against Claude-3.5-Sonnet (Table 4);
• Against 4 prompt injection detectors, after adding an LLM Judge stealth reward, detection rates all drop to 0.00% while ASR remains at 97.00% (Table 7).

The authors argue that existing defense methods are not fundamentally robust, and previous evaluations may have underestimated the prompt injection security risks faced by agents; furthermore, the metric gaming observed during diversity training indicates that how to foster truly diverse red-team attack strategies and more robust defense mechanisms remains an open question.