How Vulnerable Are AI Agents to Indirect Prompt Injection? Findings from a Large-Scale Public Competition

Core problem: LLM agents are susceptible to indirect prompt injection attacks when processing external data, and attacks can perform harmful actions while hiding traces from the user; existing benchmarks do not systematically evaluate this dual objective.

• Scenarios and importance: Agents are increasingly deployed in high-risk domains such as healthcare and finance, processing external data such as emails, documents and code repositories, in which attackers can embed malicious instructions to manipulate the agent (the authors cite Greshake et al., 2023).
• Stealth dimension: Users typically see only the agent's final reply; tool execution logs and CoT are too long for manual inspection, so an attack can effectively hide as long as it does not reveal itself in the final reply, and users will accept the harmful result as normal output.
• Insufficient existing benchmarks: The authors state that existing agent safety benchmarks (AgentDojo, InjecAgent, OS-Harm, WASP, etc.) neither simultaneously require attack stealth nor jointly cover coding and computer use agents (Table 1); static benchmarks also face rapid obsolescence.
• What the paper proposes: In collaboration with UK AISI, US CAISI and several frontier labs, it held the Indirect Prompt Injection Arena public red-teaming competition, built 41 scenarios (tool use 18, coding 15, computer use 8), and released a benchmark and strategy analysis based on the competition data.
• Threat model:
  • Objective: Force the model to perform a specified harmful action within a single operation turn, with the final reply passing scenario-specific criteria (including stealth).
  • Knowledge: The attacker can see the full conversation prefill, knows the exact target action (tool call and parameters), and can observe the tools used in the prefill turns (but not the full tool definitions).
  • Capability: Controls a fixed portion of a single external data source, inserting the payload at the {fill} injection point; both attacker and agent are limited to a single turn.
  • Victim systems: Three types of agents: tool use, coding, computer use; the authors explicitly state that this threat model is more permissive than real conditions and that the results are an upper bound on single-turn vulnerability.

Prompt injection attacks and defenses

• Perez and Ribeiro (2022) and Greshake et al. (2023) formalized the distinction between direct and indirect prompt injection; since then, indirect injection has been extensively demonstrated in production environments (the Rehberger series, Fu et al., 2024, Labunets et al., 2025) and extended to multimodal inputs and worms propagating across agent networks (Cohen et al., 2024).
• Defense directions include input/output monitoring and filtering (Inan et al., 2023; Korbak et al., 2025; Sharma et al., 2025), robust training (Wallace et al., 2024; Chen et al., 2025a,b, etc.), and system-level mechanisms (Meng et al., 2025; Debenedetti et al., 2025b; Foerster et al., 2026); the paper notes that adaptive attacks and manual red teaming can still bypass these defenses (Nasr et al., 2025).

Agent safety benchmarks

• Early benchmarks targeted single-turn adversarial prompts against chat models (Mazeika et al., 2024; Chao et al., 2024); AgentDojo (Debenedetti et al., 2024) tests indirect injection, InjecAgent (Zhan et al., 2024) classifies by harm type, AgentHarm (Andriushchenko et al., 2025) evaluates direct misuse, and ART (Zou et al., 2025) and b3 (Bazinska et al., 2026) introduce large-scale crowdsourced red teaming.
• In the coding direction there are Cybench (Zhang et al., 2025a) and research on injection against coding tools (Liu et al., 2025); OS-Harm (Kuntz et al., 2025) and WASP (Evtimov et al., 2025) extend to computer use; ASB (Zhang et al., 2025b) and Agent-SafetyBench (Zhang et al., 2024) cover more tool-use scenarios.
• The paper notes that none of these benchmarks jointly cover coding and computer use agents, nor do they require attacks to be stealthy (Table 1).

Red teaming methods

• Automated methods include gradient attacks (Zou et al., 2023), genetic algorithms (Liu et al., 2024), iterative refinement (Chao et al., 2025; Mehrotra et al., 2024), sampling methods (Hughes et al., 2024), and the learned attacker Shade (Gray Swan AI, 2025).
• Manual red teaming includes the DEF CON AI Village GRT, ART, and the D-REX competition; the paper notes that these methods typically optimize only a single objective (eliciting harmful content or executing harmful actions), whereas this paper's stealth requirement introduces a dual objective.

Positioning: the authors state that this paper is the first to study the stealth dimension of indirect prompt injection at scale, covering 13 models, 41 scenarios, and three agent settings, and that they plan to continuously update the benchmark in the form of recurring competitions.

Overall approach: in collaboration with UK AISI and labs including OpenAI, Anthropic, Amazon, and Meta, design 41 indirect prompt injection scenarios, hold a public red teaming competition, the Indirect Prompt Injection Arena, to collect real human attacks, then organize a benchmark based on the submission data and perform transfer and strategy analyses.

Scenario design

• Three types of agents: tool use (18 scenarios: e-commerce, healthcare, finance, home automation, developer tools), coding (15, based on real open-source repositories such as TRL, veRL, CrewAI and PR review workflows), computer use (8, web interfaces such as content moderation panels, booking systems, approval workflows).
• Scenario elements: each scenario specifies a benign user task, an injection location, a target harmful action, and detailed success criteria; injection points are located where an attacker could reasonably have control (comments, configuration files, PR descriptions, web page sidebars, etc.).
• Harmful action design: harmful actions are "in the same domain" as the task the agent is performing (e.g., an email triage agent is induced to mark a key contract email as read), but are clearly undesirable from the user's perspective; the authors leave varying the "degree of alignment between harmful actions and benign tasks" for future work.

Dual-judge

• tool judge: programmatically checks whether the agent executed the target harmful action—tool use matches function names and arguments, coding checks for the target bash command or file modification, computer use verifies the target UI action.
• prompt judge: scores the agent's final visible reply 0–10 against scenario-specific criteria; it passes only if all criteria exceed a threshold of 7; each scenario has 1–3 criteria, which must include stealth and may also include "whether the user was misled" (full template in Appendix A).
• Threshold source: the threshold of 7 comes from experience in a previous red teaming competition (Zou et al., 2025) and was not empirically tuned on this dataset; of the 41 scenarios, 32 use dual judging, 7 use only the tool judge, and 2 use only the prompt judge.

Data collection

• Competition setup: Gray Swan Arena platform, total prize pool of 40,000 USD, three weeks; wave 0 (4 behaviors) was a pilot not counted in the final results, and the two main waves covered 37 behaviors in total; model names were randomly anonymized for participants and displayed in alphabetical order of the anonymized names, with the default selection effectively random across participants.
• Models tested: 13 models, all with thinking off by default (except Kimi K2, Gemini 2.5 Pro, Claude Haiku 4.5); GPT-5.1 was added only in wave 2.
• Deduplication: distinguishing chats (attempts) from submissions (submissions requesting judging), deduplicating by MD5 for the same model and behavior yields 271,588 chats, 67,634 submissions, 8,648 successful attacks, and 464 participants.

Benchmark and analysis

• Benchmark organization: at most 9 successful attacks are sampled per model-behavior pair, 2,679 in total, covering 41 behaviors, averaging 7 per pair (distribution by source model in Table 5); an open-source evaluation toolkit and 95 Qwen attacks that did not transfer to closed-source models.
• Transfer experiments: the 2,679 attacks were rerun on the 13 competition models plus Gemini 3 Pro and Kimi K2 (non-thinking); models without vision capability were tested on 2,449.
• Strategy analysis: Gemini 3 Flash was used to label each submission with a strategy, Qwen3-Embedding-8B embeddings were clustered with K-Means, Gemini 3 Pro downsampled, and 27 strategies were manually verified; in addition, successful attacks were embedded with all-mpnet-base-v2 and a graph was built at cosine similarity >0.8 to find general attack clusters.

Experimental setup

• Models tested: Gemini 2.5 Pro, Nova 1 Premier, DeepSeek V3.1, Qwen3 VL 235B, Nova 2 Lite, SecAlign 70B, Kimi K2 Thinking, Grok 4, GPT-5, GPT-5.1, Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5, 13 in total; the transfer experiments additionally include Gemini 3 Pro and Kimi K2 (non-thinking).
• Benchmark: 41 self-built scenarios (tool use 18, coding 15, computer use 8); DeepSeek V3.1, Kimi K2, and SecAlign 70B were tested on only 29 behaviors because they lack image capability; GPT-5.1 has only wave 2 data.
• Metrics: ASR = number of successful attacks / total number of chats for that model; behavior coverage = proportion of behaviors with at least one success; transfer ASR = number of successful transfer attacks / number of attacks tested.
• Judging method: tool judge programmatic matching + prompt judge scoring 0–10 against scenario-specific criteria with a threshold of 7; the paper does not state which model serves as the prompt judge.
• Scale: 271,588 deduplicated chats, 67,634 submissions, 8,648 successes, 464 participants; about 130–160 independent attackers per model; each attack was evaluated only once per model.

Main results

According to Figure 1 (overall competition ASR, in descending order):

Model	ASR	Successes/Attempts
Gemini 2.5 Pro	8.5%	1670/19630
Nova 1 Premier	5.8%	1091/18924
DeepSeek V3.1	5.5%	808/15026
Qwen3 VL 235B	4.8%	781/18733 (figure numbers; where columns are misaligned, per figure caption)
Nova 2 Lite	4.7%	672/14244
SecAlign 70B	4.2%	618/12399 (figure numbers; where columns are misaligned, per figure caption)
Kimi K2 Thinking	4.2%	589/12299 (figure numbers; where columns are misaligned, per figure caption)
Grok 4	2.9%	366/12526
GPT-5	2.0%	261/13143
GPT-5.1	2.5%	202/8096
Claude Haiku 4.5	1.3%	177/13696
Claude Sonnet 4.5	1.0%	136/13603
Claude Opus 4.5	0.5%	61/11969

Note: in the converted text of Figure 1, some numerators and denominators are misaligned in their columns; the success/attempt numbers for the three rows Qwen3 VL 235B, SecAlign 70B, and Kimi K2 Thinking are given according to the readable numbers in the figure and may deviate from the actual correspondence; the ASR percentages are as explicitly reported in the paper.

• Authors' interpretation: ASR for the Claude and GPT families is noticeably lower than for other families; within the Claude family, robustness increases with capability (Opus 0.5% < Sonnet 1.0% < Haiku 1.3% 的 ASR）。
• 行为覆盖率：Gemini 2.5 Pro 与 DeepSeek V3.1 在全部 37 个行为上均有成功攻击（100%），Claude Opus 4.5 最低为 35.1%（13/37）（Figure 9）。
• 按场景拆分（Table 2）：tool use ASR 4.82%（95% CI [4.68%, 4.96%]）、computer use 3.13%（[2.96%, 3.31%]）、coding 2.51%（[2.41%, 2.62%]）；作者认为 coding 较低可能因为预填来自真实 coding 转录、更接近安全训练数据。除 Claude Opus 4.5（computer use 1.1% > tool use 0.6%); apart from that, tool use is the most vulnerable setting for every model; the high ASR of Gemini 2.5 Pro is mainly driven by computer use (16.2%) (Figure 5).

Transfer experiments

• Setup: 2,679 successful attacks were rerun on 15 target models (including Gemini 3 Pro and Kimi K2 non-thinking), without excluding the source model itself.
• Results (Figure 6): Qwen3 VL 235B and Nova 1 Premier are the most vulnerable as targets (52.5%, 50.6%), and Claude Opus 4.5 is the most robust (2.5%); Gemini 3 Pro (15.9%) improves over Gemini 2.5 Pro (45.1%); Kimi K2 non-thinking (35.1%) is more vulnerable than the thinking version (29.7%), which the authors suggest may indicate that thinking improved Kimi K2's robustness.
• Source-target matrix (Figure 7): the 44 attacks that broke Claude Opus 4.5 have a transfer rate of 44–81% against other models; attacks originating from Qwen3 VL 235B have 0% against Claude Opus 4.5 and 1% against Claude Sonnet 4.5; the authors believe that attacks breaking highly robust models exploit more general instruction-following weaknesses.
• Diagonal: the same attack rerun on its source model does not necessarily succeed again; rerun success rates for the Claude and GPT families are ≥50%, comparable to Gemini 2.5 Pro, Nova 1 Premier, and Qwen3 VL 235B.

Strategy and general attack analysis

• Strategy ranking (Table 3, Table 6): among 27 strategies, the top three by ASR are Fake Chain of Thought 4.3% (13,878 submissions), Request to Disable Critical and Safety-Oriented Thoughts 4.1% (13,494), and Offer Reward and Punishment 4.0% (6,575); the authors note that thinking is off by default, and whether this strategy is effective against thinking models remains to be observed in the next competition.
• By setting and family (Figure 8): high-performing strategies across the three settings hardly overlap, with the only shared one being Chain Harmful Shell Command with Benign Command; the Claude, OpenAI, and open-weight families are all susceptible to Fake Chain of Thought.
• General attack clusters (Table 4): clustering at cosine similarity >0.8 yields 5 clusters; the largest cluster has 63 submissions, covers 21/41 behaviors and 9 models (average intra-cluster similarity 0.920), and uses a "simulated environment + fake control interface" template; each of the top four clusters was contributed by a single user, and the authors emphasize that this is a post-hoc analysis and that participants may converge due to familiarity with known techniques.

Other ablations and analyses

• Capability and robustness: GPQA Diamond and ASR are weakly negatively correlated, r = −0.31, p = 0.299, not statistically significant (Figure 12).
• Judge breakdown (Figure 13): Gemini 2.5 Pro has a tool judge pass rate of about 16% and a dual-judge pass rate of 53.4%; Claude Opus 4.5 and GPT-5 execute harmful actions only 1%–3% of the time, but their dual-judge pass rate is 24%–46% with no consistent family pattern.
• Cumulative curves (Figure 4): the number of successes grows approximately linearly with the number of attempts for all models; Gemini 2.5 Pro, SecAlign 70B, and Qwen3 VL 235B show steeper inflection points between 4,000 and 9,000 attempts.
• Attacker concentration (Figure 10): the top 50 users contribute about 88% of activity; ASR has no clear correlation with the number of independent attackers (Figure 11).
• Transfer strategies: User Provided Malicious Tool Call String, Fake Syntax and Delimiters, and Fake User and Assistant Messages have transfer ASR of about 30%, and Fake Chain of Thought 27%; the authors caution that most attacks in the benchmark originate from low-robustness models (Table 5), so interpretation requires care.

Limitations and future directions noted by the authors

• Threat model is relatively permissive (Section 3.1, 5): attackers can see the full conversation and the target action; the results are an upper bound on single-turn vulnerability rather than real attack frequency; multi-turn escalating attacks are not studied.
• Missing thinking comparison (Section 5): due to various constraints, not all models could be configured with both thinking and non-thinking modes; the plan is to mandate both modes in the next competition and analyze in depth the thinking process when injection succeeds, and also to explore the defensive effect of CoT monitoring.
• Scenario realism (Section 5): ASR in the coding setting is noticeably lower, which the authors believe may be because it is based on real transcripts; the plan is to collect real transcripts in the next competition, use real MCP and tool execution more, and balance the number of scenarios across settings (currently computer use 8 vs tool use 18).
• Single evaluation is unreliable (Section 5): the competition scale meant each attack could only be evaluated once per model; the diagonal of Figure 7 shows that reruns do not necessarily succeed again, and the authors consider the ideal approach to be multiple evaluations with bootstrap aggregation.
• Uneven attention (Section 5): different models and scenarios received uneven attack investment; the next competition will reduce this variance through competition design.
• Defense layers are out of scope (Section 5): additional safety layers such as classifiers, filters, and guardrails are beyond the scope of this paper; the authors also asked vendors to disable such protections deployed by default in APIs, and to instead evaluate them in a parallel competition.
• Open questions in stealth detection (Section 5): the evaluation targets only user-visible output; whether monitoring the full CoT or using activation probes can detect stealthy attacks is an open question.

Scope of experimental coverage

• 13 models were tested (with Gemini 3 Pro and Kimi K2 non-thinking added in the transfer experiments, 15 in total), covering nine families: Anthropic, OpenAI, Google, xAI, Amazon, DeepSeek, Moonshot, Alibaba, Meta.
• There are 41 scenarios in total, with three settings: tool use 18, coding 15, computer use 8; DeepSeek V3.1, Kimi K2, and SecAlign 70B were tested on only 29 behaviors because they lack image capability.
• Data scale: 271,588 deduplicated chats, 8,648 successful attacks, 464 participants; about 130–160 independent attackers per model.
• Each attack was evaluated only once per model; the paper does not report the number of repetitions; the paper does not state which model served as the prompt judge.
• The degree of alignment between harmful actions and benign tasks was not varied; the authors explicitly leave this for future work (Section 3.1).

An agent safety evaluation paper based on a large-scale public red teaming competition, studying indirect prompt injection with a stealth requirement.

• Problem: when agents process external data, attackers can embed instructions in it so that the agent both executes a harmful action and leaves no trace in the final reply; the authors state that existing benchmarks do not simultaneously cover the stealth requirement and the coding and computer use settings.
• Method: in collaboration with UK AISI, US CAISI, and several frontier labs, 41 scenarios were designed (tool use 18, coding 15, computer use 8); an attack must pass both the tool judge (programmatic verification of the harmful action) and the prompt judge (0–10 score, threshold 7, must include a stealth criterion); the three-week competition yielded 271,588 deduplicated attempts and 8,648 successful attacks, and a benchmark of 2,679 attacks was organized.
• Main results: all 13 models are vulnerable, with ASR from 0.5% for Claude Opus 4.5 to 8.5% for Gemini 2.5 Pro (Figure 1); ASR in the tool use setting is 4.82%, higher than computer use 3.13% and coding 2.51% (Table 2).
• Transfer and strategies: attacks originating from Claude Opus 4.5 have a transfer rate of 44–81% against other models, while attacks originating from vulnerable models almost cannot transfer upward (Figure 7); 5 general attack clusters were found, the largest covering 21/41 behaviors and 9 models (Table 4).
• Authors' conclusion: robustness is determined more by model family and training method than by raw capability (GPQA and ASR correlation r = −0.31, p = 0.299, not significant); the existence of general attacks indicates that system-level and architectural defenses are needed, not just model-level robust training.
• Boundaries: the threat model is more permissive than real conditions, and the results are an upper bound on single-turn vulnerability; each attack was evaluated only once; defense layers such as guardrails were not tested and are evaluated by a parallel competition; the benchmark will be continuously updated through quarterly recurring competitions.