InjecAgent: A benchmark evaluation of indirect prompt injection for tool-integrated large language model agents

1. 01What problem does this paper attempt to solve?

   Tool-integrated LLM agents face the risk of indirect prompt injection; the authors propose the benchmark InjecAgent to systematically evaluate their vulnerability.

   Core question: how to systematically evaluate the vulnerability of tool-integrated LLM agents to indirect prompt injection (IPI) attacks.

   • Problem setting: LLM agents can invoke tools and access external content (emails, comments, websites, etc.); attackers can embed malicious instructions in this content to manipulate the agent into performing harmful actions such as transferring money, controlling smart home devices, or stealing private data. The authors argue that such attacks have a low technical barrier and severe consequences, making benchmark evaluation imperative.
   • Existing gaps: The authors state that previously there was no comprehensive analysis of IPI attacks on LLM agents; parallel work (Yi et al., 2023) covers only 5 simulated application scenarios.
   • Paper proposes: The benchmark InjecAgent, containing 1,054 test cases, covering 17 user tools and 62 attacker tools, with attack intents divided into two categories: direct harm and data theft; and evaluates 30 LLM agents.
   • Threat model:
     • Attacker goal: Manipulate the agent into executing attacker instructions Ia, invoking tools to cause direct harm to the user (financial, physical, data security) or to exfiltrate stolen data.
     • Attacker capability: Inject malicious content into external content retrieved by the agent (e.g., modifiable comment fields).
     • Victim system: An LLM agent L equipped with a tool set T, including ReAct-prompted agents and tool-calling fine-tuned agents.
     • Success determination: The agent executes Ia (direct harm executes a harmful tool; data theft requires completing both the extraction and exfiltration steps).

   Collapse this answer

2. 02What related research exists?

   Related work is divided into three groups: LLM agents, prompt injection attacks, and defenses; the authors state that this paper is the first formalization and benchmark of IPI on tool-integrated agents.

   LLM agents

   • ReAct (Yao et al., 2022), MindAct (Deng et al., 2023), AutoGPT (Significant Gravitas) — use in-context learning to let LLMs use tools.
   • Toolformer (Schick et al., 2023), Gorilla (Patil et al., 2023), ToolLLM (Qin et al., 2023), and OpenAI function-calling models (OpenAI, 2023b) — fine-tune LLMs with function-calling examples. This paper evaluates both types of agents.
   • Ruan et al. (2023) — proposed 330 tools and 36 toolkits and used an LM sandbox to identify agent risks; this paper's tool set and ReAct prompt are both based on that work.

   Prompt injection attacks

   • DPI (Perez and Ribeiro, 2022; Selvi, 2022; Kang et al., 2023; Liu et al., 2023; Toyer et al., 2023; Yu et al., 2023) — a malicious user directly injects a prompt into the model input, with the goal of goal hijacking or prompt leaking.
   • IPI (Abdelnabi et al., 2023; Yi et al., 2023) — the attacker injects a prompt into external content that the LLM will retrieve; there are reports of instances of OpenAI plugins being attacked (Rehberger, 2023a,b,c; Greshake, 2023, etc.).
   • Yi et al. (2023) — parallel work, evaluating IPI on simulated LLM-integrated applications, containing only 5 scenario types: email QA, web QA, table QA, summarization, and code QA. The paper points out that its scenarios are limited, whereas this paper targets real tool-integrated agents, covering 17 user cases and 62 attacker cases.

   Prompt injection defenses

   • Black-box defenses: adding extra attack-aware prompts (Toyer et al., 2023; Yi et al., 2023), adding special delimiters to external content (Yi et al., 2023).
   • White-box defenses: fine-tuning with attack examples (Yi et al., 2023; Piet et al., 2023), encoding instruction words (Suo, 2024), separating instruction and data into dual channels (Chen et al., 2024). The paper points out that these defenses mainly target simple scenarios where instructions and data are directly concatenated, and their effectiveness in tool-integrated agents remains to be explored.

   Positioning: the authors state that this paper is the first formalization of IPI attacks on tool-integrated LLM agents, and proposes a benchmark covering multiple domains as an evaluation standard; this paper does not implement defense baselines, and the comparison targets are the ASR of 30 agents on the benchmark.

   Collapse this answer

3. 03How does the paper address this problem?

   GPT-4 is used to assist in generating 17 user cases and 62 attacker cases, combined into 1,054 test cases, with ASR used to evaluate agents.

   Overall approach: first formalize the IPI attack, then use GPT-4 to assist in generating test cases (with manual revision), combine them into the benchmark InjecAgent, and finally evaluate agents by attack success rate.

   Problem Formalization

   • Agent L is equipped with tool set T; the user issues instruction Iu, requesting external content via tool Tu; the response RTu of Tu contains attacker instruction Ia; the attacker intends L to call another tool Ta to harm the user.
   • Attack success criterion: the agent executes Ia and brings adverse consequences to the user.

   Test Case Generation (three steps)

   • User case generation: manually review the 330 tools of Ruan et al. (2023), and select 17 tools whose responses contain attacker-modifiable external content as user tools; have GPT-4 generate one user case for each tool, containing the user tool, user instruction, tool parameters, and a tool response template with a "<Attacker Instruction>" placeholder; the placeholder is placed in an attacker-modifiable field (e.g., the content of a review). After generation, manual revision ensures the template is consistent with the instruction, totaling 17 user cases.
   • Attacker case generation: randomly sample 9 tools each time, have GPT-4 select a tool that can cause direct harm or a tool that can steal private data, and generate attacker instruction Ia. About 30% of generated instructions lack the parameters required by the tool, and manual revision fills them in. Finally, 30 direct harm cases and 32 data theft cases (Table 2).
   • Test case synthesis: 17 × 62 = 1,054 test cases; the enhanced setting replaces the placeholder with the concatenation of a fixed hacking prompt p and Ia (p is a fixed prefix requiring ignoring previous instructions), with 1,054 cases per setting.

   Evaluation Protocol

   • Assume the agent has correctly executed the user tool Tu and obtained RTu, and evaluate its subsequent actions starting from this point.
   • Direct harm attack: success if the next step executes the harmful tool Ta. Data theft attack: first use GPT-4 to simulate the response RTa of the data extraction tool, then judge whether the agent sends the data to the attacker; both steps must be completed for success.
   • Metrics: primary metric ASR-valid (proportion of attack successes among valid outputs), additionally reporting ASR-all (proportion among all outputs); outputs that do not conform to the ReAct format, repeatedly execute the user tool, or have no substantive content are recorded as invalid and excluded.
   • Appendix E gives the complete text of the agent ReAct prompt, the case generation prompt, and the data theft response simulation prompt.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluating 30 agents: ReAct-prompted GPT-4 base ASR 24%, enhanced 47%, fine-tuned GPT-4 only 7.1%; and analyzing influencing factors.

   Experimental Setup

   • Agents under test: 30, in two categories — ReAct-prompted agents (Qwen series 0.5B–72B, Llama2-7B/13B/70B, Mistral-7B, Mixtral-8x7B, OpenOrca-Mistral, OpenHermes-2/2.5-Mistral, Snorkel-Mistral, Nous-Mixtral-DPO/SFT, MythoMax-13b, WizardLM-13B, Platypus2-70B, StripedHyena-7B, Capybara-7B, Yi-34B, Nous-Yi-34B, Nous-Llama2-13b, Claude-2, GPT-3.5, GPT-4) and tool-calling fine-tuned GPT-3.5, GPT-4.
   • Benchmark: InjecAgent, 1,054 test cases per setting (17 user cases × 62 attacker cases), divided into base and enhanced settings.
   • Metrics: ASR-valid (primary), ASR-all, valid rate; data theft is reported in two steps, S1 (data extraction) and S2 (data transmission).
   • Implementation details: temperature set to 0; open-source models called via the Together AI API; GPT-4 is gpt-4-0613, the prompted GPT-3.5 is gpt-3.5-turbo-0613, and the fine-tuned versions are gpt-3.5-turbo-1106 and gpt-4-0613; evaluation takes about 1 hour per model per setting (Appendix B.1).
   • Scope of the main results table: Table 3 lists only the 20 agents with valid rate greater than 50%; the rest are in Appendix Table 7.

   Main Results

   ASR-valid (%, according to Table 3, base / enhanced setting):

   Swipe horizontally to read the table

   Agent	Direct harm	Data theft total	Total
   GPT-4 (ReAct)	14.7 / 33.3	31.9 / 59.9	23.6 / 47.0
   GPT-3.5 (ReAct)	18.8 / 31.4	28.8 / 47.9	23.7 / 39.8
   Claude-2 (ReAct)	7.5 / 4.4	14.8 / 2.5	11.4 / 3.4
   Llama2-70B (ReAct)	91.9 / 94.7	80.4 / 81.7	86.9 / 88.2
   Capybara-7B (ReAct)	34.0 / 84.2	36.1 / 82.7	34.9 / 83.5
   Qwen-72B (ReAct)	8.7 / 27.0	37.1 / 49.6	23.2 / 38.3
   GPT-4 (fine-tuned)	2.9 / 3.9	10.1 / 10.1	6.6 / 7.1
   GPT-3.5 (fine-tuned)	1.8 / 5.1	5.7 / 11.4	3.8 / 8.4

   (Table 3 has 20 rows in total; omitted here are Qwen-1.8B, Mistral-7B, OpenOrca-Mistral, OpenHermes-2.5/2-Mistral, Mixtral-8x7B, Nous-Mixtral-DPO/SFT, MythoMax-13b, WizardLM-13B, Platypus2-70B, Nous-Llama2-13b; among them OpenOrca-Mistral is the lowest, base 3.4%, enhanced 10.5%.)

   • The authors state that prompted agents are generally vulnerable, with Llama2-70B's ASR exceeding 80% in both settings; fine-tuned GPT-3.5/GPT-4 are more resilient.
   • All agents except Claude-2 have higher ASR in the enhanced setting; the authors believe the hacking prompt can amplify the attack effect.
   • In data theft, S2 (transmission) has the highest success rate, with S2 for fine-tuned GPT-3.5 and GPT-4 both at 100%; the authors believe transmitting data to the attacker is relatively easy and is a key point for security hardening.

   Analysis of Influencing Factors (Section 4)

   • User cases vs attacker cases: Cramér's V is used to measure association strength (Table 4); in direct harm attacks, user cases 0.28, attacker cases 0.18; in data theft attacks, 0.31 vs 0.20, with p-values all less than 0.01; Wilcoxon signed-rank test p less than 0.0005. The authors state that user cases are more strongly associated with attack success.
   • Content freedom: GPT-4 divides the 17 user cases into high (12) and low (5) content freedom groups (majority vote of three generations); Figure 2 shows the high-freedom group has higher ASR, with Wilcoxon test p less than 0.0001. The authors' explanation is: high-freedom placeholders make it easier for attack instructions to blend into normal content.
   • Sensitivity analysis: a keyword list is used to automatically count the proportion of outputs in which attack instructions are identified (sensitivity rate); Figure 3 shows that under the enhanced setting, the sensitivity of each agent is consistently higher. Claude-2 is the only agent whose ASR is instead lower in the enhanced setting; the authors state that its sensitivity and conversion rate (success converted to failure) are both notably higher, and offer the hypothesis that the fixed hacking prompt may simultaneously trigger the agent's vigilance.

   Other Ablations and Analyses

   • valid rate (Table 6): fine-tuned GPT-4 base 99.9%, enhanced 100.0%; Figure 4 shows valid rate is positively correlated with Arena Elo score.
   • Attack subcategories (Table 9, ASR-all): the ASR of each attack type (financial, physical, data security, etc.) appears to vary randomly across agents.
   • Response quality (Table 10): GPT-4's relevance and clarity scores (0–2) for failure cases; Claude-2 has the highest clarity (base 1.86, enhanced 1.92), and the authors state it often reminds the user that the content is abnormal.
   • Short ReAct prompt (Table 11/12): after switching to the LangChain short prompt, GPT-4 base ASR-valid is 22.2%, enhanced 46.1%; the Figure 6 heatmap shows the long prompt has a higher success rate, and the enhanced setting is higher.
   • Negative result: Claude-2's enhanced setting ASR-valid drops from 11.4% to 3.4%, contrary to the trend of the other agents (Table 3).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list four limitations: a single hacking prompt, instructions not mixed with benign content, only single-turn two-step scenarios, and insufficient research on fine-tuned agents.

   Limitations Pointed Out by the Authors and Future Directions

   • The enhanced setting uses only a single fixed hacking prompt (Section 8): changes in prompt text may affect attack results, and a fixed prompt is easy for developers to filter; exploring more prompts and dynamic enhancement methods is a future direction.
   • Limited examination of variation in attacker instructions (Section 8): it is assumed that external content contains only attack instructions; the effect of mixing malicious instructions with benign content (e.g., embedded in a doctor's review) was not explored.
   • More complex scenarios not examined (Section 8): only single-turn scenarios are considered, with attacker instructions at most two steps; real scenarios may be more complex.
   • Insufficient study of fine-tuned agents (Section 8): only two fine-tuned agents were studied; the authors believe more research on fine-tuning LLMs in tool-use scenarios is needed.
   • Defense exploration (Section 5): the implementation and effectiveness of existing prompt injection defenses in tool-integrated agents remain to be explored in the future.

   Scope of Experimental Coverage

   • The main results (Table 3) cover 20 agents with valid rate above 50%, with the remaining 10 in Appendix Table 7; the fine-tuned agents are only GPT-3.5 and GPT-4.
   • Each setting is evaluated once on 1,054 test cases, with temperature 0; the paper does not report multiple repetitions or variance.
   • Attack success is judged based on tool calls in the parsed output, and the tool response for the second step of data theft is simulated with GPT-4; the sensitivity rate is automatically counted with a keyword list, and the paper does not report its consistency with manual judgment.
   • The attack payload is a fixed instruction generated by GPT-4 plus manual revision, and the enhanced setting only adds one fixed hacking prompt; the paper notes that optimized attack methods such as training adversarial strings were not attempted (Section 3.2).
   • The content freedom classification is done by majority vote of three GPT-4 generations, with no manual verification reported.

   Collapse this answer

6. 06Summarize the main content of the paper

   The InjecAgent benchmark evaluates the IPI vulnerability of 30 tool-integrated agents: prompted agents are generally vulnerable, while fine-tuned agents are more resilient.

   A benchmark paper: on indirect prompt injection (IPI) attacks against tool-integrated LLM agents, the authors propose InjecAgent and systematically evaluate 30 agents.

   • Problem: Attackers embed malicious instructions in external content retrieved by the agent, which can manipulate the agent into transferring money, controlling home devices, or stealing data; previously there was no systematic evaluation.
   • Method: Based on 330 tools, 17 user tools were selected; GPT-4 was used to generate user cases and 62 attacker cases (30 direct harm, 32 data theft), combined into 1,054 test cases; there is also an enhanced setting with a fixed hacking prompt superimposed; ASR-valid is the primary metric.
   • Main results:
     • ReAct-prompted GPT-4 base setting ASR 24%, rising to 47% in the enhanced setting (Table 3 is 23.6%/47.0%).
     • Fine-tuned GPT-4 is notably more robust, with an enhanced-setting ASR of only 7.1% (Table 3).
     • ReAct-prompted Llama2-70B has ASR above 80% in both settings (86.9%/88.2%).
     • The second step of data theft (data transmission) has the highest success rate, reaching 100% for both fine-tuned GPT-3.5/GPT-4.
     • All agents except Claude-2 have higher ASR in the enhanced setting; Claude-2 instead drops from 11.4% to 3.4%.
   • Analysis conclusions (per the authors): User cases are more strongly associated with attack success than attacker cases (Cramér's V 0.28/0.31 vs 0.18/0.20); user cases with placeholders allowing high content freedom are more vulnerable.
   • Authors' implications: Such attacks have a low barrier and severe consequences; the findings raise questions about the widespread deployment of LLM agents and emphasize the urgency of defense strategies.

   Collapse this answer

Related papers