AgentDojo: A dynamic environment for evaluating prompt injection attacks and defenses on LLM agents

1. 01What problem does this paper attempt to solve?

   To evaluate the utility and security of tool-calling LLM agents under indirect prompt injection, the authors propose the dynamic benchmark AgentDojo.

   Core question: how to quantitatively evaluate the utility and security of tool-calling LLM agents under indirect prompt injection attacks.

   • Problem setting: AI agents combine LLM reasoning with external tool calls to complete tasks, but LLMs process text directly and lack a formal means of distinguishing instructions from data (the authors cite [44, 74]). Third-party data returned by tools can be injected with malicious instructions, and a successful attack can let the attacker call tools in the user's name, with consequences including theft of user data and execution of arbitrary code.
   • Shortcomings of existing benchmarks: the authors state that existing agent benchmarks either evaluate only single function calls or have no explicit attacks; existing prompt injection benchmarks have simple scenarios and do not involve tool calls; ToolEmu uses an LLM to simulate the environment and score, and an injection may fool both the simulator and the judge model.
   • Core design claim: static attack sets are unsuitable for evaluating prompt injection robustness—existing injection attacks are generic and not optimized for a specific model, and defenses against fixed attacks can easily be built that are not robust; therefore the benchmark should be an extensible dynamic environment supporting new tasks, new defenses, and adaptive attacks.
   • Paper output: the AgentDojo framework, whose first version contains 4 environments, 97 user tasks, 629 security test cases, and 70 tools (Table 1 note: elsewhere the main text says 74 tools), as well as multiple attack and defense paradigms from the literature.
   • Threat model (indirect prompt injection):
     • Attacker goal: make the agent complete an injection task, such as sending the user's security verification code to the attacker's email or transferring money to the attacker's account.
     • Attacker capability: control the attack placeholder in untrusted data returned by tools, such as an email in the user's inbox or a web page.
     • Attacker knowledge: variable; the paper ablates prior knowledge of the user name and model name, where correct knowledge slightly strengthens the attack and incorrect guesses weaken it (Table 2).
     • Victim system: an LLM agent that calls tools in four simulated environments: Workspace, Slack, Travel, and Banking.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: agents and tool-augmented LLMs, prompt injection attacks, prompt injection defenses, and agent and injection benchmarks; this paper is closest to InjecAgent and ToolEmu.

   AI agents and tool-augmented LLMs

   • ToolFormer [51], PAL [14], Chameleon [35], HuggingGPT [53], etc.—enable LLMs to call external tools to complete tasks; ReAct [69], chain-of-thought [60], etc. provide reasoning and planning capabilities.
   • Gorilla [43], ToolLLM [47]—large-scale API calling; several vendors provide function-calling interfaces [2, 9, 22].

   Prompt injection attacks

   • Willison [62], Goodside [17]—proposed the concept of prompt injection; Perez and Ribeiro [44] distinguish direct from indirect injection.
   • Greshake et al. [18], Kang et al. [23], Liu et al. [33]—showed that indirect injection can compromise real LLM-integrated applications; Pasquini et al. [42] learn injection triggers.

   Prompt injection defenses

   • Detection-based: Lakera ChainGuard [28], ProtectAI detector [45], LangChain's HF detection [29].
   • Training/prompting-based: StruQ [8], Instruction Hierarchy [59], Spotlighting data delimiting [19]; Willison points out that delimiters do not solve the problem [61].
   • Isolation-based: Dual LLM pattern [63], SecGPT [66]—isolate function calling from the main planning component. The authors state that none of the existing techniques is foolproof and that they may not provide guarantees for safety-critical tasks [61, 64].

   Agent and prompt injection benchmarks

   • AgentBench [32], WebArena [72], WebShop [68], Berkeley Function Calling Leaderboard [67], etc.—evaluate agent capabilities but have no explicit attacks.
   • ToolEmu [50]—uses an LLM to simulate a sandbox and score to evaluate agent risk; the authors point out that this approach is problematic for evaluating prompt injection because the injection may simultaneously hijack the judge model.
   • Tensor Trust [57], HackAPrompt [52], Liu et al. [34], Yi et al. [70]—prompt injection benchmarks, but with simple scenarios and no tool-call planning.
   • InjecAgent [71]—closest to this paper, but the authors point out that it is a simulated single-turn scenario: a single adversarial data item is fed directly to the LLM as tool output, without evaluating the model's planning.

   Baselines and comparisons: attack baselines are "ignore previous instructions" [17], InjecAgent injection [71], "TODO" injection, and the adaptive Max attack; defense baselines are Data delimiters [19], PI detector [45], Prompt sandwiching [30], Tool filter (a simplified form of the isolation mechanism [63, 66]); utility is compared with the Berkeley Tool Calling Leaderboard [67] (Figure 2).

   Authors' positioning: AgentDojo requires agents to dynamically call multiple tools in stateful, adversarial environments and uses formalized environment-state checks instead of LLM judging, distinguishing it from all the above benchmarks.

   Collapse this answer

3. 03How does the paper address this problem?

   AgentDojo builds task suites with stateful environments plus formalized success/failure checks, takes the Cartesian product of user tasks and injection tasks, and provides extensible preset attack and defense interfaces.

   Overall approach: place the agent in a stateful simulated environment to perform real tool calls, and use deterministic state-checking functions to separately determine whether the user task (utility) and the attacker goal (security) are achieved, avoiding the use of an LLM judge that could be hijacked by injection.

   Environments, tools, and tasks

   • Four environments: Workspace (email/calendar/cloud drive, 24 tools, 40 user tasks, 6 injection tasks), Slack (11, 21, 5), Travel (28, 20, 7), Banking (11, 16, 9), totaling 70 tools, 97 user tasks, and 27 injection goals (Table 1).
   • State modeling: environment state is a collection of mutable objects (e.g., inbox, calendar, cloud drive), pre-filled with fake data that is manually written or generated with GPT-4o/Claude 3 Opus assistance and manually checked; some states are marked as attack placeholders.
   • Tools: registered as Python functions (FastAPI style), with documentation added to the agent prompt, and outputs fed to the model in YAML format.
   • Task difficulty: user tasks cover medium-to-long context retrieval (up to 7,000 GPT-4 tokens of data and 4,000 tokens of tool descriptions) and tool-call chains of up to 18 calls; injection tasks range from 1 step to 20 steps, with sensitivity ranging from sending an ordinary email to forwarding a two-factor verification code.

   Success determination

   • utility function: a deterministic binary function that takes the model output and the environment state before and after execution and determines whether the user task is completed; each user task also comes with a ground truth tool-call sequence, used to place injections where the model will actually query.
   • security function: similarly determines whether the attacker goal is achieved; injection tasks also come with ground truth call sequences, for attacks that know the tool API.
   • Test cases: following Zhan et al. [71], the Cartesian product of user tasks and injection tasks for each environment yields 629 security test cases; running user tasks alone constitutes the benign utility test.

   Agent and defense interfaces

   • agent interface: only a query function needs to be implemented, receiving the user instruction, tool list, and environment state; it supports composing LLMs, tool-execution loops, injection detectors, etc. into modular pipelines (Appendix Figures 11 and 12 give components and examples).
   • Built-in defenses: Data delimiters (following Hines et al. [19], add delimiters to tool outputs and instruct the model to ignore instructions within them), PI detector (ProtectAI's BERT classifier detects each tool output and aborts on a hit), Prompt sandwiching (repeat the user instruction after each function call), Tool filter (a simplified isolation mechanism: first select the needed tools, then contact untrusted data).

   Attack interface and metrics

   • attack interface: the attack function takes an injection task and returns injection text for each attack placeholder; get_injection_candidates uses the user task's ground truth call sequence to find placeholders that will be queried (Appendix A).
   • Three metrics: Benign Utility (user task completion rate without attack); Utility Under Attack (the proportion that both completes the user task and has no side effects under attack; its complement is untargeted ASR); Targeted ASR (the proportion in which the attacker goal is achieved).
   • Attack set: for an attack set {A1,…,An}, success by any attack counts as success, modeled as an adaptive attacker that selects the best attack for each case (citing [6]).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Full evaluation of 10 LLMs plus attack/defense ablations on GPT-4o: utility without attack is at most 78.22%, GPT-4o ASR without defenses is 47.69%, and Tool filter reduces it to 6.84%.

   Experimental setup

   • Models tested (10): Claude 3 Opus, Claude 3 Sonnet, Claude 3.5 Sonnet, GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Gemini 1.5 Flash, Gemini 1.5 Pro, Llama 3 70B, Command R+; all except Llama 3 use the official API's function calling.
   • Benchmark: the full AgentDojo set of 629 security test cases and 97 user tasks; attack and defense ablations use only GPT-4o (the authors state it has the highest benign utility; Claude Opus has similar utility but access is restricted).
   • Baseline attack: "Important message" injection (found effective in preliminary experiments); the ablation compares TODO, Ignore previous, InjecAgent injections, and adaptive Max.
   • Baseline defenses: Data delimiters, PI detector (ProtectAI BERT), Prompt sandwiching (listed as Repeat prompt in the table), Tool filter.
   • Metrics and determination: Benign Utility, Utility Under Attack, Targeted/Untargeted ASR, all determined by deterministic utility/security functions based on environment state, without an LLM judge.
   • Error and cost: 95% confidence intervals are reported (statsmodels proportion_confint); running the full 629 cases on GPT-4o costs about $35, and the 97 utility cases about $4 (Appendix D); the paper does not report the number of repeated runs.

   Main results

   According to Table 3 (Important message attack, 95% confidence intervals omitted):

   Swipe horizontally to read the table

   Model	Benign utility	Utility under attack	Targeted ASR
   Claude 3.5 Sonnet	78.22	51.19	33.86
   GPT-4o	69.00	50.08	47.69
   Claude 3 Opus	66.61	52.46	11.29
   GPT-4 Turbo	63.43	54.05	28.62
   Claude 3 Sonnet	53.10	33.23	26.71
   Gemini 1.5 Pro	45.63	28.93	25.60
   Gemini 1.5 Flash	36.09	34.18	12.24
   Llama 3 70b	34.50	18.28	20.03
   GPT-3.5 Turbo	33.86	34.66	8.43
   Command-R+	25.44	25.12	0.95

   • The authors state that more capable models tend to be more vulnerable to attack, regarding this as a form of inverse scaling [38]; they explain that low-utility models often fail to correctly execute the attacker goal even when the injection succeeds.
   • The authors state that benign utility and utility under attack are strongly correlated, with most models losing 10%–25% absolute utility under attack (Figure 6b).
   • The authors state that in the benign setting the most capable is Claude 3.5 Sonnet, followed by GPT-4o, and that the former has a better utility-security trade-off.

   Attack ablation (GPT-4o)

   • Injection wording matters greatly: according to Table 4, targeted ASR is TODO 3.66%, Ignore previous 5.41%, InjecAgent 5.72%, and Important message 57.7%, respectively; adaptive Max has an untargeted ASR of 68.36% (the main text states Max raises the success rate by a further 10%).
   • Attacker knowledge: according to Table 2, the baseline is 45.8%; when both the username and model name are correct it is 47.7% (+1.9%); guessing the wrong user or model name lowers it to 23.2% and 23.7%, respectively. The authors state that the benefit of prior knowledge is limited, and wrong guesses weaken the attack.
   • Injection position: the authors state that placing the injection at the end of the tool-returned text is most effective, with an average success rate of up to 70% for GPT-4o (Figure 21a); the Slack suite reaches an ASR of 92%, which the authors suggest may be related to the attacker controlling a larger proportion of tool outputs (Figure 21b); for Travel suite injection task 6 the success rate is 0%, and the authors find that the model often completes only one of the two unrelated malicious subtasks.

   Defense evaluation (GPT-4o)

   According to Table 5 (against the strongest attack):

   Swipe horizontally to read the table

   Defense	Benign utility	Utility w. attack	Targeted ASR
   No defense	69.0%	50.01%	57.69%
   Delimiting	72.66%	55.64%	41.65%
   PI detector	41.49%	21.14%	7.95%
   Repeat prompt	85.53%	67.25%	27.82%
   Tool filter	73.13%	56.28%	6.84%

   • The authors state that several defenses instead increase benign utility (presumably because they emphasize the original instruction more); the PI detector has many false positives and harms utility.
   • Tool filter is most effective: ASR drops to 6.84% (elsewhere in the main text written as 7.5%); the authors note that it fails when tools cannot be planned in advance, or when the tools needed to solve the task suffice to carry out the attack (the latter accounts for 17% of test cases).
   • The authors believe Repeat prompt is effective against current attacks but unlikely to resist adaptive attacks (e.g., injections instructing the model to ignore future instructions).

   Other ablations and analyses

   • Difficulty comparison: the authors state that AgentDojo benign tasks are harder than the Berkeley Tool Calling Leaderboard (Figure 2; Llama 3 70B used a different prompt).
   • DoS attacks: untargeted attacks such as requests to stop, profanity, and Captcha are all less able to derail the model than targeted attacks with malicious goals (Figure 20).
   • Cost: the full security cases on GPT-4o cost about $35 (Appendix D).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list five extension directions (stronger attacks and defenses, automated task generation, harder tasks, multimodality, injection constraints); experiments cover 10 models, and defenses are tested only on GPT-4o.

   Limitations and future directions noted by the authors

   • Attacks and defenses are relatively simple (Section 5): the current attacks and defenses used are relatively simple; future work could add more complex defenses (e.g., isolated LLM [63, 66]) and attacks [15].
   • Tasks are manually specified (Section 5): extending to more tasks and attack goals may require automating the current process of manually specifying tasks and utility criteria, while not sacrificing evaluation reliability.
   • Harder tasks are needed (Section 5): tasks that tool-selection defenses (or more complex isolation mechanisms) cannot directly solve are especially worth adding.
   • Multimodality is not supported (Section 5): it could be extended to multimodal agents handling both text and images, expanding the range of tasks and attacks.
   • Injections are unconstrained (Section 5): adding constraints such as length or format to prompt injections could better characterize real attacker capabilities.
   • Failure scenarios of the Tool filter (Section 4.3): it fails when the tool list cannot be planned in advance, or when the tools needed to solve the task suffice to carry out the attack (17% of use cases); AgentDojo does not yet cover scenarios where a user gives multiple tasks over time without resetting context.

   Scope of experimental coverage

   • 10 models were tested (Table 3); the main attack is Important message injection; the attack-wording ablation includes 4 injections plus adaptive Max (Table 4).
   • Defense evaluation was conducted only on GPT-4o, including 4 defenses (Table 5); the attacker-knowledge ablation also used only GPT-4o (Table 2).
   • Each model was run on the full 629 security test cases and 97 user tasks; the paper reports 95% confidence intervals but does not report the number of repeated runs.
   • The injection-position analysis targets GPT-4o (Figure 21a); DoS-type untargeted attacks were compared across four suites (Figure 20).
   • The benign difficulty comparison uses reported results from the Berkeley Tool Calling Leaderboard; Llama 3 70B used a different prompt (Figure 2 footnote).

   Collapse this answer

6. 06Summarize the main content of the paper

   AgentDojo: a dynamic prompt injection attack-defense benchmark containing 97 tasks and 629 security cases; existing models are insufficient in both utility and robustness, and both attack and defense remain challenging.

   AgentDojo is a dynamic benchmark framework for evaluating the utility and security of tool-calling LLM agents under indirect prompt injection, designed to address the problems that existing agent benchmarks have no explicit attacks, injection benchmarks have no realistic multi-turn tool calling, and LLM-simulated judges are easily hijacked by injection.

   • Problem addressed: attackers hide malicious instructions in untrusted data returned by tools, hijacking the agent to perform malicious tasks in the user's name (e.g., stealing data, transferring money); how to reliably measure progress in attack and defense under this threat.
   • Method highlights: four stateful environments (Workspace, Slack, Travel, Banking), 70 tools, 97 user tasks, and 27 injection goals combined via Cartesian product to yield 629 security test cases; deterministic utility/security functions check environment state to determine success or failure; attacks, defenses, and agents are all extensible interfaces.
   • Main results: for 10 models without attack, benign utility is at most 78.22% (Claude 3.5 Sonnet, Table 3); GPT-4o without defenses has a targeted ASR of 47.69%, while Command R+ is only 0.95% (Table 3); the authors state that more capable models are more vulnerable to attack.
   • Attack and defense findings: injection wording matters greatly; the Important message attack has an ASR of 57.7%, far higher than the 5.72% of the InjecAgent injection (Table 4); placing the injection at the end of the tool output is most effective (up to 70%); the Tool filter defense reduces GPT-4o's ASR to 6.84%, and the PI detector to 7.95% but with benign utility dropping to 41.49% (Table 5).
   • Authors' conclusion: AgentDojo poses a challenge to both attackers and defenders—existing attacks can break some but not all security properties, and existing defenses are all imperfect; the authors believe that resisting stronger adaptive attacks may require a breakthrough in LLMs' ability to distinguish instructions from data, and hope AgentDojo will serve as a continuously updated benchmark to measure progress.

   Collapse this answer

Related papers