AgentVigil: A General Black-Box Red-Teaming Method for Indirect Prompt Injection in LLM Agents

1. 01What problem does this paper attempt to solve?

   How to conduct general indirect prompt injection risk assessment against black-box LLM agents, given that existing methods are either manually constructed or applicable only to specific agents.

   Core problem: how to perform generic indirect prompt injection risk assessment for black-box LLM agents.

   • Problem setting and importance: LLM agents combine external tools to complete complex tasks, and attackers can insert malicious instructions into the external data sources the agent interacts with, causing it to deviate from the user's task and execute the attacker's intent; the authors consider this to have serious consequences.
   • Shortcomings of existing methods: The authors state that existing red-teaming methods either manually construct attack instructions or are designed for only one type of agent, and cannot serve as a generic method to evaluate the indirect prompt injection risk of LLM agents; model-level large-scale evaluation methods cannot be directly applied to agents due to differences in system components and mechanisms.
   • Core challenges: Real agents are black-box systems, and the only feedback is a sparse binary signal of attack success or failure; user tasks are diverse and execution behavior is dynamic; agent architectures are complex and varied.
   • Paper's approach: Drawing on traditional software fuzzing, the paper proposes the AGENT VIGIL framework, which iteratively optimizes injection prompts through a high-quality initial seed library, adaptive seed scoring, and MCTS-based seed selection. The authors state this is the first automated indirect prompt injection evaluation method for black-box LLM agents.
   • Threat model (Section 3):
     • Attacker's goal: Mislead the agent into completing a goal specified by the attacker rather than the user's intent, such as visiting a phishing website, downloading malicious files, or sending private information.
     • Knowledge and capabilities: blackbox setting; the attacker cannot access the LLM internals or the agent architecture; the attacker can interact with the agent like a normal user, but can only manipulate external data sources (such as products on a shopping website or calendar events).
     • Feedback: For each user task, only binary feedback of attack success or failure can be observed (such as checking the inbox).
     • Victim system: An LLM agent that uses external data sources to complete tasks; the user is assumed to be benign.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into three categories: LLM agents, prompt injection attacks and defenses; this paper uses the manually constructed attacks of AgentDojo and VWA-adv as baselines.

   LLM agents

   • Web agents (Nakano et al., 2021; Deng et al., 2024, etc.): assist humans in interacting with web pages.
   • Coding agents (Le et al., 2022; Gao et al., 2023, etc.): assist in code writing, completion and debugging.
   • Personal assistants (Schick et al., 2024; Qin et al., 2023, etc.): help with everyday tasks such as calendars and email.

   Prompt injection attacks

   • Manual attacks: e.g., manipulating context with escape characters (Willison, 2022), instructing the LLM to ignore prior context (Perez & Ribeiro, 2022; Schulhoff et al., 2023), simulating task completion (Willison, 2023), as well as injecting malicious web content targeting specific agent types (Wu et al., 2024a; Liao et al., 2024; Xu et al., 2024) or manipulating interface elements (Zhang et al., 2024). The paper notes that such attacks require expertise and have unstable success rates.
   • Automated attacks: AgentPoison (Chen et al., 2024d) and VWA-adv (Wu et al., 2024b) use gradient-based methods and require white-box access to target components; GPTFuzzer (Yu et al., 2023) and RLBreaker (Chen et al., 2024c) target direct prompt injection and require detailed feedback, with limited applicability in complex real-world agents.

   Prompt injection defenses

   • Training-dependent methods: e.g., StruQ (Chen et al., 2024a), the ProtectAI detector (ProtectAI, 2024), Llama Guard (Inan et al., 2023), etc.; the paper notes that they require substantial computational resources and frequent updates, and may degrade model performance due to over-regularization.
   • Training-free methods: input delimiters (Hines et al., 2024), prompt repetition (lea, 2023), response consistency checking (Liu et al., 2024), etc.; the paper notes that these mainly detect attacks after execution; tool access validation (Debenedetti et al., 2024) restricts the agent to pre-approved tools, and the paper notes that it limits functionality and remains vulnerable to attacks within the tool set.

   Baselines and benchmarks: the baseline attacks are the manually constructed adversarial prompts proposed by AgentDojo (Debenedetti et al., 2024) and VWA-adv (Wu et al., 2024b); Appendix B.2 additionally adds two baselines, OpenPromptInjection (Liu et al., 2024) and InjecAgent (Zhan et al., 2024).

   The authors position the key difference between this paper and GPTFuzzer as follows: GPTFuzzer assumes full control of the input and is a single-turn setting, targeting direct prompt injection to jailbreak LLMs; this paper targets indirect prompt injection in multi-step agents, where the attacker can only influence external content, which the authors state makes it the first automated attack framework for black-box LLM agents in real-world scenarios.

   Collapse this answer

3. 03How does the paper address this problem?

   AGENT VIGIL uses a fuzzing process: initial seed pool + five mutations + hybrid scoring of success rate and coverage + MCTS seed selection.

   The overall idea draws on software fuzzing and proposes AGENT VIGIL: given a target agent and a seed set of attack instructions, heuristically select seeds, mutate them and input them to the agent, score based on the output and add them to the seed pool, iterating (Figure 2).

   Initial corpus construction (Section 4.2)

   • Sources: adversarial prompt templates collected from human heuristic experience, online resources and existing prompt injection research (Debenedetti et al., 2024; Liu et al., 2024).
   • Template design: with placeholders to accommodate different variables (the LLM used, the user task, the attacker goal), covering strategies such as role-playing, delimiter-based attacks and prompt obfuscation.

   Mutation design (Section 4.3)

   • Five mutation methods: Shorten (compression), Expand (add context), Rephrase (rewrite while preserving semantics), Crossover (fuse two parent seeds), GenerateSimilar (generate seeds with similar style but different content), randomly selected at each iteration.
   • Auxiliary models: prompt templates are used to have an auxiliary LLM perform mutations; the authors state that these basic mutations require only moderately capable models, such as Llama-3-8B and GPT-4o-mini; the experiments actually use GPT-4o-mini.

   Seed scoring (Section 4.4)

   • Hybrid scoring: the final score is a weighted sum of attack success rate (ASR) and a coverage bonus, the latter rewarding seeds that make previously failed tasks newly succeed, with the formula seed_score = ASR + C · coverage_bonus / num_questions (Algorithm 1), where C is the coverage factor.
   • Purpose: to provide intermediate feedback beyond binary success/failure feedback, balancing exploitation of known successful patterns and exploration of new attack patterns.

   MCTS-based seed selection (Section 4.5)

   • Tree structure: a tree recording mutation history and seed relationships is maintained, using the UCB1 algorithm to balance exploiting high-scoring seeds and exploring new variants; the UCB score combines a node's empirical performance with an exploration bonus that grows with the logarithm of total visits and decays inversely with the node's visit count (Algorithm 3).
   • Reason for choosing UCB1 over UCT: the authors state that evaluating each new seed is computationally expensive, and UCB1 balances exploration and exploitation without deep tree expansion.
   • Update and selection: after evaluation, visit counts are propagated along the ancestor chain (Algorithm 2); each time the 1 or 2 seeds with the highest UCB score are selected for mutation.

   Success determination

   • Attack success is determined by the benchmark environment: AgentDojo provides formal evaluation criteria that check the environment state to determine whether the injection task succeeded; VWA-adv returns whether the adversarial task succeeded. This binary result is passed to AGENT VIGIL as the feedback signal.

   Collapse this answer

4. 04What experiments did the paper conduct?

   On AgentDojo and VWA-adv, ASR reaches 71%/70%, nearly twice the baseline; it transfers to unseen tasks and models; it is ineffective against Claude-3.5-Sonnet.

   Experimental setup

   • Models tested: o3-mini, GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet on AgentDojo; GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, Gemini-2-flash-exp on VWA-adv; Appendix B.1 adds QwQ-32B and o3-mini-2025-01-31 (specific checkpoints in Appendix A).
   • Benchmarks and scale: AgentDojo's adversarial tasks are randomly split into a fuzzing set of 142 and a test set of 173; VWA-adv is split into a fuzzing set of 99 and a test set of 100, using only text trigger tasks.
   • Fuzzing configuration: the auxiliary mutation model is GPT-4o-mini; on AgentDojo, each iteration generates 3 mutations for 10 rounds in total, and each new seed is evaluated on a randomly sampled quarter of the tasks in each suite; on VWA-adv, each iteration generates 10 mutations for 10 rounds in total.
   • Baselines: the manually constructed adversarial prompts proposed by each of the two benchmarks; Appendix B.2 adds OpenPromptInjection and InjecAgent.
   • Metric and determination: attack success rate (ASR), formally determined by the benchmark framework based on environment state; the transfer experiments select the 5 highest-scoring seeds, and the success rate is computed over the union of these prompts.
   • Defense evaluation: pi_detector, repeat, delimit for AgentDojo (on the fuzzing set, o3-mini); safety, paraphrase, combined for VWA-adv (on the fuzzing set, GPT-4o).

   Main results

   Table 1: Transfer attack success rates (fuzzing performed on AgentDojo against o3-mini and on VWA-adv against GPT-4o)

   Swipe horizontally to read the table

   Benchmark	Task set	Attack	o3-mini	GPT-4o	GPT-4o-mini	Claude-3.5-Sonnet	Gemini-2-flash-exp
   AgentDojo	Fuzzing	Manual baseline	0.38	0.22	0.28	0.12	-
   AgentDojo	Fuzzing	AGENT VIGIL	0.71	0.22	0.49	0.03	-
   AgentDojo	Test	Manual baseline	0.34	0.25	0.28	0.08	-
   AgentDojo	Test	AGENT VIGIL	0.65	0.19	0.43	0.04	-
   VWA-adv	Fuzzing	Manual baseline	-	0.36	0.08	0.47	0.49
   VWA-adv	Fuzzing	AGENT VIGIL	-	0.60	0.47	0.31	0.67
   VWA-adv	Test	Manual baseline	-	0.44	0.29	0.51	0.50
   VWA-adv	Test	AGENT VIGIL	-	0.59	0.54	0.42	0.67

   • The authors state that the generated prompts transfer effectively to unseen tasks, nearly twice the baseline performance; transfer to GPT-4o-mini is relatively good, while transfer to GPT-4o and Claude-3.5-Sonnet is relatively poor.
   • Exception: the prompts of both the baseline and AGENT VIGIL are essentially ineffective against Claude-3.5-Sonnet (0.03–0.12 on AgentDojo), and the authors state that it exhibits strong robustness to complex adversarial prompts; on VWA-adv, AGENT VIGIL's ASR against Claude (0.31/0.42) is lower than the baseline (0.47/0.51), and after manual inspection the authors speculate that Claude may be more susceptible to simple prompt attacks.
   • The authors state that AGENT VIGIL's success rates of about 50% and 60% on GPT-4o-mini and GPT-4o indicate that the instruction hierarchy defense mechanism is not sufficiently effective.

   Evaluation against defenses

   • AgentDojo (Table 2, o3-mini, fuzzing set): without defense, AGENT VIGIL 0.71 vs. baseline 0.38; under pi_detector, 0.25 vs. 0.13; under repeat, 0.12 vs. 0.21 (lower than the baseline); under delimit, 0.49 vs. 0.36. The authors state that the success rates of both attacks decrease under defenses but remain high, indicating that these defenses are insufficient; delimit is weaker than pi_detector and repeat.
   • VWA-adv (Table 3, GPT-4o, fuzzing set): without defense, 0.60 vs. baseline 0.36; under safety, 0.29 vs. 0.34; under paraphrase, 0.33 vs. 0.27; under combined, 0.27 vs. 0.30. The authors state that with defenses introduced, AGENT VIGIL's performance decreases and matches the baseline, possibly because complex attack prompts struggle against defenses in VWA-adv's limited context; the combined defense does not further reduce the success rate compared with single defenses, and the authors believe that some attack prompts can bypass multiple defenses simultaneously.

   Ablations and coverage curves

   • Ablations (Figure 3, AgentDojo): after removing the initial corpus (replaced with baseline manual prompts), the success rate stalls after about four iterations; after removing adaptive scoring and MCTS selection (replaced with uniform random selection), improvement slows. The authors state that all three components are critical for sustained improvement.
   • Coverage curves: Figure 3 shows that on AgentDojo the average coverage rises from about 0.2 to above about 0.4 at 10 steps, with the full version consistently higher than the two ablated versions; Figure 4 shows that on VWA-adv coverage rises from about 0.55 to about 0.78 (10 steps).

   Other ablations and analyses

   • Appendix B.1/B.2 (Table 4, AgentDojo): on o3-mini-2025-01-31, Fuzzing/Test ASR is 0.73/0.76, and on QwQ-32B it is 0.72/0.74; the baselines AgentDojo 0.47/0.49 and 0.45/0.47, OpenPromptInjection 0.38/0.39 and 0.20/0.20, InjecAgent 0.15/0.11 and 0.14/0.12.
   • Appendix B.1 practicality: AgentDojo benign task success rates are Llama3.3-70B-Instruct 42%, Qwen2.5-72B-Instruct 54%, QwQ-32B 74%, o3-mini 79%, on which basis QwQ-32B was chosen for further experiments.
   • Appendix B.3 scenario breakdown (Table 5): on AgentDojo's four suites (Slack, Workspace, Travel, Banking), AGENT VIGIL is higher than the baseline in all cases, e.g., for o3-mini on the Slack test set, 0.97 vs. 0.70; on VWA-adv, illusioning 0.82/0.76 vs. baseline 0.51/0.62, goal misdirection 0.58/0.42 vs. 0.00/0.20.
   • Real-world case (Section 5.4): on a locally deployed WebArena shopping website (based on magento2), prompts generated in Section 5.2 were injected into reviews using an ordinary user account, successfully misleading the default WebArena agent into visiting arbitrary URLs (including phishing sites, malicious downloads and privacy-leaking queries).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors do not have a dedicated limitations section; the experimental discussion notes performance degradation under VWA-adv defense and poor transfer to Claude; coverage is limited to two benchmarks.

   Limitations noted by the authors and future directions

   • The paper has no dedicated Limitations or Future Work section. Shortcomings acknowledged by the authors in the experimental discussion include: on VWA-adv, with defenses introduced, AGENT VIGIL's performance decreases and matches the baseline, which the authors believe may be because complex attack prompts struggle against defenses in VWA-adv's limited context (Section 5.2).
   • Transfer to Claude-3.5-Sonnet is poor: on AgentDojo both the baseline and AGENT VIGIL are ineffective, and on VWA-adv AGENT VIGIL is lower than the baseline; the authors speculate that Claude may be more susceptible to simple prompt attacks (Sections 5.1, 5.2).
   • The Impact Statement states that no single test or defense method is invulnerable, and that continued research and proactive updates remain necessary.
   • In the conclusion, the authors state that AGENT VIGIL exposes the limitations of existing agent defenses, highlighting the need for more robust safety measures.

   Scope of experimental coverage

   • 7 models were tested: o3-mini, GPT-4o, GPT-4o-mini, Claude-3.5-Sonnet, Gemini-2-flash-exp, QwQ-32B, o3-mini-2025-01-31 (checkpoints given in Appendix A).
   • The benchmarks are AgentDojo (fuzzing set 142 tasks, test set 173 tasks) and VWA-adv (99/100, text trigger tasks only); real-world validation is a single WebArena shopping website case.
   • Defense evaluation includes pi_detector, repeat, delimit for AgentDojo and safety, paraphrase, combined for VWA-adv; tool_filter was excluded due to incompatibility with o3-mini, and VWA-adv's image-text consistency defense was excluded due to API call overhead.
   • The fuzzing budget is 10 iterations, with 3 mutations per round on AgentDojo and 10 per round on VWA-adv; the paper does not report total query counts, runtime or statistical significance tests.
   • The paper discusses why defenses such as StruQ and IsolateGPT were not included in the evaluation: StruQ was validated only on small open-source models and lacks agent task capability, and IsolateGPT depends on a specific system design and is difficult to adapt to different architectures (Section 5.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   AGENT VIGIL uses fuzzing to automatically attack black-box agents, with ASR of about 70% on two benchmarks, nearly twice the baseline, and can transfer to unseen tasks and models.

   An automated red-teaming paper targeting indirect prompt injection in LLM agents. The problem the authors aim to solve is: existing evaluation methods either construct attacks manually or apply only to specific agents, and cannot generically evaluate the injection risk of black-box agents.

   • Method: AGENT VIGIL draws on software fuzzing, starting from a corpus of high-quality templates, using five mutation strategies (executed by GPT-4o-mini) to generate new prompts, scoring them by attack success rate plus a coverage reward, and then using a UCB1-based MCTS algorithm to select seeds, iterating and optimizing.
   • Main results: On AgentDojo, ASR against the o3-mini agent is 71% (baseline 38%); on VWA-adv, ASR against the GPT-4o agent is 70% (baseline 36%); the authors state this is close to twice the baseline.
   • Transferability: On unseen test tasks in AgentDojo, 65% against o3-mini; on VWA-adv, 67% when transferred to Gemini-2-flash-exp, which was unseen during fuzzing; but essentially ineffective against Claude-3.5-Sonnet (0.03–0.04 on AgentDojo).
   • Defense evaluation: Under three defenses on AgentDojo, ASR remains 0.12–0.49; under VWA-adv defenses, performance decreases and is on par with the baseline (0.27–0.33).
   • Real world: On a local WebArena shopping website, injection through reviews successfully misled the agent into visiting an arbitrary URL.
   • Authors' conclusion: AGENT VIGIL exposes the limitations of existing agent defenses, and the authors believe it can serve as a foundation for understanding agent threats and developing next-generation security solutions.

   Collapse this answer

Related papers