Adaptive attacks break indirect prompt injection defenses for LLM agents

1. 01What problem does this paper attempt to solve?

   Evaluate the performance of existing indirect prompt injection defenses under adaptive attacks, and examine the ability of adversarial attacks to bypass them in agent environments.

   Existing indirect prompt injection (IPI) defenses for large language model (LLM) agents lack robustness testing against adaptive attacks, posing a security risk of targeted bypass.

   • Application background and risks: LLM agents execute operations by calling external tools and are widely used in high-risk domains such as finance, healthcare, autonomous driving, and chemistry laboratories; however, integrating external tools introduces the threat of indirect prompt injection (IPI) attacks, where attackers can embed malicious instructions in external data read by the agent, inducing unauthorized operations, data leakage, or even physical harm.
   • Shortcomings of existing defenses: Although research has proposed various IPI defense schemes at the detection, input, and model levels, most of these defenses are tested only under non-adaptive threats, lacking robustness checks under targeted adaptive attacks after the defense mechanisms are fully disclosed.
   • Core argument: In traditional security and machine learning security, non-adaptive attacks tend to underestimate system vulnerabilities, making seemingly effective defenses fail under adaptive attacks; evaluating IPI defenses must introduce targeted adaptive adversarial example construction.
   • Threat model:
     • Attacker objective: Induce the agent to execute a specified malicious attack tool, completing direct harm (e.g., unauthorized transactions) or sensitive data theft and exfiltration.
     • Attacker knowledge: Assumes the attacker has white-box access, fully knowing the agent model, defense model architecture, parameters, and detailed prompts.
     • Attacker capability: The attacker can only manipulate the content of external data sources (e.g., emails or comments), inserting gradient-optimized adversarial strings before and after malicious instructions.
     • Victim system: An agent system composed of a backbone LLM and a tool set, deployed with a single defense such as detection filtering, prompt isolation, or adversarial fine-tuning.

   Collapse this answer

2. 02What related research exists?

   Reviewing agent security risks, LLM adversarial jailbreak attacks, and adaptive attack trajectories, the authors state this is the first work to explore adaptive attacks for agent security.

   • LLM agent security risks:
     • Overall risk analysis: Related surveys (He et al., 2024; Deng et al., 2024) and benchmark evaluations (Yuan et al., 2024a; Zhang et al., 2024b; Tang et al., 2024) categorize agent risks into unintentional risks without malicious attackers (Ruan et al., 2024; Ma et al., 2024) and malicious intentional attacks.
     • Malicious attack paradigms: Mainly include indirect prompt injection attacks that inject malicious instructions into tool responses (Zhan et al., 2024; Debenedetti et al., 2024), attacks targeting knowledge base poisoning of RAG agents (Chen et al., 2024), backdoor attacks that implant triggers through fine-tuning (Yang et al., 2024; Wang et al., 2024; Hubinger et al., 2024), and other agent attacks (Zhang et al., 2024a; Gu et al., 2024).
   • LLM adversarial attacks and jailbreaks:
     • Prompt engineering and jailbreaks: Models aligned via RLHF (Ouyang et al., 2022) or DPO (Rafailov et al., 2023) can still be bypassed by jailbreak prompts (Wei et al., 2023; Yuan et al., 2024b).
     • Automated adversarial attacks: GCG (Zou et al., 2023) uses discrete coordinate gradient optimization of adversarial suffixes to induce non-compliant outputs; AutoDAN (Liu et al., 2024a; Zhu et al., 2023) generates semantically coherent adversarial prompts to evade perplexity detection.
   • Adaptive attack mechanisms:
     • Cross-domain defense testing: Adaptive attacks are widely adopted in evaluating adversarial defenses in computer vision (Athalye et al., 2018; Tramèr et al., 2020; Carlini and Wagner, 2017), later extended to baseline evaluation of LLM jailbreak defenses (Jain et al., 2023; Andriushchenko et al., 2024).
   • Comparison baselines and benchmarks:
     • The paper evaluates on InjecAgent (Zhan et al., 2024) and AgentDojo (Debenedetti et al., 2024) benchmarks, selecting 8 defenses as baselines: fine-tuned detector, LLM detector, perplexity filtering, instructional prevention, prompt isolation, sandwich prevention, paraphrasing, and adversarial fine-tuning.
   • Positioning differences from existing work:
     • The authors state this is the first study to evaluate defenses and adaptive attacks for LLM agents in indirect prompt injection scenarios; compared to prompt injection on base LLMs, agent attacks require directly driving tool execution, with higher input-output complexity.

   Collapse this answer

3. 03How does the paper address this problem?

   For 8 IPI defenses, use white-box gradient optimization to design adaptive attacks: GCG, multi-objective GCG, two-stage GCG, and AutoDAN, adapted to agent tool calls.

   The paper proposes a white-box adaptive attack framework against IPI defenses for LLM agents, inserting adversarial strings into external data through gradient optimization to manipulate agent tool execution and bypass various defenses.

   Formal Definition and Attack Objective

   An agent consists of a backbone model and a tool set. Benign user instructions trigger tool execution and return external content; the attacker concatenates malicious instructions with adversarial strings and embeds them into external content. The attack loss function is defined as minimizing the negative log-likelihood of generating the target sequence: L_attack = − log P_M(target ∣ Prompt(I_u, I_a ⊕ S)) where the model maximizes the probability of generating the target sequence given the input. For ReAct agents, the target sequence is designed as a specific thought prefix, guiding the model to generate the intention to call the attacker's tool in the chain of thought; for fine-tuned agents, the target sequence is designed as a key-value structure consistent with function call syntax, directly triggering the execution of the attack tool.

   GCG Optimization for Input-Level and Model-Level Defenses

   • Input-level defenses: Includes instruction guarding (explicitly requiring the model to ignore external instructions in the prompt), data prompt isolation (wrapping tool responses with specific delimiters), sandwich guarding (repeating user instructions after tool responses), and text rewriting (breaking adversarial strings by restating external content).
   • Model-level defenses: Uses adversarial fine-tuning, fine-tuning the model with unsuccessful adversarial examples to resist malicious instructions.
   • Direct GCG attack: For the above defenses without additional detectors, directly uses the GCG algorithm to search for the optimal adversarial string in the discrete token space. For the Llama3-8B model, prefix insertion works better than suffix insertion.

   Multi-Objective GCG and AutoDAN for Detection-Level Defenses

   • Multi-objective GCG optimization: For fine-tuned detectors (based on DeBERTaV3 discrimination) and LLM detectors (prompting an LLM for binary classification), the attack needs to simultaneously satisfy attack triggering and stealth escape objectives, jointly optimizing the loss function:

   L_joint = α L_attack + (1 − α) L_detect When the detector and agent share a vocabulary, joint gradient optimization is used; when vocabularies differ, alternating single-step optimization is used.

   • Semantic coherence AutoDAN: For perplexity filtering defenses (blocking strings above a perplexity threshold), conventional GCG-generated strings have high perplexity and are easily filtered. The authors use the AutoDAN algorithm, which selects tokens from left to right, to generate semantically coherent adversarial strings to bypass the threshold.

   Two-Stage GCG for Rewriting Defenses

   Text rewriting defenses can rewrite external content to weaken token-level perturbations. The authors design two-stage GCG (T-GCG): first train an adversarial prefix so that malicious instructions drive the target output; then train a second adversarial string appended to the input, guiding the rewriting model to exactly preserve the first-stage adversarial instruction sequence after rewriting, thereby triggering the agent to execute the attack tool. The appendix provides the specific structure of the relevant defense prompts.

   Collapse this answer

4. 04What experiments did the paper conduct?

   When testing 8 defenses on InjecAgent and AgentDojo, the adaptive attack ASR all exceeded 50%, and the detector detection rate dropped to near 0%.

   Experimental Setup

   • Models under test: Evaluated two types of LLM agents, including a ReAct-prompt-based agent (backbone model Vicuna-7B-v1.5) and a function-calling fine-tuned agent (backbone model Llama-3.1-8B-Instruct); detection defenses involve a DeBERTaV3 fine-tuned detector.
   • Evaluation benchmarks: Main experiments used the InjecAgent benchmark, randomly sampling 100 test cases from 1054 (including 25 direct harm attacks, 27 data theft attacks, and 16 user scenarios); the appendix used the AgentDojo benchmark with 16 test cases.
   • Defense baselines: Implemented 8 defense strategies, including fine-tuned detector (FD), LLM detector (LD), perplexity filtering (PF), instructional prevention (IP), data prompt isolation (DPI), sandwich prevention (SP), paraphrasing (P), and adversarial fine-tuning (AF).
   • Evaluation metrics: Primary metric is attack success rate ASR (ASR-all, proportion of successful attacks among all cases), reporting ASR-defense after deploying defenses and ASR-adaptive attack after adaptive attacks; supplemented by target rate, detection rate DR, and valid output rate.
   • Evaluation process and criteria: In single-turn interactions, direct harm is considered successful if the model outputs execution of a harmful tool; data theft is two-step: first step outputs execution of a data extraction tool, then gpt-4-0613 simulates the tool response, and the second step outputs execution of a data transfer tool, judged as fully successful.
   • Computational and training details: Each adversarial example is trained on a single NVIDIA A100 GPU for about 30 minutes, with early stopping if loss does not decrease for 100 consecutive steps; adversarial fine-tuning uses LoRA with 4-bit quantization for 15 epochs.

   Main Results

   Swipe horizontally to read the table

   Defense method	Vicuna-7B defense ASR (%)	Vicuna-7B adaptive attack ASR (%)	Llama3-8B defense ASR (%)	Llama3-8B adaptive attack ASR (%)
   Perplexity filtering	6	76	0	61
   Instructional prevention	9	91	1	79
   Paraphrasing	1	79	0	54
   Data prompt isolation	1	94	Not reported	Not reported
   LLM-based detector	1	74	0	81
   Fine-tuned detector	20	84	1	74
   Sandwich prevention	15	68	Not reported	Not reported
   Adversarial fine-tuning	12	65	4	85

   (From Figure 2; under no-defense baseline, Vicuna-7B ASR is 56%, Llama3-8B is 9%; Llama3-8B was not tested with DPI and SP due to inherent structure)

   • Authors' interpretation of overall bypass effectiveness: The authors state that all targeted adaptive attacks achieve ASR over 50% on the tested defenses and agents, higher than the initial ASR before deploying defenses, showing that existing defenses can be bypassed by adaptive attacks.
   • Base model tolerance differences: Llama3-8B's initial ASR without defense (9%) is lower than Vicuna-7B's (56%), but under adaptive attacks both defenses are broken, with Llama3-8B reaching 85% adaptive ASR under adversarial fine-tuning.
   • Defense mechanism effectiveness differences: Among prompt-level modifications, sandwich prevention shows relatively better defense performance within valid output ranges; perplexity filtering and paraphrasing defenses resist non-adaptive strings but are broken by targeted adaptive attacks.

   Detection Rate and Non-Adaptive Attack Comparison

   • Detection rate drop test: Under original IPI attacks, FD detection rates for Vicuna-7B and Llama3-8B are both 61%, and LD rates are 34% and 72% respectively (Table 2); after adding adaptive adversarial strings, FD detection rates drop to 1% and 10%, and LD rates drop to 0% for both.
   • Adaptive vs. non-adaptive comparison: Under perplexity filtering, non-adaptive adversarial strings achieve ASR of 24% and 22% on Vicuna-7B and Llama3-8B, respectively, while adaptive strings rise to 76% and 61% (Table 3); under paraphrasing defense, non-adaptive strings have ASR of 53% and 8%, adaptive strings rise to 79% and 54%.
   • Stealth objective achievement: The authors state that detection rates dropping to near 0% indicate the stealth escape objective in multi-objective optimization has been achieved, and future attack improvements should focus on enhancing the attack trigger objective.

   Attack Stage Decomposition and Cross-Evaluation

   • Multi-stage impact differences: In data theft attacks, adaptive attacks achieve first-step tool call ASR of 0.90 ± 0.07 on Vicuna-7B and 0.81 ± 0.12 on Llama3-8B, but second-step tool call ASR are 0.75 ± 0.17 and 0.71 ± 0.16, with smaller improvements (Table 4).
   • Stage limitation mechanism: The authors state the reason is that adversarial strings are only trained to guide the model to produce the first-step target output, having only indirect influence on subsequent interaction steps.
   • Cross-defense specificity: In cross-evaluation of adversarial strings trained against specific defenses, adaptive attacks optimized for a specific defense typically achieve the highest ASR on that defense (Figure 5).

   Long Context and Valid Output Rate Analysis

   • Format validity rate changes: Most input-level defenses reduce valid output rate, while sandwich prevention produces more invalid format outputs, with a valid rate of 58% (Figure 4); after applying adaptive attacks, due to forced generation of thinking prefixes, the valid rate recovers to 78%.
   • Long-context benchmark results: On the longer-context AgentDojo benchmark (average 3823 tokens), Llama3-8B under no-defense setting has GCG ASR of 56.25%, under LLM detector multi-objective GCG ASR is 31.25%, and under instructional prevention GCG ASR is 43.75% (Table 6).
   • Long-context attack difficulty: The authors state that adaptive attack effectiveness on AgentDojo is lower than on InjecAgent, speculating that the main reason is the significantly longer context and multi-turn interactions, increasing the difficulty of adversarial string optimization.

   Other Ablations and Analyses

   • Target rate vs. ASR differences: Under Vicuna-7B, data prompt isolation achieves a target rate of 96% (ASR 94%), while adversarial fine-tuning has a target rate of 93% but ASR of 65% (Figure 3).
   • AutoDAN semantic tolerance: AutoDAN's strict target rate on Vicuna-7B is 36%, increasing to 62% if semantically approximate outputs are counted (Section 5.2).
   • Adversarial fine-tuning training set: Data collected from unsuccessful test cases, with 215 cases for Vicuna-7B and 816 for Llama3-8B (Appendix A).
   • Optimization computational cost: Optimizing adversarial strings for a single test case takes about 30 minutes on a single A100 GPU (Appendix A).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that current attacks are limited to first-step actions, rely on white-box access, have not tested combined defenses, and defense coverage is not exhaustive.

   Limitations and future directions noted by the authors

   • Optimizing only the first action: The adaptive attack only trains adversarial strings to control the agent's first action after receiving external content, with minimal improvement to the second step of data theft attacks; the authors note that improving the attack strategy to account for long-term effects is a future research direction (Section 10).
   • Dependence on white-box access assumptions: The current attack assumes white-box access to the agent model, defense model, and detailed prompts; the authors note that studying black-box and gray-box attacks is equally crucial for comprehensive evaluation (Section 10).
   • Combined defenses not addressed: The current adaptive attack targets only individual defenses, whereas combining different defenses may establish stronger protection mechanisms (e.g., jointly applying three detection strategies); exploring adaptive attacks against combined defenses is future work (Section 10).
   • Defense strategy coverage not exhaustive: This study does not exhaust all potential defenses adaptable to IPI, such as LLM self-evaluation (Phute et al., 2024) or alternative fine-tuning methods (Zhang et al., 2024c) (Section 10).

   Experimental coverage

   • Models tested: The evaluation covers agents built on 2 backbone LLMs (Vicuna-7B-v1.5 and Llama-3.1-8B-Instruct), as well as 1 detection model based on DeBERTaV3 and gpt-4-0613 used as a tool in the simulated environment.
   • Test benchmarks and cases: The main experiments evaluate on 100 test cases drawn from InjecAgent (50 direct harm cases, 50 data theft cases), and the appendix evaluates 16 test cases on AgentDojo.
   • Defense method coverage: A total of 8 defense strategies were evaluated: detection-level (FD, LD, PF), input-level (IP, DPI, SP, P), and model-level (AF); among them, Llama3-8B excluded DPI and SP due to its conversation structure.
   • Metrics and evaluation dimensions: Reports ASR-all, Target rate, DR, and Valid rate under single-turn interaction; the paper does not evaluate the impact of defenses on utility for normal benign user requests.
   • Compute and optimization overhead: Each adversarial sample is trained on a single NVIDIA A100 GPU for about 30 minutes, with optimization steps ranging from 100 to 2000; the paper does not report the latency and query overhead of defenses and attacks in actual deployment.

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors state that all 8 IPI defenses have ASR exceeding 50% under adaptive attacks, emphasizing that agent defense evaluation needs to introduce adaptive testing.

   1. Positioning and core problem: The paper evaluates 8 representative defense schemes for large language model (LLM) agents against indirect prompt injection (IPI) attacks, exploring the effectiveness of existing defenses under white-box adaptive attacks with full knowledge of the defense mechanisms.
   2. Method design: For detection-level, input-level, and model-level defenses, a series of white-box adversarial attack methods adapted to agent tool calls are designed, including GCG adapted to chain-of-thought thinking formats and function call formats, multi-objective GCG balancing escape and attack objectives, two-stage GCG against text rewriting, and AutoDAN against perplexity filtering.
   3. Main experimental results: In evaluations on 100 test cases from InjecAgent, targeted adaptive attacks achieve attack success rates (ASR) exceeding 50% across both Vicuna-7B and Llama3-8B agents and 8 defenses (Figure 2), with detection rates of both fine-tuned detectors and LLM detectors dropping to near 0% (Table 2).
   4. In-depth analysis: Direct harm attacks achieve an average ASR above 0.87 after adaptive optimization (Table 4); data theft shows less improvement in the second-step call because the adversarial string only optimizes the first-step action; on the long-context benchmark AgentDojo, adaptive attacks become harder, with ASR between 31.25% and 56.25% (Table 6).
   5. Authors' conclusion: The authors state that current IPI defenses are all broken under adaptive attacks, and non-adaptive evaluation tends to underestimate system security vulnerabilities, emphasizing that future agent defense design must include adaptive attack testing as a core standard for evaluating security.

   Collapse this answer

Related papers