IPI defense frameworks for LLM agents: taxonomy, evaluation and attack exploitation

1. 01What problem does this paper attempt to solve?

   The IPI defense field is fragmented and lacks a unified taxonomy and comprehensive evaluation; this paper conducts an SoK and exposes the root causes of defense failures.

   Core problem: defense frameworks for indirect prompt injection (IPI) against LLM agents lack a unified taxonomy and comprehensive evaluation covering safety, utility, and overhead, and their failure root causes also lack systematic analysis.

   • Problem setting: LLM agents with tool-calling capabilities are widely deployed, and attackers can hijack their tool calls by manipulating external data the agent receives, causing data leakage or unauthorized operations (per the authors).
   • Existing shortcomings (per the paper): the field exhibits "siloed innovation," with frameworks highly specialized or homogeneous in design; many frameworks are evaluated only in static simulated environments, ignoring dynamic environments; utility mostly considers only task success rate, ignoring overhead and false positives on benign queries; defenses suffer from a "bottleneck effect," where once the core mechanism is understood by an attacker it is bypassed.
   • Core observation: existing defenses are mainly designed and evaluated against template-based attacks, and static benchmarks only test a framework's ability to recognize specific syntactic patterns, which the authors argue overestimates defense robustness.
   • Paper outputs: a SoK comprising a five-dimensional taxonomy, an evaluation of 10 representative frameworks on three benchmarks, six categories of failure root causes (RC 1–6), and three targeted adaptive attacks.
   • Threat model (Section 2.3, following Progent's modeling):
     • Goal: make the tool sequence τ ultimately executed by the agent equal the attacker's desired τ m.
     • Knowledge: no prior knowledge of system A, including its architecture and the LLM it uses.
     • Capability: can only manipulate static resources in environment E; when A calls a tool, E returns a malicious response om := temp(qm), where temp is a static template.
     • Victim system and trusted base: an agent system composed of the user, the agent/defense framework A, the tool set T, and the environment E; except for tool return results, the memory module, tool descriptions, SDK, and operating system are all considered trusted.

   Collapse this answer

2. 02What related research exists?

   Related work is divided into three groups: IPI attacks, six defense paradigms, and existing adaptive attacks; the paper positions itself as the first systematic SoK (per the authors).

   IPI attacks and benchmarks

   • Template-based attacks: InjecAgent [63] and AgentDojo [23] concatenate injected tasks using templates; ChatInject [14] disguises malicious payloads as the model's internal dialogue structure. BIPIA [60] targets the QA LLM setting.
   • Search- and optimization-based attacks: the IPI version of GCG [37] refines injection prompts via white-box optimization; AgentVigil [53] and AutoHijacker [36] use LLMs to adjust injection prompts in black-box settings.
   • Existing adaptive attacks: Zhan et al. [62] use multi-objective GCG and AutoDAN to bypass multiple defense frameworks; Nasr et al. [39] use various optimization and search algorithms to bypass 12 defense frameworks. The paper notes that these works mainly optimize against prompt-based or fine-tuning-based defenses and overlook potential weaknesses of more complex defense frameworks.

   Six defense paradigms (Section 3.1, one representative per category)

   • Detection: PromptArmor [44] and LLMZ+ [18] use prompts to have an LLM judge whether data contains injection; the AlignmentCheck component of LlamaFirewall [19] is a prompted chain-of-thought auditor.
   • Prompt engineering: PPA [52] generates unpredictable data delimiters; FATH [49] wraps information from different sources with hash tags; DefensiveTokens [15] insert special tokens optimized for safety objectives.
   • Fine-tuning: SecAlign [16] uses DPO to train refusal of injected instructions; Meta SecAlign [17] is its open-source improved version; Wallace et al. [48] propose instruction hierarchy.
   • System design: IsolateGPT [58] (plan-exec decoupling), ACE [31] (code-then-exec), F-secure [55] (Dual LLM), FIDES [20] (information flow control), CaMeL [22] (combines the previous three).
   • Runtime checking and policy enforcement: Task Shield [29], MELON [65], and IPIGuard [12] perform runtime auditing; Conseca [47], AgentArmor [50], and Progent [43] use deterministic policies for mandatory access control.

   Baselines and benchmarks: the safety evaluation baselines are undefended GPT-4o and Llama-3.1-8B; the benchmarks are AgentDojo, ASB, and InjecAgent (with 629, 2040, and 1054 test cases, respectively).

   Positioning of this paper: the authors state this is the first comprehensive analysis (SoK) of IPI-centric defense frameworks, and that it differs from existing adaptive attacks in that it does not perform simple optimization or search but designs attacks targeting architectural logic flaws based on failure root causes.

   Collapse this answer

3. 03How does the paper address this problem?

   A five-dimensional taxonomy organizes the field, three benchmarks evaluate safety/utility/overhead, failure logs induce root causes, and then three logic-driven adaptive attacks are designed.

   Overall approach: first use a five-dimensional taxonomy to systematically organize IPI defense frameworks, then uniformly evaluate representative frameworks on three benchmarks, induce root causes from failure logs, and finally design adaptive attacks targeting the root causes to verify that the flaws can be exploited.

   Five-dimensional taxonomy (Section 3)

   • Technical paradigm (main axis): six categories—detection, prompt engineering, fine-tuning, system design, runtime checking, policy enforcement; system design is further divided into four principles: plan-exec decoupling, code-then-exec, Dual LLM, and information flow control (IFC).
   • Intervention stage: pre-inference (detection), intra-inference (prompt engineering, fine-tuning, system design), post-inference (runtime checking, policy enforcement).
   • Model access: white-box (fine-tuning and DefensiveTokens) and black-box (the vast majority of frameworks).
   • Interpretability: deterministic (system design and policy enforcement contain deterministic components) and probabilistic (paradigms that rely on LLM judgment).
   • Degree of automation: full-automation and semi-automation (requires manual labeling of safety labels, confirmation of alerts, or writing policies, e.g., CaMeL, IsolateGPT, Progent).

   Evaluation design (Section 4)

   • Framework selection: select 10 representatives across the six technical paradigms—LlamaFirewall (detection), Tool Filter (prompt engineering), Meta SecAlign (fine-tuning, using its public Llama-3.1-8B-Instruct LoRA adapter), IsolateGPT/ACE/CaMeL (three system design architectures), Task Shield/MELON (runtime checking), Progent (policy enforcement, the only open-source framework in this category).
   • Metrics: safety uses ASR (proportion of injected instructions successfully executed); utility uses the task completion rate in AgentDojo's no-attack scenarios; overhead uses wall-clock time and token usage.
   • Unified configuration: except for fine-tuning, the backend LLM is uniformly GPT-4o (2024-11-20), temperature 0; tool calls use the OpenAI function calling template; frameworks requiring manual configuration, such as CaMeL, are evaluated only on their automated components.
   • Failure analysis: two authors independently reviewed all failure logs to induce root causes, and disagreements were resolved through cross-checking and discussion.

   Six categories of failure root causes (Section 4.2)

   • RC 1/2: imprecise access control over tool selection and tool parameters (e.g., parameters within a tool sequence allowed by the plan are hijacked).
   • RC 3: incomplete isolation of malicious information (e.g., in CaMeL, an error message passes injected content back to the privileged LLM).
   • RC 4: judgment errors by LLMs used for checking/detection (e.g., LlamaFirewall misjudges a malicious action as aligned).
   • RC 5: insufficient coverage of safety policies (policies generated by the LLM do not constrain tool parameters).
   • RC 6: poor generalization of fine-tuned models (Meta SecAlign is trained for RAG scenarios and does not target injection via tool returns).

   Three adaptive attacks (Section 5)

   • Semantic-Masquerading IPI (targeting RC 1&2): treats the user query as known, uses an LLM to rewrite the attack payload to use tools that are the same as or functionally similar to those of the user's legitimate task, making the malicious intent indistinguishable to defense filters; a separate payload judge LLM verifies that the rewritten payload still achieves the original attack goal, with feedback-driven rewriting on deviation, up to 3 iterations, falling back to the original payload on failure; 949 adjusted payloads were generated in total. Because AgentDojo is a stateful environment and the ground truth changes after rewriting, attack success is instead determined by another LLM judging whether the trajectory executed the injected instruction.
   • Cascading IPI (targeting RC 4): the payload is split into head/body/end, the body uses an If-then-else structure, and each conditional branch targets a specific LLM component—judge-type branches force it to output a benign response, filter-type branches force it to repeat the entire payload to bypass extraction/filtering, and the last branch makes the main LLM execute the injected task; the templates for each branch are manually written according to the target component's system prompt.
   • Isolation-Breach IPI (targeting RC 3): exploits channels such as error message propagation or context feedback loops to inject malicious intent into the trusted planner's context, causing it to generate a plan containing malicious tools, thereby defeating downstream isolation mechanisms; triggered by reading two emails in sequence, and presented in the paper as a case study.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Safety/utility/overhead evaluation of 10 frameworks × 3 benchmarks, failure analysis of six root-cause categories, validation with three adaptive attacks, plus a DeepSeek-V3 generalization experiment.

   Experimental setup

   • Frameworks evaluated: LlamaFirewall, Tool Filter, SecAlign (Meta SecAlign LoRA), IsolateGPT, ACE, CaMeL, Task Shield, MELON, Progent, Progent-LLM (a variant where the LLM automatically generates policies); baselines are undefended GPT-4o and Llama-3.1-8B.
   • Backend models: GPT-4o (2024-11-20), temperature 0; fine-tuning-based ones use Llama-3.1-8B-Instruct; Appendix F uses DeepSeek-V3 for generalization validation.
   • Benchmarks and scale: AgentDojo (4 scenarios/attack methods, 74 tools, 629 cases), ASB (10 scenarios, 420 tools, 2040 cases), InjecAgent (6 scenarios, 62 tools, 1054 cases); utility and overhead are measured only in AgentDojo no-attack scenarios.
   • Metrics: ASR, Utility (task completion rate), FP rate, wall-clock time, token usage.
   • Attack success determination: benchmark evaluations use each benchmark's original determination; for semantic masquerading attacks, because the payload is rewritten, an LLM is used instead to judge whether the trajectory executed the injected instruction (Figure 5).
   • Reliability of failure analysis: two authors independently reviewed all failure logs and discussed disagreements until consensus; the paper does not report the number of repetitions.

   Main results

   Safety evaluation (ASR, lower is better; per Table 2, GPT-4o backend, numbers are each benchmark's Average column and the overall average):

   Swipe horizontally to read the table

   Framework	AgentDojo	ASB	InjecAgent	Overall average
   GPT-4o (baseline)	15.62%	30.54%	2.66%	23.69%
   LlamaFirewall	1.74%	9.85%	0.10%	9.13%
   Tool Filter	2.39%	6.08%	0.00%	4.59%
   IsolateGPT	0.52%	0.00%	0.00%	0.12%
   ACE	N/A	0.15%	0.00%	0.10%
   CaMeL	1.21%	7.40%	0.00%	4.81%
   Task Shield	0.94%	0.25%	0.00%	0.43%
   MELON	0.24%	4.17%	0.00%	2.67%
   Progent	0.00%	0.00%	N/A	0.00%
   Progent-LLM	2.72%	7.75%	0.00%	4.63%
   Llama3.1-8B (baseline)	6.04%	58.97%	46.87%	45.91%
   SecAlign	2.74%	16.57%	0.19%	11.77%

   • The authors state that most frameworks reduce ASR below 10% in most attack scenarios; policy enforcement and system design categories perform best, while prompt and fine-tuning categories are relatively weaker.
   • The authors point out that apart from Progent (handcrafted policies containing substantial prior knowledge), no framework achieves 0% ASR; Tool Filter and CaMeL have relatively high average ASR on ASB, which the authors attribute to ASB containing attacker tools that mimic legitimate functions, making frameworks that rely on intent recognition prone to misjudgment.
   • In Figure 3, the curves for each attack template cluster tightly, from which the authors conclude that the defenses eliminate the influence of template semantics, and that residual ASR stems from core design flaws.

   Utility and overhead (AgentDojo, per Table 3)

   • Utility: GPT-4o baseline 80.41%; Task Shield 83.51%, Progent-LLM 77.32%, Progent 72.17% are close to the baseline; IsolateGPT 37.11% and CaMeL 59.79% show the largest drops; Llama3.1-8B baseline 28.87%, SecAlign 29.90%.
   • Overhead: Task Shield averages 501294.32 tokens and MELON 53338.41, far above GPT-4o's 8020.05; Progent's time of 3.95s is below the baseline's 4.98s; Progent-LLM's time rises to 16.35s, comparable to the system design category.
   • FP rate (Appendix B, Table 8): most checking-based frameworks are around 10%, which the authors consider directly related to the utility drop; Task Shield is an exception, with FP reaching 75.00% in the workspace scenario but utility unaffected, which the authors attribute to it using prompts to correct actions rather than blocking them.

   Adaptive attacks

   • Semantic-Masquerading IPI (Table 5, AgentDojo): against IsolateGPT ASR 1.91% (3.9X, direct payload 0.48%), Tool Filter 3.06% (2.9X, originally 1.05%), CaMeL 1.90% (2X, originally 0.95%), LlamaFirewall 3.37% (4.0X, originally 0.84%); the authors state that ASR increases by more than twofold on all four frameworks.
   • Cascading IPI (Table 6, AgentDojo): against Task Shield ASR 5.37% (2.6X, Important template 2.07%), MELON 4.53% (4.8X, originally 0.95%); the authors note that the payload terms were not optimized and only slight adjustments were made to the component prompts.
   • Isolation-Breach IPI: a case study targeting IsolateGPT (InjecAgent scenario, GmailReadEmail task paired with AugustSmartLockUnlockDoor intent); the authors state that the content of the first email propagates from Spoke to Hub and contaminates its context, the Hub adds the malicious tool to the plan, and the second email triggers execution; because the attack flow is long, the paper does not conduct a large-scale evaluation.

   Other ablations and analyses

   • DeepSeek-V3 generalization (Appendix F, Table 9, strongest template per benchmark): baseline average ASR 15.54%, lower than GPT-4o; framework trends are consistent with GPT-4o, with Progent 0.00%, IsolateGPT 0.24%, Task Shield 2.60%.
   • Distribution of failure cases: the authors report that many bypass cases for Task Shield and MELON stem from RC 4 (misjudgment); Progent's handcrafted policies outperform LLM-generated policies (0.00% vs. 4.63% overall average in Table 2).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out areas for improvement such as attack success determination, payload optimization, and evaluation scale; the experiments cover 10 frameworks, 3 benchmarks, and 3 backend models.

   Limitations and future directions noted by the authors

   • Success determination method: in the semantic masquerading attack, the state-change rules for the 949 rewritten payloads must be written manually and are not scalable, so an LLM is used instead to judge whether the attack succeeded (Section 5.1).
   • Attack strength not fully explored: the authors believe attack effectiveness could be further improved by designing a stronger adaptive payload pipeline, and state that the focus of this SoK is not designing comprehensive attack details, with the semantic masquerading attack being only a relatively straightforward demonstration (Section 5.1).
   • Payload terms not optimized: Cascading IPI is only a preliminary experiment, and the payload terms were not optimized, with only slight adjustments made according to the prompts of the LLM components (Section 5.2).
   • Case study instead of large-scale evaluation: the Isolation-Breach attack has a long workflow, making large-scale evaluation unrealistic, so the paper provides only a case study (Section 5.3 footnote).
   • Future defense direction: the authors state that future defenses should go beyond ad-hoc isolation, for example by mathematically enforcing the non-interference property to ensure that contaminated input cannot influence the privileged control flow through any channel (including error logs and side effects), which is needed to provide provable safety guarantees (Section 5.3).
   • Call at the conclusion level: the authors state that agent safety needs to go beyond static benchmarks and address core architectural flaws (Section 6).

   Experimental coverage

   • There are 10 defense frameworks under test (including the Progent-LLM variant), covering all six technical paradigms in the paper's taxonomy; there are 3 backend models: GPT-4o, Llama-3.1-8B-Instruct, and DeepSeek-V3 (Section 4, Appendix F).
   • Safety evaluation covers all attack templates of the three static benchmarks AgentDojo, ASB, and InjecAgent; utility and overhead evaluation use only AgentDojo, and ACE is excluded from utility analysis because it supports only single-turn interaction (Section 4.1 footnote).
   • Quantitative evaluation of the adaptive attacks is done only on AgentDojo: the semantic masquerading attack tests 4 frameworks, and Cascading IPI tests 2 frameworks; Isolation-Breach has only a case study (Section 5).
   • Frameworks requiring manual configuration (such as CaMeL) are evaluated only on their automated components, with the parts requiring manual configuration disabled (Section 4).
   • The paper does not report the specific model of the auxiliary LLM used in the attacks, the number of experimental repetitions, or the agreement between LLM judgments and human judgments.

   Collapse this answer

6. 06Summarize the main content of the paper

   SoK: five-dimensional taxonomy + evaluation of 10 frameworks + six failure root-cause categories + three adaptive attacks; the authors state that residual ASR stems from architectural flaws.

   An SoK on IPI-centric defense frameworks for LLM agents, which the authors state is the first comprehensive analysis in this field.

   • Problem addressed: the IPI defense field is fragmented and lacks a unified taxonomy; evaluations are mostly in static environments and look only at task success rate, ignoring dynamic environments, overhead, and false positives; there is a lack of analysis from the perspective of defense failure.
   • Method highlights: propose a five-dimensional taxonomy (technical paradigm, intervention stage, model access, interpretability, degree of automation); evaluate 10 frameworks on AgentDojo, ASB, and InjecAgent under a unified configuration (GPT-4o, temperature 0); two authors independently analyze failure logs to induce six root-cause categories (imprecise tool selection/parameter access control, incomplete isolation, detection LLM misjudgment, insufficient policy coverage, poor fine-tuning generalization); based on these, design three logic-driven adaptive attacks.
   • Safety results: most frameworks reduce ASR below 10% in most scenarios; Progent overall average 0.00%, IsolateGPT 0.12%, versus GPT-4o baseline 23.69% and Llama3.1-8B baseline 45.91% (Table 2); the authors point out that apart from Progent, no framework achieves 0% ASR.
   • Utility and overhead: IsolateGPT's average Utility drops to 37.11% (GPT-4o baseline 80.41%); Task Shield's average token consumption is 501294.32, dozens of times the baseline's 8020.05 (Table 3).
   • Adaptive attacks: Semantic-Masquerading IPI raises ASR against LlamaFirewall by 4.0X (0.84%→3.37%), and Cascading IPI raises it against MELON by 4.8X (0.95%→4.53%); the Isolation-Breach attack against IsolateGPT is shown as a case (Tables 5, 6, Section 5.3).
   • Authors' conclusion: static benchmarks overestimate defense robustness, and residual ASR reflects architectural-level design flaws; future defenses need to go beyond ad-hoc isolation and block contaminated information from influencing privileged control flow in a provable manner.

   Collapse this answer

Related papers