MetaPI: Building a prompt injection benchmark from LLM agent benchmarks
MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark
The authors propose MetaPI, which automatically converts agent utility benchmarks such as AppWorld into IPI evaluation suites; the authors state that under adaptive attacks no defense maintains both strong utility and strong safety.
- Indirect prompt injection (IPI) benchmarks lag behind rapidly evolving agent utility benchmarks in task and environment coverage, and attack scenarios are static and narrow; the authors argue this limits the scale and realism of safety evaluations.
- Proposes meta-benchmarking: using the MetaPI four-module pipeline to automatically convert utility benchmarks into end-to-end IPI evaluations, including injection point identification, owner-authority rule-based goal generation, PAIR-style payload optimization, and real-environment validation.
- Generates 183 attacks on AppWorld, VitaBench, Terminal-Bench and evaluates 8 defenses; the authors state that no defense achieves both strong utility and strong safety, e.g., CaMeL has ASR 0.0% on AppWorld but Clean Uti of only 33.7%.
- Limitations noted by the authors: only the text injection surface is covered, not multimodal; some pipeline stages rely on LLM judgment and are hard to fully verify. The experiments cover 50 initially selected tasks per benchmark for three benchmarks, and reported values are single runs.
- Threat model
- A passive attacker places the payload into external data in the environment before task execution and does not interact with the running agent; a strong attacker model is adopted: it may know the target agent and defense design, the backbone LLM, and the benchmark environment, and in some settings also the user task; it can only modify mutable external data segments it owns or controls. Injection positions are divided into participant (the task depends on its content) and bystander (encountered incidentally); attacks at the participant position must cross the owner-authority boundary.
- Evaluated models
- deepseek-v4-flashgpt-5.6-lunakimi-k3qwen3.7-flashclaude-sonnet-5Meta-SecAlignDataSentinel
- Benchmarks
- AppWorldVitaBenchTerminal-Bench
- Metrics
- ASR (net / effective)Clean UtilityToken usage
Researchers propose MetaPI, which uses a meta-benchmarking approach to transform existing LLM agent capability benchmarks into end-to-end indirect prompt injection (IPI) evaluation suites, reusing their real tasks, environments, and infrastructure. The toolkit can automatically identify plausible injection points, generate realistic attack goals, optimize adaptive attack payloads, and validate attacks in real environments; its modular interface supports plugging in heterogeneous environments, agents, defenses, and attack algorithms without redesigning the evaluation pipeline. The authors apply MetaPI to AppWorld, VitaBench, and Terminal-Bench, evaluating representative agents and defenses across multiple backbone LLMs, and the results show that in realistic, adaptive agent indirect prompt injection scenarios, none of the evaluated defenses achieves both strong utility and strong security.
Related events and news
1 events · 1 news mentions
News that mentions this paper
模安局盘点NeurIPS 2026 AI安全七大研究重点
- 公众号 模安局names the paperNeurIPS 2026 AI安全的七个研究重点
MetaPI: Constructing Prompt Injection Benchmarks from Any Agent Benchmarks
Paper analysis
What problem does this paper attempt to solve?
IPI safety benchmarks lag behind utility benchmarks; the authors propose meta-benchmarking that automatically converts utility benchmarks into IPI evaluation suites.
Core question: how can IPI safety evaluation keep up with rapidly evolving agent utility benchmarks and evaluate defenses in realistic, diverse environments.
- Problem setting: LLM agents use high-privilege tools to complete complex tasks, and malicious instructions embedded in external content (emails, web pages, tool outputs) can cause indirect prompt injection (IPI), leading to unintended high-risk operations (Section 1).
- Shortcomings of existing benchmarks: the authors state that IPI benchmarks are small in scale and slow to evolve—AgentDojo has only 97 benign tasks and 4 in-memory Python simulated environments, whereas utility benchmarks (such as Terminal-Bench, EnterpriseOps-Gym) have developed much faster in task count and environment realism; existing IPI benchmarks have limited support for dynamic attack targets and adaptive payload optimization (Section 1, 5).
- Core observation: IPI is not confined to a specific domain; wherever an agent reads and acts on external content, there is an attack surface; utility benchmarks already provide diverse tasks, tools, stateful environments, and success criteria, and reusing them lets safety evaluation inherit their breadth and continuous evolution (Section 1).
- Threat model: the realism of an attack depends on the data's owner and authority.
- Goal: place a payload at an injection location so that the agent executes a malicious goal that exceeds the owner's authority.
- Knowledge: a strong attacker model—may know the target agent and defense design, the backbone LLM, the benchmark environment, and under some settings the user task.
- Capability: a passive attacker who places the payload before task execution and can only modify mutable external data segments that it owns or controls, and cannot modify content provided by the user or generated by the agent.
- Victim system: an LLM agent running in benchmark environments such as AppWorld, VitaBench, Terminal-Bench that reads external content and calls high-privilege tools.
- What the paper proposes: MetaPI—a scalable toolkit that automatically identifies injection points, generates attack targets under owner-authority rules, optimizes adaptive payloads, and validates end-to-end in real environments (Section 1, 3).
What related research exists?
The related work falls into four groups: utility benchmarks, IPI benchmarks, injection attacks and optimization, and IPI defenses; the authors position their own work within a scalable meta-generation framework.
Agent utility benchmarks
- AppWorld, SWE-Bench, OSWorld, Terminal-Bench, and MCP-type benchmarks (MCP-Universe, MCP-Bench, MCPMark)—provide diverse tasks and realistic environments; MetaPI is compatible with these benchmarks and builds an IPI testbed by leveraging their continuous evolution (Section 5).
IPI benchmarks
- AgentDojo, InjecAgent, ASB, BIPIA, AgentDyn, WASP, PIArena, LivePI—the paper states they are smaller in scale and slower to evolve, with limited support for dynamic attack targets and adaptive payload optimization in multi-step agent environments (Section 5).
- Concurrent work AutoDojo introduces adaptive injection optimization into AgentDojo, and DecodingTrust-Agent (DTap) extends domain and environment coverage; the paper states that both remain tied to predefined benchmark environments and may struggle to keep up with agent capability evolution (Section 5).
Prompt injection attacks and optimization
- Greshake et al. (2023) et al. demonstrated IPI in various LLM applications; optimization-based attacks include PAIR (Chao et al., 2025), adaptive attacks by Nasr et al. (2025), PISmith, and RL-based methods—the paper states that iterative methods are especially effective in complex agent settings, and MetaPI provides a unified interface for such algorithms (Section 3.3, 5).
IPI defenses and security architectures
- Prompt-level (Spotlighting), model-level (Meta-SecAlign), detection-based (PromptArmor, DataSentinel, MELON), system-level (CaMeL, ProGent, etc.); the paper cites several SoK works as surveys and selects representative methods with open-source implementations from each category for experiments (Section 5).
Baselines and benchmarks: the defenses compared in the experiments are eight: Sandwiching, Spotlighting, Meta-SecAlign, DataSentinel, MELON, PromptArmor, CaMeL, DRIFT; the conversion targets are three utility benchmarks: AppWorld, VitaBench, Terminal-Bench (Section 4.1).
Authors' positioning: unlike existing IPI benchmarks tied to fixed safety environments, MetaPI converts existing utility benchmarks into IPI benchmarks, and the authors state it is the first unified IPI evaluation platform spanning the four dimensions of environment, agent, defense, and attack algorithm (Section 1, 5).
How does the paper address this problem?
MetaPI uses a four-module pipeline to convert utility benchmarks into IPI evaluation: connect benchmarks, generate attack strategies, optimize payloads, and validate in real environments.
Overall approach: use the MetaPI four-module pipeline (Figure 1) to reuse the tasks, environments, and criteria of utility benchmarks, automatically generate and end-to-end validate IPI attacks, with modular interfaces supporting heterogeneous environments, agents, defenses, and attack algorithms.
Threat model formalization
- Action classification: necessary write (changes the environment and is required by the task), necessary read (provides information needed for a write), accidental read (all other read actions) (Section 2).
- Data segments and owners: each read observation is split into segments by source, and each segment is associated with an owner; among necessary reads, segments the task depends on are necessary segments, and the rest are accidental segments.
- Injection rules: the injector of an accidental segment is a bystander, and the target is unrestricted; the injector of a necessary segment is a participant, and the target must exceed its legitimate authority (owner-authority rule; Table 1 gives positive and negative examples).
- Target classification: action-deflection (redirecting a write the agent was going to perform) and action-injection (inducing an additional action), orthogonal to the owner dimension.
M1 Benchmark Connector
- Runs the target agent in the benchmark environment and converts execution trajectories into a unified format; connects to different benchmarks via lightweight benchmark adapters (trajectory unifier, environment describer, benchmark-specific prompts) (Section 3.1).
- Implemented as a unified observe–think–act runtime with a 40-step budget, with defenses mounted through six hook stages (Appendix B.1).
M2 Strategy Generator
- Uses an LLM to analyze the agent (analyzer agent, 10 tools, at most 60 model calls) in four stages: identify necessary writes → trace the reads and the owners of the values they depend on → select injection locations → generate targets and scoring rubrics (Appendix B.2).
- Rubrics prefer deterministic judgment (checking tool calls and arguments); only semantic targets use an LLM judge.
- Three quality checks: feasibility (the location is mutable and does not break the observation structure), harmfulness (the target is malicious and the participant location exceeds authority), scoreability (the success condition is specific and does not hold in benign runs); failures are discarded (Section 3.2).
M3 Payload Optimizer
- Black-box iteration: a propose–inject–observe–refine loop, with a unified interface supporting different update rules such as genetic, feedback refinement, and RL (Section 3.3).
- Experiments use a PAIR-style algorithm with an upper limit of 12 rounds and attacker model temperature 0.7; each round replays the benign prefix in the real environment, injects the candidate payload, lets the victim agent act within a 4–10 step window after the injection point, and scores on a four-tier ladder (not called / wrong tool / right tool wrong args / matched), with only matched counted as 1.0 (Appendix B.3).
- Optional adaptive attack: provide the attacker LLM with a description of the target agent and defense mechanism.
M4 Outcome Validator
- Injects the optimized payload into the original environment and reruns from scratch (from-scratch mode, with defenses in place throughout), scoring utility with the benchmark's own protocol and attack success with the rubric (Section 3.4, Appendix B.4).
- All paper experiments use proxy delivery mode (replacing anchors in observation strings); baseline runs that already satisfy the attack criteria are excluded, preventing success without a payload.
What experiments did the paper conduct?
183 attacks generated across three benchmarks, 8 defenses evaluated; the authors state that no defense achieved both strong utility and strong safety.
Experimental setup
- Models tested: the victim agent backbone defaults to deepseek-v4-flash, with gpt-5.6-luna, kimi-k3, qwen3.7-flash, claude-sonnet-5 also used; the Meta-SecAlign defense uses Llama-3.1-8B-Instruct plus the Meta-SecAlign-8B adapter (Appendix G.2).
- Benchmarks: AppWorld, VitaBench, Terminal-Bench, 50 initially selected tasks each; after manual screening, Terminal-Bench retains 30 that enter the pipeline, and 27 benign trajectories are collected (Section 4.1, Table 4).
- Attack corpus: after owner-authority rules and quality checks, 183 attacks in total (AppWorld 51, VitaBench 87, Terminal-Bench 45), covering 107 tasks (Table 2).
- Defenses: Sandwiching, Spotlighting (prompt-level), Meta-SecAlign (model-level), DataSentinel, MELON, PromptArmor (detection-level), CaMeL, DRIFT (system-level), all ported from open-source implementations (Section 4.1, Appendix D).
- Metrics: ASR (attack success rate; for tool-call-type targets, only initiating the malicious call with the required arguments is required, not successful execution), Clean Uti (task completion reward on runs without attacks), Tok (token usage as a multiple relative to None); the main table uses net ASR (with the fixed attack set as denominator), and Appendix H reports effective ASR (excluding attacks confirmed not to have reached the injection point) (Section 4.1, Appendix H).
- Evaluation method: primarily deterministic tool_call rubrics (149/183), the rest use an LLM judge (deepseek-v4-flash, instructed to judge strictly); VitaBench utility is scored by an LLM rubric judge (Appendix B.4, C.2).
- Attack setup: the attacker LLM defaults to deepseek-v4-flash (temperature 0.7), PAIR-style optimization with an upper limit of 12 rounds; reported values are single runs (Appendix G.3).
Main results
According to Table 3 (common task subsets, victim backbone deepseek-v4-flash, net ASR / Clean Uti / Tok):
Swipe horizontally to read the table
Defense AppWorld ASR AppWorld Uti VitaBench ASR VitaBench Uti Terminal-Bench ASR Terminal-Bench Uti None 50.0 82.5 45.9 96.0 44.4 82.2 Sandwiching 20.8 73.6 24.7 88.8 22.2 68.9 Spotlighting 25.0 80.4 20.0 92.6 37.8 77.8 Meta-SecAlign 2.1 22.9 11.8 13.6 N/A N/A DataSentinel 41.7 77.1 43.5 92.5 28.9 73.3 MELON 35.4 65.1 25.9 88.5 26.7 66.7 PromptArmor 35.4 78.0 43.5 90.5 35.6 82.2 CaMeL 0.0 33.7 N/A N/A 0.0 0.0 DRIFT 22.9 70.5 18.8 83.7 8.9 68.9 - The authors state that no defense performs well on both axes: Spotlighting lowers ASR and largely preserves utility but leaves a large residual vulnerability; CaMeL lowers ASR to 0 but with a large utility loss; Meta-SecAlign has low ASR but Clean Uti of only 22.9%/13.6% (its backbone is Llama, so it is not directly comparable to the DeepSeek None row, Table 3 footnote a).
- The authors state that PromptArmor, MELON, Meta-SecAlign, DRIFT reported near-zero ASR with utility close to no defense on benchmarks such as AgentDojo, but perform insufficiently under this setup.
- CaMeL is incompatible with VitaBench's user simulator (Table 3 footnote b); Appendix E.2 notes that its zero ASR also reflects integration and delivery limitations, with many runs never reaching the injection point.
Analysis of injection location and target type
- Injection location: in Figure 3 (AppWorld, deepseek), the average ASR of necessary and accidental locations is similar (annotated on the figure: None row 55 vs 46), and the direction of the difference changes with defense; the authors suggest one possible explanation is that necessary-location content is more relevant but more strictly constrained by owner-authority.
- Target type: in Figure 4, ASR for action-deflection is higher than for action-injection (None row 56 vs 36; similar trends in rows such as CaMeL and DRIFT); the authors state that deflection applies to fewer tasks but is easier to succeed, while injection applies more broadly but is harder to succeed, and both types matter.
Model ablation and optimization process
- Victim/attacker models (Figure 5, AppWorld, no defense): as victims, closed-source models such as gpt-5.6-luna (50%), claude-sonnet-5 (28%), kimi-k3 (24%) and kimi-k3 are more robust, but ASR is still above 20%; as attackers, deepseek (65%), kimi-k3 (53%), claude-sonnet-5 (50%) have higher attack capability, while gpt-5.6-luna is only 29%—the authors state the reason is that its safety alignment makes it refuse to generate injection payloads.
- Optimization convergence: Figures 6 and 7 show the average best-so-far optimization score rising with rounds and converging within a few rounds (the deepseek curve on the figure rises from about 0.35 to above 0.6); 71–76% of attacks reach matched during optimization, with a median of 3–5.5 rounds for those that do, while those that do not use the full 12 rounds (Table 5, Appendix A).
Other ablations and analyses
- Attack composition (Table 5): action-deflection/injection is 104/79; participant owners account for 43% (AppWorld), 47% (VitaBench), 69% (Terminal-Bench); median payload length is 354–480 characters.
- Generation cost (Appendix A, G.3): trajectory collection, strategy generation, and payload optimization for the three benchmarks cost about $20, $28, and $18 respectively, $0.32–$0.39 per attack; defense evaluation costs about $87.
- effective ASR (Table 10): after excluding attacks confirmed not to have reached the injection point, the None row ASR is 58.5% (AppWorld), 47.0% (VitaBench), 52.6% (Terminal-Bench), and the ordering is broadly consistent with net ASR.
- Benign trajectories (Table 4): average benign reward is 0.74 (AppWorld), 0.85 (VitaBench), 0.85 (Terminal-Bench).
What are some points that can be further explored?
The authors list two types of limitations—text constraints and LLM dependence—and three future directions; the experiments cover three benchmarks, five backbones, and a single run.
Limitations and future directions noted by the authors
- Only text-field injection is supported; the multimodal injection surface is not yet covered (Section 6 Limitations).
- Some pipeline stages (dependency analysis, attack target selection) rely heavily on LLMs and are hard to fully verify (Section 6).
- Future directions: extend to a broader attack surface, use MetaPI for model alignment, reduce reliance on LLM judgment (Section 6).
- Appendix G.5 adds: the analysis stage (determining which writes a task requires, the owner of each segment, whether the target exceeds authority) is performed by an LLM under explicit rules and deterministic checks, and can still err; errors change which attacks exist rather than how they are scored; the evaluation covers three benchmarks and their task samples, and each number is a single run.
- Appendix E.2: CaMeL's zero ASR also reflects integration and delivery limitations (many runs never reached the injection point, and the structured raw_response may take precedence over the modified text); the authors retain the reported results but distinguish them from evidence that "every payload was delivered and resisted."
Scope of experimental coverage
- The victim backbone models tested are deepseek-v4-flash, gpt-5.6-luna, kimi-k3, qwen3.7-flash, claude-sonnet-5, 5 in total; the main results table (Table 3) uses only deepseek-v4-flash; the model ablation (Figure 5) is run without defenses (Section 4.1, Appendix G.2).
- The defense evaluation covers 8 open-source defenses in four categories: prompt, model, detection, system; CaMeL is unavailable on VitaBench, and Meta-SecAlign has no result on Terminal-Bench due to OOM (Table 3 footnote).
- The attack corpus is 50 initially selected tasks per benchmark for three benchmarks, and the final 183 attacks cover 107 tasks; Terminal-Bench retains 30 tasks after manual screening (Table 2, 4).
- The optimization budget is a PAIR-style algorithm with an upper limit of 12 rounds; the paper does not report the number of repetitions and states that reported values are single runs with variation between runs (Section 4.1, Appendix G.3).
- The paper discusses why multimodal injection and an attacker interacting during the run do not apply: current injection is limited to text fields of tool observations, and the attacker is a passive attacker who places content before execution (Section 2, Appendix G.5).
Summarize the main content of the paper
MetaPI automatically converts utility benchmarks into IPI evaluation; on three benchmarks, the authors state that no defense maintains both strong utility and strong safety.
MetaPI is a scalable toolkit that automatically converts agent utility benchmarks into end-to-end IPI evaluation suites, addressing the problem that IPI safety benchmarks lag behind utility benchmarks.
- Problem to solve: Existing IPI benchmarks are small in scale, have static scenarios, and lack environmental realism; the authors argue this limits the scale and realism of defense evaluations.
- Method highlights: A four-module pipeline — Benchmark Connector unifies trajectories; Strategy Generator generates attack goals and rubrics according to an owner-authority threat model (distinguishing participant and bystander injection positions); Payload Optimizer uses PAIR-style black-box iterative payload optimization (capped at 12 rounds); Outcome Validator reruns from scratch in the real environment for verification.
- Attack corpus: 183 attacks generated from 50 initially selected tasks each from AppWorld, VitaBench and Terminal-Bench, covering 107 tasks (Table 2).
- Main results (Table 3, deepseek-v4-flash backbone): ASR without defenses was 50.0%/45.9%/44.4%; Spotlighting achieved ASR 20.0% and Clean Uti 92.6% on VitaBench, but with large residual vulnerabilities; CaMeL achieved ASR 0.0% on AppWorld but Clean Uti 33.7%; Meta-SecAlign achieved ASR as low as 2.1% but Clean Uti only 22.9%. The authors state that no defense achieved both strong utility and strong safety.
- Other findings: action-deflection succeeded more easily than action-injection (Figure 4); closed-source models as victims still had ASR above 20% (Figure 5); gpt-5.6-luna as an attacker achieved ASR of only 29%, which the authors attribute to its safety alignment refusals.
- Authors' conclusion: IPI robustness needs to be evaluated in the diverse environments where agents actually operate and under adaptive attacks; the authors hope MetaPI will become an open, scalable foundation for defense evaluation, and plan to release code and configurations (Appendix G.1 notes that the public version is in preparation).