CaMeL system defending against prompt injection by design

1. 01What problem does this paper attempt to solve?

   The authors aim to address the problem of prompt injection attacks and private data leakage when LLM agents process untrusted data.

   LLM agents are highly susceptible to prompt injection attacks when processing untrusted external data, leading to malicious instructions hijacking the execution flow and leaking private data.

   • Scenario and importance: Modern large language models are widely used to build autonomous agents that interact with the external environment through APIs and interfaces. When processing data from untrusted sources (such as web pages, third-party tool return values, or shared files), malicious actors can inject instructions to induce the system to execute unauthorized operations or exfiltrate private data (Introduction, Section 2).
   • Shortcomings of existing defenses: The authors point out that existing defenses based on prompt delimiters, prompt sandwiching, or safety fine-tuning are all heuristic approaches lacking deterministic security guarantees; simple Dual LLM isolation, while limiting control flow planning, still allows attackers to cause malicious operations by manipulating data flow (such as replacing tool call parameters) (Section 2, Section 3).
   • Core observation and hypothesis: Agent actions include both control flow and data flow. The authors argue that security constraints must be applied in both the control flow and data flow dimensions. By binding metadata (capabilities) to data values and maintaining a dependency graph, fine-grained information flow security policy checks can be implemented at the system level (Section 2, Section 5).
   • Proposed approach: The authors propose the CaMeL (CApabilities for MachinE Learning) defense framework, where a privileged LLM extracts control flow and generates Python code, an isolated quarantine LLM only parses untrusted data, and a custom interpreter maintains the data flow graph and enforces security policies based on capabilities, providing security guarantees without modifying the underlying model (Section 1, Section 5).
   • Threat model:
     • Attacker goal: In the PI-SEC security game, the attacker aims to construct malicious states that cause the agent to execute unsafe actions outside the allowed action set (Omega_prompt), such as data exfiltration or malicious state changes (Section 3, Section 4).
     • Attacker knowledge and capabilities: The attacker can inject malicious content (prompt injection) into external data sources accessible to the agent (such as emails, documents, spreadsheets), but cannot directly modify the user's initial input (Section 3).
     • Victim system: An LLM-based agent system with tool-calling capabilities, possessing read/write access to external APIs and persistent state (Section 3, Section 4).
     • Explicit non-goals: The authors state that they do not defend against plain-text attacks that do not affect control flow and data flow (such as inducing false summaries) or plain-text phishing attacks; they assume the user prompt and memory have not been compromised (Section 3.1).

   Collapse this answer

2. 02What related research exists?

   The authors systematically review heuristic prompt defenses, dual-LLM isolation architectures, and traditional information flow security theory, and compare existing baselines.

   The paper discusses prompt injection defense, isolation architectures, and information flow security, with related research mainly falling into three categories:

   • Heuristic and model-layer prompt injection defenses:
     • Hines et al. (2024) proposed Spotlighting, which uses special characters to mark untrusted context boundaries and instructs the model to ignore instructions within them; Learn Prompting (2024) proposed Prompt Sandwiching, which repeatedly displays the original task after each tool output. The authors state these methods are heuristic and lack security guarantees.
     • Wallace et al. (2024) and Wu et al. (2024) fine-tune models to ignore untrusted instructions through Instruction Hierarchy; Chen et al. (2024) (StruQ) trains models to follow structured queries. The authors state these methods remain probabilistic approaches and can still be broken in benchmark tests.
     • Abdelnabi et al. (2024) monitor internal activation vectors to detect task drift; Sharma et al. (2024) (SPML) uses a formal language to define tasks; Zverev et al. (2025) (ASIDE) separates instructions and data at the architecture level using different input embeddings; ProtectAI (2024) and Debenedetti et al. (2024a) propose detecting injection and exfiltration through classifiers or canaries.
   • Isolation architectures and system-level defenses:
     • Willison (2023) proposed the Dual LLM pattern, using a privileged LLM to process trusted prompts and plan tool calls, and a quarantine LLM to process untrusted data without tool-calling permissions. The authors point out its limitation is the inability to defend against data flow tampering (parameter manipulation).
     • Wu et al. (2025) (IsolateGPT) defends against untrusted tools through isolation, but all cross-tool access to tool outputs requires explicit user approval, which the authors point out can easily lead to user fatigue.
     • Wu, Cecchetti, and Xiao (2024) use information flow control (IFC) to track trusted parts of tool outputs to decide whether to feed them to the model.
   • Traditional software security concepts and recent concurrent work:
     • Classic concepts borrowed: control flow integrity (CFI, Abadi et al., 2009), access control (Anderson, 2010), information flow control (IFC, Denning, 1976; Denning and Denning, 1977; Myers and Liskov, 1997), and hardware/system capability mechanisms (Needham and Walker, 1977; Watson et al., 2010; Woodruff et al., 2014; Watson et al., 2015).
     • Concurrent work: Zhong et al. (2025) (RTBAS) uses coarse-grained labels and classifiers to infer text dependency relationships, which the authors point out has less expressive policies than CaMeL; Abdelnabi et al. (2025) defines policies for inter-agent communication based on synthetic data; Kim, Choi, and Lee (2025), Li et al. (2025) (ACE), and Costa et al. (2025) proposed dual-model and tool-call policy checking mechanisms with blueprints similar to CaMeL.
   • Comparison baselines and benchmarks:
     • Evaluation benchmark: The AgentDojo benchmark (Debenedetti et al., 2024b), containing 4 suites: Workspace, Banking, Slack, and Travel, with a total of 949 attack tests.
     • Comparison baselines: Native tool calling (Native Tool Calling API / Undefended model), Tool Filter (Debenedetti et al., 2024b), Spotlighting (Hines et al., 2024), and Prompt Sandwiching (Learn Prompting, 2024).

   The authors position CaMeL as a system-level defense based on software security principles, not relying on model behavior modification or probabilistic heuristic rules, but rather providing deterministic security guarantees through isolating control flow, precisely tracking the data flow graph, and enforcing fine-grained capability policies.

   Collapse this answer

3. 03How does the paper address this problem?

   CaMeL achieves isolation through a privileged planning model and a quarantine parsing model, with an interpreter tracking data dependencies and enforcing fine-grained capability policies.

   CaMeL draws on control-flow integrity (CFI), access control, and information flow control (IFC) from traditional software security to build a system-level protection layer wrapped around the LLM.

   Security Game Formalization and Overall Architecture

   • PI-SEC Security Game: The paper formalizes the threat model as a game where the attacker provides an initial malicious state mem*, the agent executes the user prompt and produces a tool call trace Trace, and the verifier checks whether every triple in the trace is within the allowed set:

   Verify(Trace, \Omega) = \bigwedge_{(tool, args, mem_{step}) \in Trace} ((tool, args, mem_{step}) \in \Omega) This formula indicates that the verifier checks whether each tool call, its arguments, and memory state in the execution trace belong to the allowed action set Ω (Section 4, Figure 3).

   • Core System Components: The system includes a privileged LLM (P-LLM), a quarantine LLM (Q-LLM), an external tool set, security policies, capabilities, and a CaMeL custom interpreter (Section 5, Figure 5).

   Dual-Model Isolation and Structured Data Processing

   • Privileged LLM (P-LLM): Responsible for converting user natural language requirements into Python code expressing the execution plan. P-LLM only receives trusted user prompts and tool function signatures, and never touches data returned by external tools or Q-LLM output content (Section 5.1).
   • Quarantine LLM (Q-LLM): Has no tool invocation permissions, and is called as a function (query_ai_assistant) by code generated by P-LLM, specifically using the LLM's built-in structured output capabilities to parse untrusted unstructured data into Pydantic BaseModel instances (Section 5.1).
   • Information Sufficiency Feedback: The system injects a boolean field have_enough_information into Q-LLM's output schema. When information is insufficient, it returns false, and the interpreter throws a NotEnoughInformationError, prompting P-LLM to regenerate and fix the code, but Q-LLM is strictly forbidden from telling P-LLM what specific information is missing to prevent injection (Section 5.1).

   Custom Interpreter and Dataflow Dependency Tracking

   • Restricted Python Dialect: Implemented using Python's built-in ast library for recursive interpretation. Strictly prohibits while loops, generators, module imports, eval/exec, break/continue, function and lambda definitions, and methods with in-place modification side effects (e.g., list.append), enforcing functional styles like comprehensions (Section 5.4, Appendix H.1).
   • Dynamic Dependency Graph Maintenance: The interpreter tracks data dependencies for each variable (e.g., c = a + b means c recursively depends on a and b). The system provides NORMAL and STRICT interpretation modes: in STRICT mode, all variables assigned inside control flow branches (if/for) depend on the branch condition or iteration object, and all statements executed after a Q-LLM call depend on that call's input parameters (Section 5.4, Section 7, Figure 7).

   Capability-Based Security Policy Enforcement and Exception Control

   • Capability Metadata: Each data value is bound with metadata tags recording the data source (User, CaMeL transformation, specific Tool, and internal sources like email sender) and the set of allowed readers (Public or user email list) (Section 5.3).
   • Security Policies in Python: Security policies are defined directly as Python functions (not a custom DSL), recursively checking arguments and their dependency graphs before tool execution (e.g., verifying recipients are in the allowed reader set before sending external emails). Compliance returns Allowed(), violations return Denied() and halt execution (Section 5.2, Figure 6).
   • Security Error Handling: The interpreter allows P-LLM to retry up to 10 times to fix exceptions. If an exception depends on untrusted data, error details are automatically redacted to prevent error content from becoming a prompt injection channel; side effects from earlier partial execution cannot be rolled back (Section 5.4, Appendix H.2).

   Collapse this answer

4. 04What experiments did the paper conduct?

   In AgentDojo evaluations, CaMeL eliminates injection attacks on multiple models, with utility well preserved in most suites except Travel.

   Experimental Setup

   • Models under test: Gemini 2.5 Flash (gemini-2.5-flash-preview-05-20), Gemini 2.5 Pro (gemini-2.5-pro-preview-05-06), Claude 3.5 Haiku (claude-3-5-haiku-20241022), Claude 4 Sonnet (claude-sonnet-4-20241022 without reasoning), Claude 4 Sonnet* (with 16,000 reasoning tokens budget), GPT-4.1 (gpt-4.1-2025-04-14), o4-mini (o4-mini-2025-04-16 high reasoning effort), o3 (o3-2025-04-16 high reasoning effort), Claude 3.5 Sonnet, GPT-4o Mini, GPT 4.1 Nano (Section 6.1, Section 6.3).
   • Datasets and benchmarks: AgentDojo benchmark, covering 4 suites: Workspace (40 tasks), Banking (16 tasks), Slack (20 tasks), Travel (20 tasks), including 949 attack tests (Figure 9, Table 1).
   • Comparison baselines: Native Tool Calling API (official native tool calling/undefended models for each model), Tool Filter, Spotlighting, Prompt Sandwiching (Section 6.1, Section 6.3).
   • Primary metrics: Benign task utility (Utility, task success rate), Utility under attack, Number of successful attacks (out of 949), Policy Triggering Rate, Input/Output token amplification factor (Section 6.1, Section 6.2, Section 6.5).
   • Evaluation and judgment method: Automated judgment using the AgentDojo environment (not LLM-based review), checking task completion or attack success based on environment state and goal assertions (Section 6.2.1, Appendix H.4).
   • Sample size and statistics: Full test of 949 attacks, tables report means and standard deviations, the paper does not report the number of random seeds for repeated runs (Table 2, Table 4).

   Main Results

   The table below shows the comparison of native tool calling and CaMeL defense performance for the main models under test on AgentDojo (per Table 4, Table 2):

   Swipe horizontally to read the table

   Model	Native API successful attacks	CaMeL (no policy) successful attacks	CaMeL successful attacks	Native API overall utility	CaMeL overall utility
   Claude 4 Sonnet	75 ± 58.6	13 ± 6.0	11 ± 4.5	86.6% ± 6.8	74.2% ± 8.7
   Claude 4 Sonnet*	75 ± 58.6	10 ± 3.8	11 ± 4.5	83.5% ± 7.4	70.1% ± 9.1
   Gemini 2.5 Flash	297 ± 268.7	1 ± 0.0	1 ± 0.0	55.7% ± 9.9	35.1% ± 9.5
   Gemini 2.5 Pro	163 ± 140.1	0 ± 0.0	0 ± 0.0	73.2% ± 8.8	41.2% ± 9.8
   o3 High	11 ± 4.5	1 ± 0.0	0 ± 0.0	84.5% ± 7.2	77.3% ± 8.3
   o4 Mini High	2 ± 0.0	1 ± 0.0	1 ± 0.0	79.4% ± 8.1	76.3% ± 8.5

   • Attack defense effectiveness: The authors state that even without enabling safety policies (relying only on model isolation), the number of successful attacks is greatly reduced; with full policies enabled, the number of successful attacks for Gemini 2.5 Pro and o3 High drops completely to 0 (Table 4).
   • Attribution of residual successful attacks: The authors note that the very few residual attacks judged successful by AgentDojo (e.g., 11 for Claude 4 Sonnet, all concentrated in the Travel suite) are not due to control flow or data flow hijacking, but rather the model calling print to output review text containing attack words, which triggers the benchmark evaluation rules; this falls under pure text rather than a prompt injection vulnerability (Section 6.2.1, Section 6.2.2).
   • Benign utility performance: The authors note that advanced models using CaMeL experience relatively small utility loss (e.g., o3 High drops from 84.5% to 77.3%, o4 Mini High from 79.4% to 76.3%); however, there is a noticeable utility drop on Gemini models, mainly due to insufficient specification of the Travel suite API output format (Table 2).

   Comparison with Existing Defense Baselines (Claude 3.5 Sonnet)

   • Evaluation content: Comparison of CaMeL, no-policy CaMeL, Spotlighting, Tool Filter, Prompt Sandwiching, and undefended models on Claude 3.5 Sonnet under 949 attacks for safety and utility (Section 6.3, Table 5, Table 7).
   • Experimental results: Undefended model has 44 ± 31.3 successful attacks, Prompt Sandwiching 24 ± 14.5, Tool Filter 8 ± 2.5, Spotlighting 5 ± 0.6, CaMeL 0 ± 0.0 (Table 7); for utility, undefended model is 90.72% ± 5.77, Spotlighting 92.78% ± 5.15, Prompt Sandwiching 89.69% ± 6.05, Tool Filter 73.20% ± 8.81, CaMeL 63.92% ± 9.56 (Table 5). On GPT-4o Mini, native tool calling suffers 276 successful attacks, while CaMeL reduces it to 0 (Section 6.3).
   • Author interpretation: The authors state that CaMeL significantly outperforms all compared defenses, providing verifiable safety guarantees, while other heuristic defenses are prone to robustness collapse under adaptive attacks.

   Failure Mode and Utility Impact Analysis

   • Evaluation content: Systematic classification of failure attributions for Claude 3.5 Sonnet across 96 tasks in 4 suites (Section 6.1.2, Table 1, Table 14).
   • Experimental results: A total of 8 failure modes are recorded (Table 1), among which "API not sufficiently specified" accounts for 13 occurrences (all in the Travel suite); "Data requires action" accounts for 5; "Q-LLM insufficient context" accounts for 5; "Q-LLM over-processing/strict evaluation" accounts for 3; "P-LLM incorrect assumption" accounts for 3; "AgentDojo evaluation bug" accounts for 3; "Query misunderstanding" accounts for 1; "Task ambiguity" accounts for 1.
   • Author interpretation: The authors note that "Data requires action" is an inherent limitation that dual-LLM isolation cannot solve with a single forward planning pass; newer models, by parsing unknown structures via Q-LLM, improve Claude series utility in the Travel suite from 25% for 3.5 to 55% for 3.7 and 75% for 4 (Section 6.1.2).

   Runtime Overhead and Cheap Model Substitution Experiments

   • Evaluation content: Measurement of token overhead introduced by CaMeL, and the impact of using lightweight models as Q-LLM on system utility and cost (Section 6.1.3, Section 6.5, Figure 13, Table 10).
   • Experimental results: Measured with Claude 3.5 Sonnet, in attacked tasks CaMeL consumes a median of 2.82× input tokens and 2.73× output tokens (Figure 13, Table 10); using Claude 3.5 Haiku as Q-LLM with Claude 4 Sonnet as P-LLM reduces system utility by only about 1%, with median cost savings estimated at 12% (Section 6.1.3).
   • Author interpretation: The authors believe the parsing task undertaken by Q-LLM is of low complexity; using lightweight or even local models is not only cost-effective but also avoids third-party service providers reading tool outputs, enhancing privacy.

   Other Ablations and Analyses

   • NORMAL vs STRICT mode policy triggering rates: when not attacked, overall triggering rate is 33.87% for NORMAL and 53.23% for STRICT (per Table 8); when attacked, overall triggering rate is 26.54% for NORMAL and 45.34% for STRICT (per Table 9).
   • Overall utility under attack (Claude 3.5 Sonnet): CaMeL is 66.70% ± 3.00, native model is 86.83% ± 2.15, Spotlighting is 89.36% ± 1.96 (per Table 6).
   • Claude 4 Sonnet per-suite utility under attack: CaMeL is 68.8% ± 7.6 in Banking, 68.6% ± 8.9 in Slack, 75.0% ± 7.2 in Travel, 78.9% ± 3.4 in Workspace (per Table 3).
   • Policy triggering rate suite distribution (Claude 4 Sonnet*): under benign tasks, Workspace is 0.42, Banking is 0.58, Slack is 0.94, Travel is 0.30 (per Figure 10a).
   • Baseline defense median token increase (not attacked): Spotlighting input 1.08×, output 0.99×; Tool Filter input 7.77×, output 1.05×; Prompt Sandwiching input 1.14×, output 1.16× (per Table 10).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out limitations and future directions including high ecosystem adaptation costs, policy maintenance and user fatigue, side-channel leakage risks, and language complexity.

   Limitations and future directions identified by the authors

   • High implementation and ecosystem adaptation costs: Building a capability-based system requires significant development and refactoring costs, and requires all external tools and third-party services in the environment to support the capability mechanism; otherwise, utility will decrease (Section 9.1).
   • Security policy maintenance burden and user fatigue: The system requires writing, encoding, and maintaining fine-grained security policies; when policies are too strict or ambiguous, frequent user confirmation requests (downgrade/declassification processes) can easily lead to alert fatigue, causing users to unconsciously approve malicious operations (Section 9.2, Section 9.3).
   • Side-channel information leakage risks: The system still has side-channel risks such as exploiting indirect dependency inference (e.g., looping to pull external resources based on private variables) and causing execution interruption through induced exceptions to leak single-bit information about private variables (Section 7, Section 9.3).
   • Possible small control flow concatenation evasion: Similar to how traditional control flow integrity is bypassed by return-oriented programming (ROP), attackers may construct a series of small legitimate operations that comply with policies and propagate them through the data flow, thereby piecing together a malicious control flow (Section 6.4, Section 9.3).
   • Python host language complexity: The dynamic features of the Python language are complex and its exception mechanism can easily introduce side channels. The authors propose that future work could explore functional languages (such as Haskell) that handle errors and I/O more explicitly as the interpreter target language, to reduce the difficulty of resolving security policy conflicts (Section 10).
   • Formal verification and automated policy generation: The authors note that the current implementation mainly provides empirical evidence; future work needs to formally verify the interpreter and security properties to prove it is vulnerability-free, and explore combining AirGap or DSL to automatically generate context-dependent security policies (Section 10).

   Experimental coverage

   • Model scope tested: The experiments covered 10 models: Gemini 2.5 Flash/Pro, Claude 4 Sonnet (including reasoning version), o3 High, o4 Mini High, Claude 3.5 Sonnet, Claude 3.5 Haiku, GPT-4o Mini, and GPT 4.1 Nano (Section 6.1, Section 6.3).
   • Test environment and attack scale: The experiments were based on the AgentDojo benchmark, covering 4 scenario suites: Workspace, Banking, Slack, and Travel, with a total of 949 attack samples (Section 6.1, Section 6.2).
   • Compared defense baselines: Baseline evaluations covered 4 approaches: native tool calling, Spotlighting, Tool Filter, and Prompt Sandwiching (Section 6.3).
   • Ablation condition settings: Ablation experiments covered CaMeL without security policies, and a comparison of the interpreter's NORMAL mode and STRICT mode (Section 6.2.1, Section 6.2.2, Appendix D).
   • Unreported information: The paper did not report the specific number of independent random repetitions and random seed settings for each task, nor did it report specific network latency data when each model calls external APIs (Section 6.1, Section 6.5).

   Collapse this answer

6. 06Summarize the main content of the paper

   The paper proposes the CaMeL defense framework based on capabilities and isolation, aiming to eliminate prompt injection risks in AgentDojo through deterministic system design.

   The paper proposes an agent defense system called CaMeL, based on traditional software security principles, aiming to address prompt injection and data exfiltration problems faced by large language models through architecture-level design rather than model fine-tuning.

   • Core positioning and challenges: When agents process untrusted external inputs such as web pages, documents, or emails, malicious injected instructions may tamper with parameters or hijack behavior. The authors point out that mainstream prompt engineering and fine-tuning approaches are all heuristic protections that cannot provide deterministic security guarantees; simply isolating the planning model also cannot prevent attackers from manipulating tool arguments by polluting the data flow.
   • CaMeL defense mechanism: The system decomposes tasks into a dual-model architecture. The privileged model only accesses trusted user initial instructions and generates a restricted Python program representing control flow; the quarantine model has no tool-calling permissions and is only used to parse untrusted text into structured data conforming to Pydantic specifications. The custom interpreter tracks the data dependency graph at runtime and maintains capability labels. Before calling tools, security policies check the data source and recipient permissions, and violating operations are directly blocked.
   • Main experimental results:
     1. In AgentDojo's 949 attack evaluations, CaMeL reduced the successful attack counts for Gemini 2.5 Pro and o3 High from 163 and 11 with native calls to 0 (Table 4).
     2. On Claude 3.5 Sonnet, CaMeL achieved 0 successful attacks, outperforming the baselines Tool Filter's 8 and Spotlighting's 5 (Table 7).
     3. In benign tasks, o3 High with CaMeL achieved 77.3% overall utility, and o4 Mini High achieved 76.3%, maintaining high usability compared to 84.5% and 79.4% with native calls (Table 2).
     4. When using the inexpensive Claude 3.5 Haiku as the quarantine model, the overall utility of the Claude 4 Sonnet system decreased by only about 1%, with median cost expected to decrease by 12% (Section 6.1.3).
   • Limitations and security insights: The authors point out that when agents need to dynamically decide next actions based on unread data ("data-dependent actions"), pure forward planning has inherent limitations; the system also faces side-channel leakage risks such as exception handling and dependency inference, and manual configuration of security policies poses a user fatigue risk. The authors believe that building system-level security protections around untrusted models is a practical path to enhancing agent robustness.

   Collapse this answer

Related papers