Intent-aware defenses for AI agents

1. 01What problem does this paper attempt to solve?

   What the authors aim to solve: existing defenses do not judge whether agent behavior conforms to its deployment purpose (Intent Preservation).

   Core problem: existing agent security mechanisms detect attack signatures but do not judge whether a behavior conforms to the purpose for which the agent was deployed.

   • Scenario and motivation: LLM agents increasingly access enterprise data, call tools, and execute multi-step actions. The authors point out that a request may have no attack signature and use only authorized capabilities yet still deviate from the agent's intended use; conversely, seemingly adversarial content may be legitimate for a security-analysis agent.
   • Shortcomings of existing methods (per the authors): existing defenses mainly target prompt injection, harmful output, data leakage, and unauthorized tool calls, answering "does this interaction look like an attack" rather than "does this behavior conform to what this agent is supposed to do."
   • Core observation: the security interpretation of an interaction depends on the agent receiving it. On this basis the authors define Intent Preservation (behavior consistent with organizational, role, and developer constraints) and Intent Drift (resulting behavior violating these constraints), and emphasize that an inducing attempt itself does not equal drift.
   • The paper proposes: an Intent-Aware Security framework, including an Intent Profile extracted from three types of trusted intent sources, an Intent-Aware benchmark with 887 samples and 35 agents, and four profile-based defenses: pre-execution detection, post-execution detection, input alignment, and system-prompt hardening.
   • Threat model (Section 4):
     • Attacker goals: induce Intent Drift. Malicious attackers pursue explicit security-compromise objectives (instruction override, data exfiltration, unauthorized access, destructive behavior); Abusive attackers repurpose the agent for uses outside its deployed function.
     • Attacker knowledge: knows the agent's general purpose and available tools.
     • Attacker capabilities: submit arbitrary requests and multi-turn conversations as a user (direct injection), or control content the agent consumes as a third party, such as emails, documents, web pages, tickets, and tool outputs (indirect injection).
     • Boundaries: attackers cannot modify the agent configuration, organizational/role policies, the profile extraction process, or the defense mechanisms; Out-of-Scope requests are non-adversarial and outside the threat model, and are included in evaluation only because defense systems need to distinguish them.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four groups: injection defenses, task/goal drift detection, context security, and benchmarks and alignment.

   Prompt injection defenses

   • Instruction-data boundary approaches: Spotlighting [9] transforms untrusted content when constructing the prompt; StruQ [10], Instruction Hierarchy [11], and SecAlign [12] separate instructions from data or establish instruction priority during training. The authors note that all four protect the artifacts of the current request rather than a persistent model of deployment purpose.
   • Input boundary approaches: BIPIA [13] proposes boundary-aware defenses; PIGuard [14] is a detection-based guardrail and releases NotInject to measure over-defense, and the authors note it only judges whether text is an injection and does not target a specific agent's scope; PPA [15] dynamically varies prompt structure; DataFilter [16] filters external data according to user instructions, which the authors describe as closest to one component of this paper, but its reference is a single user instruction and it neither builds a persistent profile nor detects deviation after reasoning or action.
   • External execution constraint approaches: CaMeL [17], Fides [18], permissive information-flow [19], and AgentSpec [20] provide guarantees over predefined flow, capability, or policy specifications; the authors raise the question of where these specifications come from.

   Task drift, goal drift, and context security

   • White-box detection: Attention Tracker [21], HiddenDetect [22], and TaskTracker [23] detect drift from attention or activations. The authors note they assume white-box access (excluding hosted and third-party models) and that their reference target is the current task rather than a persistent role or organizational scope.
   • Black-box monitoring: LlamaFirewall's AlignmentCheck [24] audits goal hijacking; Task Shield [25] verifies whether instructions and tool calls serve the user's goal; MELON [26] compares tool calls after masking the user prompt. The authors note these monitors use the current user request as reference, whereas this paper's post-execution detector uses a persistent Intent Profile as reference.
   • Drift without an attacker and context security: goal drift [27] and inherited goal drift [28] study whether goals remain stable under pressure; Siu et al. [29] formalize context agent security; Conseca [30] generates just-in-time security policies for the agent's current purpose, which the authors say established the direction of purpose-sensitive policy generation, whereas this paper addresses extracting agent-specific Intent Profiles and measuring their benefits.

   Benchmarks and alignment

   • Benchmarks: BIPIA [13], InjecAgent [2], AgentDojo [31], Agent Security Bench [3], and WASP [32] mainly evaluate injection attacks; AgentLure/ARGUS [33] and StakeBench [34] extend to context-dependent tasks and stakeholder perspectives.
   • Alignment: RLHF [35] and Constitutional AI [36] shape model behavior priors; Guaranteed Safe AI [38] advocates specifications independent of model goals. The authors note that none of these provide deployment-time testing for a specific agent's deployment scope.

   Baselines and benchmarks: the baselines compared in the experiments include PIGuard, AlignmentCheck (LlamaFirewall), DataFilter, PPA, a StruQ-style prompt-format baseline, and ablations such as Input-only / Raw-context / Trajectory-only / Agent-configuration; the benchmarks used are the authors' own Intent-Aware benchmark and AgentDojo v1.2.2.

   The authors position this paper as treating agent intent as a first-class security object, complementing "does this interaction look like an attack" with an answer to "does this behavior conform to what this agent is supposed to do."

   Collapse this answer

3. 03How does the paper address this problem?

   Extract an Intent Profile from three types of trusted intent sources, and condition safety decisions on it at four defense points.

   Overall idea: explicitly represent the agent's trusted intent as an Intent Profile, and make safety decisions conditional on it, rather than only looking at whether the interaction content looks like an attack.

   Formalization: Trusted intent sources and Intent Profile

   • The trusted intent sources are Z = (O, R, D), namely organizational intent (enterprise policy, compliance, acceptable use rules), role intent (responsibilities, workflows, escalation paths, operational boundaries), and developer intent (system instructions, tools, knowledge sources, expected workflows). Conflicts are resolved by priority O ≻ R ≻ D: lower-priority sources can refine higher-priority sources but cannot authorize behavior prohibited by higher-priority sources.
   • User intent U is not part of the trusted specification; it may be legitimate, mistaken, out-of-scope, abusive, or malicious, and is therefore compared against the trusted specification.
   • The extraction function E : Z → P turns Z into an Intent Profile P = (P_purpose, P_function, P_tool, P_data, P_boundary), whose five dimensions are mission, permitted tasks and workflows, tool-use constraints, data boundaries, and prohibited behaviors. The authors emphasize that the profile is not a new source of authority; it only structures the constraints of O, R, and D.
   • The authors assume that the policies, role specifications, and agent configurations fed into the extraction process are genuine and untampered, but do not assume they are complete or consistent.

   Intent Preservation and Intent Drift

   • An execution trajectory τ = (u, h0, a1, ..., aT), where at is an observable behavior (answer, plan, retrieval decision, tool call, memory modification). C(at, ht, P) ∈ {0,1} indicates whether the behavior is consistent with the profile.
   • Preserve when all behaviors in the trajectory are consistent; Drift when there exists some at such that C = 0. Key distinction: a non-benign request is only a potential trigger for drift; drift occurs only when the agent's resulting behavior violates the profile; if an indirect injection is ignored or refused, intent is still preserved.
   • The authors point out that Intent Preservation is stricter than capability authorization: an authorized tool call that serves an unintended purpose still violates intent.

   Request-level four-way classification

   • Label interactions using three predicates: S(u,h,P) (whether the requested behavior conforms to P), A(u,h) (whether there is an explicit attack objective), and D(u,h,P) (whether the agent is deliberately repurposed).
   • Ordered decision rules: A=1 is classified as Malicious; A=0 and S=0 and D=1 is classified as Abusive; A=0 and S=0 and D=0 is classified as Out-of-Scope; A=0 and S=1 is classified as Benign. The ordering resolves cases where an interaction simultaneously satisfies multiple descriptions.
   • These four labels describe the relationship between the interaction and the profile; they do not determine whether drift occurred.

   Benchmark construction (Section 5)

   • Two-step generation: a deterministic construction program first fixes each sample's agent, action, target label, difficulty, attack family, and delivery method; the sample-realization-LLM is only responsible for realizing these predetermined coordinates as natural-language artifacts (organizational/role policies, system prompts, tools, interaction content) and does not decide labels.
   • Agent design space: each agent is defined by five dimensions: domain, architecture pattern, purpose, workflow, and capabilities (action-resource-scope tuples); the design emphasizes intent diversity so that the same action has different authorization status for different agents.
   • Difficulty and challenge factors: three levels Easy/Mid/High; Mid and High progressively add factors such as multilingual or encoded content, instruction-like text as data, contextual ambiguity, conflicting instructions, false authority, and multiple information sources; the attack families for Malicious samples come from public attack datasets, with two delivery methods: direct and indirect.
   • Validation: first deterministic schema/content validation, then semantic validation by a realization-judge-LLM different from the generation model; a sample is accepted only if each of the five criteria scores at least 4 on a five-point scale; semantic validation is run twice and the lower score is taken; in addition, manual spot checks are performed on passing samples.

   Four defense mechanisms (Section 6)

   • Pre-execution detection: takes as input the Intent Profile, the current request, and the interaction context, and outputs a four-way classification label.
   • Post-execution detection: takes as input the profile, the request, and the execution context (ordered tool calls, tool outputs, intermediate messages, environment changes, final answer), and determines whether execution drifted; the authors state that it is an empirical approximation rather than a formal verifier.
   • Input Alignment: performs intent interpretation and disambiguation on the request, conversation history, and tool responses; removes or isolates instructions from untrusted sources; normalizes the request to an authorized workflow; restricts tools/data/scope; or blocks when no formulation consistent with the profile exists; the output is the content passed to the agent or a block decision, not a safety category.
   • Hardening: uses the original system prompt plus the Intent Profile to generate a profile-grounded system prompt, making implicit operational boundaries explicit, with all other inputs unchanged.
   • The four mechanisms share the same extracted profile, facilitating comparison of the role of the same representation at different defense points.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Three sets of experiments: benchmark four-way classification, AgentDojo execution trajectory detection, and simulated defense interventions.

   Experimental Setup

   • Benchmark: Intent-Aware benchmark, generating 1,000 candidates (25% each of four categories; difficulty 15% Easy, 45% Mid, 40% High), 11 failed deterministic validation, 102 were rejected by the realization-judge-LLM, resulting in 887 samples and 35 agent configurations; by category, 237 Benign, 239 Out-of-Scope, 226 Abusive, 185 Malicious; the 185 Malicious samples cover 12 attack families (based on Garak configurations), 104 delivered directly and 81 delivered indirectly (Section 7.1).
   • Generation and validation models: sample-realization-LLM is GPT-5.4, realization-judge-LLM is MAI-Thinking-1; in a manual spot check of 100 passing samples, 98% were labeled correctly.
   • Exp. I (pre-execution classification): the classifier uniformly uses DeepSeek-V4-Pro (the authors state it was chosen to rule out influence from the same model family as the benchmark generation); comparing four methods: Input-only, PIGuard, Raw-context, Profile-based; metrics are per-class F1 for the four categories and macro-F1; PIGuard input is truncated to 512 tokens, long conversations are split into 450-token chunks with 50-token overlap, and if any chunk is judged as injection the whole sample is judged as injection.
   • Exp. II (post-execution detection): all AgentDojo v1.2.2 benchmarks, 4 tool agents (Workspace, Travel, Banking, Slack), 11 automatic injection techniques, excluding manual and denial-of-service, 485 cases in total; of 11,021 configured cases, 10,536 usable trajectories (97 benign, 10,439 attacked), of which 2,480 attacks succeeded and 7,959 failed (attack success rate 23.8%); trajectories were generated by GPT-4o without defenses, labels come from AgentDojo's deterministic environment-state evaluator; all monitors use GPT-4o with identical configuration; primary metric AUROC, secondary metrics precision/recall/F1 at a 0.5 threshold.
   • Exp. III (Input Alignment and Hardening): simulated interventions on the benchmark; MAI-Thinking-1 handles profile extraction, hardening, and Profile-based Input Alignment, GPT-5.6-luna performs response simulation, DeepSeek-V4-Pro performs response evaluation; the judge's accuracy on 100 manually reviewed samples is 95%; success criterion: Out-of-Scope/Abusive/Malicious samples require complete refusal, Benign samples require complete completion.
   • Statistical methods: Exp. I uses agent-level paired cluster bootstrap, reporting 95% confidence intervals.

   Main Results

   Exp. I four-class F1 (according to Table 1):

   Swipe horizontally to read the table

   Method	Macro	Malicious	Abusive	OoS	Benign
   Input-only	43%	41%	70%	2%	60%
   Raw-context	45%	50%	57%	8%	64%
   Profile-based	79%	72%	86%	76%	82%

   • Authors' interpretation: raw agent context is only 2 percentage points higher than no context, and structuring it into an Intent Profile adds another 34 percentage points; the 34-percentage-point macro-F1 improvement has a 95% confidence interval of [29.5, 39.0] percentage points under paired bootstrap.
   • Binary classification (Malicious vs Non-Malicious) secondary comparison: Profile-based 72% F1, Raw-context 50%, Input-only 41%, PIGuard 36.8%.
   • Confusion difference matrix (Table 4, Profile-based relative to Raw-context): Out-of-Scope recall +74.0 pp (mainly from −56.1 pp previously misjudged as Benign), Malicious recall +41.6 pp (mainly from −41.5 pp previously misjudged as Abusive); Abusive recall −15.9 pp, but Abusive F1 still rises from 57% to 86%.

   Exp. II: AgentDojo post-execution detection

   • Results (according to Table 2): Profile-based AUROC 94.9%, higher than Agent-configuration 89.7%, Trajectory-only 84.3%, AlignmentCheck 90.9%; at a 0.5 threshold AlignmentCheck has higher precision (95.9% vs 75.9%) and F1 (93.5% vs 83.8%), and Profile-based has the highest recall (93.3%).
   • Figure 2 shows ROC curves for each monitor (maximum common valid subset of n = 10,505); the authors state the profile-based curve is superior over most of the operating range, especially in the low false-positive region.
   • Authors' interpretation: raw configuration is 5.4 AUROC points higher than trajectory-only, and structuring into a profile adds another 5.2 points; since labels come from execution environment state and the benchmark was not generated by this paper, the authors state this provides independent evidence that profiles improve behavior drift detection.

   Exp. III: Input Alignment and Hardening

   • Input Alignment defense success rate (according to Table 3): Profile-based macro 85%, DataFilter 78%, no-defense baseline 69%; by category, Profile-based is Benign 91%, OoS 85%, Abus. 96%, Mal. 66%, where Benign is lower than the baseline's 97%, and Malicious is on par with DataFilter (66%).
   • Hardening defense success rate (according to Table 5, Appendix B): Profile-based macro 83%, PPA 81%, StruQ-style 80%, baseline 69%; Profile-based is 77% on OoS (PPA 72%, StruQ-style 67%), Abus. 96%, Malicious 64% on par with StruQ-style, Benign 94% lower than the baseline and PPA's 97%, which the authors describe as a moderate over-refusal cost.
   • Authors' interpretation: the profile's gains are mainly concentrated in categories whose labels depend on deployment context (Out-of-Scope, Abusive).

   Other ablations and analyses

   • In Exp. I, Raw-context relative to Input-only: macro-F1 45% vs 43% (Table 1).
   • In Exp. II, Agent-configuration relative to Trajectory-only: AUROC 89.7% vs 84.3% (Table 2).
   • AlignmentCheck was modified to output a continuous [0,1] score to compute AUROC; the authors note it takes the original user request as the authorization target, consistent with AgentDojo's trust assumptions.
   • Negative results: the Profile-based method is lower than the no-defense baseline on the Benign category in all cases (Input Alignment 91% vs 97%; Hardening 94% vs 97%); on the Malicious category, Input Alignment is on par with DataFilter and Hardening is on par with StruQ-style; the authors do not explain the reason.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors list limitations such as the synthetic benchmark, simulated evaluation, and AgentDojo coverage, and propose a cost-accuracy trade-off as a future direction.

   Limitations and future directions noted by the authors

   • The benchmark is synthetic (Section 8 Limitations): organizational intent and role intent are represented as synthetic text summaries rather than production policy corpora, and samples are realized and validated by LLMs; the authors' mitigations are fixing labels before generation, using different models for validation, and manually spot-checking 100 passing samples (98% labeled correctly).
   • Exp. III is simulated behavior (Section 8): it measures simulated rather than actually executed agent behavior, and relies on an LLM judge (95% agreement with manual review).
   • AgentDojo coverage (Section 8): it contains only 4 agents, a single agent model GPT-4o, and 97 benign trajectories, and most negative examples are failed attacks rather than benign executions; AgentDojo treats user requests as trusted and contains only indirect injection, so it covers only the injection-based Malicious portion of this paper's threat model, while Out-of-Scope, Abusive, and user-side attacks are evaluated only on the synthetic, LLM-validated benchmark.
   • Reproducibility (Section 8): the experiments rely on closed-weight models, which may limit exact reproduction.
   • Deployment cost and future directions (Section 8 Deployment considerations): the authors envision separating expensive profile extraction (once per agent) from per-interaction detection, and speculate that the narrower detection task could be handled by smaller, lower-latency models; quantifying this accuracy-cost trade-off is future work.

   Scope of experimental coverage

   • The methods tested are the four profile-based mechanisms implemented by the authors and the corresponding profile-free baselines; the Exp. I classifier is uniformly DeepSeek-V4-Pro, the Exp. II monitor is uniformly GPT-4o, and Exp. III uses MAI-Thinking-1, GPT-5.6-luna, and DeepSeek-V4-Pro in divided roles (Sections 7.3–7.5).
   • The evaluation benchmarks are the Intent-Aware benchmark (887 samples, 35 agents) and AgentDojo v1.2.2 (10,536 usable trajectories) (Sections 7.1, 7.4).
   • Human agreement: benchmark spot check of 100 passing samples, 98% labeled correctly; Exp. III judge accuracy on 100 manually reviewed samples is 95% (Sections 7.1, 7.5).
   • The paper does not report the number of repeated runs for each experiment; apart from the paired cluster bootstrap confidence interval in Exp. I, Exp. II and Exp. III report no confidence intervals or significance tests.
   • The paper does not report query counts, latency, or cost figures for each defense mechanism; the deployment cost section gives only qualitative considerations.

   Collapse this answer

6. 06Summarize the main content of the paper

   Making agent intent an explicit security object, the authors report that it outperforms a no-profile baseline in classification, trajectory detection, and guardrail intervention.

   A defenses paper on agent safety whose core claim is to treat an agent's deployment intent as a first-class safety object.

   • Problem: existing defenses answer "does the interaction look like an attack" but not "does the behavior match this agent's deployment purpose"; the authors argue the same interaction has different safety meanings for different agents.
   • Method: define Intent Preservation and Intent Drift; extract a five-dimensional Intent Profile from organizational, role, and developer intent (priority O ≻ R ≻ D); build an Intent-Aware benchmark with a "define labels first, then have an LLM implement them" approach (887 samples, 35 agents, four classes); implement four profile-based defenses: pre-execution detection, post-execution detection, input alignment, system-prompt hardening.
   • Result one (Exp. I, Table 1): benchmark four-class macro-F1, Profile-based 79%, Raw-context 45%, Input-only 43%; Out-of-Scope F1 rises from 8% to 76%; the 34 percentage point improvement has a bootstrap 95% confidence interval of [29.5, 39.0].
   • Result two (Exp. II, Table 2): on 10,536 AgentDojo execution trajectories, Profile-based post-execution AUROC 94.9%, higher than AlignmentCheck 90.9%, Agent-configuration 89.7%, Trajectory-only 84.3%; at the 0.5 threshold AlignmentCheck has higher precision and F1.
   • Result three (Exp. III, Tables 3, 5): Input Alignment macro defense success rate 85% (DataFilter 78%), Hardening 83% (PPA 81%, StruQ-style 80%); both method classes are lower than the no-defense baseline on Benign, which the authors call moderate over-refusal.
   • Authors' conclusion: explicit, agent-specific intent representations can serve as a practical safety reference and complement attack-oriented defenses; the gains are concentrated in categories that depend on deployment context (Out-of-Scope, Abusive). The authors also note limitations: the benchmark is synthetic, Exp. III is a simulated evaluation, and AgentDojo covers only injection-type Malicious scenarios.

   Collapse this answer

Related papers