Skip to content

PROVISO: Provenance-Aware Action Selection for Safe LLM Agents

PROVISO: Provenance-Informed Action Selection for Secure LLM Agents

arXiv: Jailbreaks, Prompt Injection, Poisoning & Defenses· arXiv:2610.27115· Bo Yan, Qi Zhang, Ziwen Kan, Fnu Suya, Song WangUniversity of Central Florida; University of Tennessee, KnoxvilleIndexed by this site · 2026-10-21· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

PROVISO compares candidate actions by source at commitment points and submits the one with higher value; the authors claim that on four tasks no baseline exceeds it on both safety and utility.

Threat modelAt the commitment point, the controller chooses between P0 and P1: P0 is the action proposed by the policy, and P1 is refusal or another continuation. It sees K (request, policy, tool interfaces) and G (tool returns, documents, skill packages), as well as sandbox rollouts before the commitment; native scores are used only for training and evaluation. At admission, P1 is refusal. The paper does not state the attacker's knowledge level.

Questions
Agents act through tools and third-party skills, and a single commit may be irreversible. Existing defenses do not track whether content comes from the user or from outside, nor compare with alternative actions, so they trade off safety against utility.
Methods
After the policy proposes an action and before execution, PROVISO writes trusted observations, external observations, the proposed action, and the alternative action into memory with source roles, compares them using role queries, and submits the action with the higher value after calibration on the development set.
Experiments and results
The authors state that on the four tasks no baseline exceeds PROVISO on both safety and utility. Relative to the strongest baseline, the F1 gain is 13.84 points on SkillsBench and 7.07 points on ToolSafe. On AgentDojo, PROVISO's targeted ASR is 0.28%, and lower on BIPIA.
Limitations and future work
The authors write that the controller is fitted separately per benchmark and that cross-task transfer was not studied; candidate selection requires first rolling out both continuations. Evaluation uses published benchmarks and their attacks; adaptive attacks against PROVISO were not evaluated. The main-experiment agent is Gemma 4 31B IT.
Experimental setupPROVISO · SkillsBench、MalSkillBench et al. · Prec、Recall et al.
Evaluated models
PROVISO
Benchmarks
SkillsBenchMalSkillBenchSkillVetBenchMaliciousSkillBenchANCHORToolSafeAgentDojoAppWorldAgentTrap
Metrics
PrecRecallF1FPRH&Rbenign completionACCbenign utilityutility under attackASRmacro-F1
AI summaryPROVISO is a controller that decides whether to submit before an LLM agent executes an action, uses provenance-typed memory to distinguish trusted from external observations, and compares proposed actions with alternative actions. On four safety tasks covering the agent lifecycle, no baseline exceeds it on both safety and utility; F1 is 13.84 points higher than the strongest baseline on SkillsBench and 7.07 points higher on ToolSafe, and it reaches 88-99 F1 on three skill detection benchmarks.

PROVISO is a controller that decides whether to submit before an LLM agent executes an action, uses provenance-typed memory to distinguish trusted from external observations, and compares proposed actions with alternative actions. On four safety tasks covering the agent lifecycle, no baseline exceeds it on both safety and utility; F1 is 13.84 points higher than the strongest baseline on SkillsBench and 7.07 points higher on ToolSafe, and it reaches 88-99 F1 on three skill detection benchmarks.

Related events and news

1 events · 1 news mentions

News that mentions this paper

大模型安全周报:防线开始溃于组合,智能体授权与审计系统性缺口

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing defenses do not track the source of content, nor compare against alternative actions, and thus must trade off between safety and utility.

    At the commitment point, there is a lack of a commit decision based on content provenance and relative to alternative actions.

    • Scenario: Agents change external state through tools and third-party skills, and a single action may be irreversible. The authors state that risk spans the lifecycle: malicious skills before installation, harmful user requests, unsafe tool calls in context, and indirect prompt injection in tool outputs.
    • Existing shortcomings: Guardrails and skill scanners judge the input as a whole and do not distinguish user authorization from content that comes only from external sources. Most defenses block in isolation at fixed thresholds, and blocking has a cost; the authors state that baselines therefore trade off safety against utility.
    • Setup: Operations and goals must be authorized by trusted observations, while parameters may be supported by trusted or external observations. External content can supply values but cannot rewrite the goal; the same instruction may be legitimate from the user and potentially harmful from tool output.
    • Method: PROVISO sits between the policy and the executor, writes observations and two candidates into memory with source roles, and after role queries commits the action with the highest calibrated value. Allowing or refusing, and choosing between two continuations, are special cases of this formulation.
    • Threat model:
      • Entry points: A malicious skill can be installed before running; a user can issue a harmful task; a tool call can be unsafe in context; tool output can carry attacker-planted instructions.
      • Commitment point: The policy has proposed P0 and the executor has not yet run. The controller also sees an alternative P1: refusal at admission, or another continuation at candidate selection.
      • Visible information: Trusted observations K (request, policy, tool interfaces) and external observations G (tool returns, documents, web pages, skill packages). Candidate selection includes rollouts in a recoverable environment; native scores are used only for training and evaluation.
      • Knowledge level: The paper does not state white-box, black-box or gray-box. When a skill has no user instruction, its declared purpose serves as K.
  2. What related research exists?

    The authors position this paper between lifecycle benchmarks, holistic-judgment guardrails, and rule-based provenance defenses.

    Section 2 of the paper discusses related work in three categories; experimental comparisons are listed separately.

    Agent Safety Benchmarks

    • Skills and misuse: Skill benchmarks collect malicious and benign packages before installation (Yang et al., 2026; Guo et al., 2026; Hossain et al., 2026; Wang et al., 2026c). Misuse benchmarks test whether harmful requests are executed (Andriushchenko et al., 2025; Zhang et al., 2024; Song and Qi, 2026).
    • Tools and injection: ToolEmu (Ruan et al., 2024) and ToolSafe (Mou et al., 2026) evaluate single tool calls in context. InjecAgent (Zhan et al., 2024), AgentDojo (Debenedetti et al., 2024) and ASB (Zhang et al., 2025) evaluate indirect prompt injection.
    • Relation to this paper: The authors state that each benchmark comes with its own defense family, and this paper uses a single formulation to cover four of these stages.

    Guardrails and Skill Scanning

    • Request-level models: Llama Guard (Inan et al., 2023), WildGuard (Han et al., 2024) and Qwen3Guard (Zhao et al., 2025) classify requests or replies as safe or unsafe.
    • Action-level guardrails: GuardAgent (Xiang et al., 2025) and AGrail (Luo et al., 2025) use LLM reasoning against safety requirements; AgentSpec (Wang et al., 2026a) enforces user rules; MCIP (Jing et al., 2025) checks the contextual integrity of tool interactions; Safiron (Huang et al., 2026), ShieldAgent (Zhang et al., 2024) and TS-Guard (Mou et al., 2026) judge actions or step-by-step calls.
    • Scanners and critique: SkillGate, Cisco, Sentry, SkillSpector and SkillFortify screen packages before installation using static rules, LLM semantic analysis, or both. The authors state that the methods in this section treat the input as a whole and compare against fixed thresholds or rule sets; PROVISO instead separates trusted from external observations and compares with alternative actions.

    Prompt Injection Defenses and Provenance

    • Isolation and detection: StruQ (Chen et al., 2025a), SecAlign (Chen et al., 2025b) and BIPIA (Yi et al., 2025) separate instructions from data; PIGuard (Li et al., 2025b) and MELON (Zhu et al., 2025) flag injections in tool returns. The introduction states that Chen et al. (2025a,b) and Debenedetti et al. (2025) modify the agent itself and cover only injected instructions.
    • Execution constraints: System-level methods use dependency planning, dynamic isolation, or permissions aligned with the task, including Debenedetti et al. (2024), IPIGuard (An et al., 2025), DRIFT (Li et al., 2025a), Debenedetti et al. (2025), Progent (Shi et al., 2025) and Task Shield (Jia et al., 2025).
    • Explicit provenance: Wang et al. (2026b), AGATE (Zhang et al., 2026), APPA (Kravchenko et al., 2026), She et al. (2026) and Santos-Grueiro (2026) check authorization using rules, policies or graphs. The authors state that these methods rely mainly on explicit rules; PROVISO instead learns the relation between provenance and action value from outcomes.

    Baselines and Benchmarks

    • Baseline methods: For skill tasks, SkillGate, plus the static and semantic modes of Sentry, Cisco and SkillSpector, and the static SkillFortify. Misuse and unsafe calls share MCIP, WildGuard, AgentSpec and Safiron; misuse adds AGrail and GuardAgent; calls add TS-Guard, Qwen3Guard and ShieldAgent. Injection tasks use Tool Filter, DRIFT, IPIGuard, BIPIA, PIGuard and SecAlign.
    • Benchmarks: SkillsBench, ANCHOR, ToolSafe, AgentDojo, plus MalSkillBench, SkillVetBench and MaliciousSkillBench.

    The authors position this paper as provenance-informed action selection at the commitment point: a single controller covers multiple lifecycle stages and learns value from outcomes, rather than only doing whole-input scoring or rule-based authorization.

  3. How does the paper address this problem?

    PROVISO uses memory with source types and role queries to commit, at the commitment point, the action with higher calibrated value.

    PROVISO estimates the value of candidate actions at the commitment point and submits the higher one after calibration. Admission and candidate selection are two cases of the same decision.

    Problem setting

    • Instance: a decision consists of trusted observations K, external observations G, a proposed action P0, and an alternative P1. K comes from the user and the deployment, including the goal, allowed scope and operations, authorized resources or payees, and termination conditions. G is the returns, documents, web pages, and skill packages read during action; after reading G, the agent's own reasoning also inherits the external source.
    • Permission: operations and goals must be permitted by K, and parameters may be supported by K and G. The authors state that G cannot change the goal; indirect prompt injection is content in G that attempts to act as K. This rule is not given to the controller as a program; the controller learns from outcomes, and memory preserves the distinction between K and G. When a skill package has no user instruction, its declared purpose acts as K.
    • Candidates: P0 is the default action or skill installation when there is no intervention. P1 is rejection in admission, and another continuation in candidate selection. When the environment is recoverable, both continuations are first rolled out in a copy, and only the selected one is executed in the real environment after commitment; observations during rollout belong to the candidate that produced them.
    • Value: the deployment's own scorer gives a native score from 0 to 1. The controller estimates V∗(x,a)=E[y(x,a)|x,a], i.e., the expectation of the outcome of that action given only pre-commitment information, and submits the one with the highest value. In the payment example, the amount may come from an invoice, but if the payee comes only from external content and is not authorized by K, the proposal should not be submitted.

    Memory with source types

    • Role embeddings: the adapter splits the instance into text segments; after encoding by the frozen encoder Enc, a learned embedding is added according to the role K, G, P0, or P1. The same sentence coming from a user request or a tool return yields different memory rows. A component may have no row; at admission, the two fixed actions can be represented by the query alone.
    • Long text: when G is very long, the evidence selector keeps high-scoring segments by similarity to the commitment query and adds the mean of all segments. The selector parameters are trained with native outcomes; an adapter without a selector has no trainable parameters. On SkillsBench, package text is encoded by window, keeping at most 8 windows plus the global mean.
    • AgentDojo: the commitment point is at the start of the task; there are no external observations before commitment, and G contains only a fixed stage description; tool returns are written into the candidate that produced them rather than as a separate G row. P0 is the full trajectory of the deployed agent, whose prompt marks external content with boundary markers per BIPIA and restates the task; P1 is the full recovery trajectory produced by DRIFT. Both trajectories are rolled out by Gemma 4 31B IT.

    Role-query deliberation

    • Five queries: four role queries plus a commitment state with no input role pass through L deliberation blocks. The experiments use L=2, width 384, and 8 heads per layer. Each block performs masked cross-attention, self-attention among queries, and a feedforward update in sequence; memory remains unchanged during deliberation.
    • Re-retrieval: retrieval in a later block is conditioned on the comparison formed in an earlier block. The authors illustrate with the payment example: the first block can find that the proposed payee differs from K, and the second block then goes back to G to find where that payee came from.
    • Scale: the feedforward layer has four SwiGLU experts with top-two routing. Trainable parameters are 17.40M. The architecture is shared across benchmarks, and weights are fitted per benchmark. The encoder is not updated.

    Valuation, calibration, and training

    • Two readouts: in candidate selection, a small feedforward network estimates the value of each candidate from authority, evidence, that candidate, and the commitment state. In admission, it predicts only the probability that P0 is unsafe. The authors give V∗(x,P1)−V∗(x,P0)=2p(x)−1, so the estimated value of P0 is set to 0 and the probability is read only from the commitment state.
    • Loss: candidate selection is the binary cross-entropy of each candidate's value with its native outcome, plus a pairwise ranking term. Admission is the cross-entropy of that probability with the label; when finer labels are available, a risk-category term is added. Auxiliary terms include role supervision on query states, the selector loss, and load balancing.
    • Calibration is separate: after fitting, a per-action offset is selected on the development set to maximize F1, unsafe recall, or the benign pass rate. The authors state that fitting learns which action is better, and calibration determines how much evidence is needed before deviating from the default. The offset for P0 is fixed at 0; P0 is blocked when the estimated unsafe probability is above the P1 offset.
  4. What experiments did the paper conduct?

    The authors state that on the four tasks no baseline exceeds PROVISO on both the safety and utility axes simultaneously, with exceptions on individual metrics.

    Experimental Setup

    • Tasks and scale: SkillsBench has 150 malicious packages and 1,500 benign packages, with five folds by source. ANCHOR tests 300 harmful tasks and 80 benign tasks, split 936/176/380. ToolSafe tests 340 unsafe calls and 367 benign calls, split 5,769/706/707. AgentDojo has 2,516 attacked turns and 97 clean turns, with five folds by user task (Table 7).
    • Models: The controller is a 17.40M-parameter network and does not itself call a generative model. The agents for ANCHOR and AgentDojo are Gemma 4 31B IT. The encoder is a frozen ModernBERT-large; SkillVetBench and MaliciousSkillBench use E5-large-v2. ToolSafe's four role summaries are pre-written by gpt-oss-20b.
    • Baselines: Each task uses the defense designed for it, retaining the original prompts, parsing and action mapping, and is scored on the same test population. Trained baselines use released checkpoints and are not refit on the authors' splits (Appendix B).
    • Metrics: Skill packages report Prec, Recall, F1, FPR. ANCHOR reports H&R (lower is safer) and benign completion rate. ToolSafe reports ACC, F1 and Recall for unsafe calls, with severity greater than 0 treated as unsafe. AgentDojo reports benign utility, attacked utility and targeted ASR.
    • Thresholds: ToolSafe blocks when the unsafe probability reaches 0.005. ANCHOR refuses when the predicted risk exceeds the development-set threshold of 0.01. SkillVetBench uses a fixed threshold of 0.7; the rest mostly select points on the development set by F1, recall, benign pass-through or macro-F1 (Table 9). ANCHOR uses the Harm & Risk judge released with that benchmark; the main text does not give the judge model name.
    • Tests: On SkillsBench and ToolSafe, the authors ran a test-group-paired test on the F1 gain over the relatively strongest baseline (Section 5.1, Appendix E).

    Main Results

    | Task | Metric | PROVISO | Comparison baseline | Source | | SkillsBench | F1 | 85.81 | SkillGate 71.97 | Table 1 | | SkillsBench | Recall | 82.67 | Sentry semantic 100.00 | Table 1 | | ANCHOR | H&R | 0.00 | AGrail 2.65 | Table 2 | | ANCHOR | Benign completion | 100.00 | WildGuard 82.50 | Table 2 | | ToolSafe | F1 | 99.12 | TS-Guard 92.05 | Table 3 | | AgentDojo | Attacked utility | 85.41 | SecAlign 83.35 | Table 4 | | AgentDojo | Targeted ASR | 0.28 | BIPIA 0.20 | Table 4 | | AgentDojo | Benign utility | 87.63 | SecAlign 88.66 | Table 4 |

    • Two axes: The comparison column is the baseline at the extreme of that metric in the table, not the same strongest method. The authors state that in Figure 3 no baseline is simultaneously better than PROVISO on both the safety and utility axes; the abstract says no baseline, and the conclusion says no published defense. The figure caption says upper right is better; the vertical-axis text of panel (b) is H&R (lower is safer), which is inconsistent with "upper right is better." In panel (d), PIGuard is marked outside the plotting range.
    • Exceptions: The skill semantic scanner has higher recall. The authors state that PROVISO's threshold maximizes F1 on the development set, while benign packages are the majority. On AgentDojo, BIPIA's targeted ASR is lower and SecAlign's benign utility is higher; the authors state the latter's ASR is more than ten times higher. On clean tasks, PROVISO only chooses between two obtained trajectories, and it cannot complete tasks that neither can complete. The paper states AGrail completes 35 of the 80 benign tasks.
    • Gains: The authors report F1 gains over the relatively strongest baseline on SkillsBench and ToolSafe of 13.84 and 7.07 points, with paired tests p < 0.01. In Table 14, the Holm p for the two are .002 and <.001 respectively.

    Additional Skill Benchmarks

    • Setup: The same controller is used for MalSkillBench, SkillVetBench and MaliciousSkillBench, which differ in scale, malicious proportion and collection method (Section 5.2).
    • Numbers: In Table 5, F1 is 98.62, 93.33, 88.15; Prec is 99.26, 95.79, 92.99; FPR is 0.73, 4.00, 9.72. MaliciousSkillBench's macro-F1 is 86.02, and it uses the released split, not folds by source.
    • Authors' interpretation: The authors state that the SkillsBench results do not depend on a single data source, and that F1 on these three benchmarks is 88–99 with precision at least 90%.

    Component Analysis

    • Evidence selection: On SkillsBench, removing selector supervision gives F1 69.08, and further removing local memory gives 59.38; the full model is 85.81 (Table 15). Figure 4 labels the former as 69.1 (−16.7). With the model, threshold and memory budget fixed, permuting windows equally makes F1 64.35; restoring the four highest-ranked original windows returns it to 85.91, and reversing the order changes no decision (Appendix F.2).
    • Query computation: A parameter-matched MLP reading the same selected memory gives F1 48.62, 37.19 points below the full model (Table 16). The authors state the gain comes mainly from what enters memory and from the role query computation as a whole.
    • Second retrieval: On the payment task based on the AgentDojo banking tools, mean utility over five seeds drops from 88.48% to 78.54%, and unsafe transfers are below 0.4% on both sides. The drop is 13.15 points larger in conflicting cases than in consistent cases; on two-hop evidence chains, utility is 99.57% and 81.41% (Section 5.3). Figure 4 labels the suppressed utility as 78.5 (−9.9).

    Robustness

    • Repeated training: On ToolSafe, the mean F1 over 20 independent fits is 98.67, 95% interval [97.89, 99.33], range 98.12 to 99.12 (Appendix G).
    • Summary backbone: In Table 20, ToolSafe F1 is Granite-3.3-8B 98.98, gpt-oss-20b 99.12, Phi-4 98.98, Mistral-7B 98.53, Muse-Glimmer-30B 99.27. The authors state the selected model maintains 99.12 across a fairly wide range of logit thresholds; the main text of Figure 6 does not give readings at each offset point.
    • Another agent: This is not the Gemma setting of Table 2. On all 300 harmful and 80 benign ANCHOR instances, Granite-3.3-8B without the controller has mean risk 30.777, and with the controller 0.717, with benign completion 48/80 in both cases (Table 24).

    Other Ablations and Analyses

    • No role embeddings: SkillsBench F1 85.32; dense expert updates: 86.49, gain interval includes 0; full-window memory: 81.19 (Table 16).
    • Learned cross-attention replaced with uniform weights: F1 64.95 on the same 707 ToolSafe calls, a paired difference of +34.18 relative to learned weights (Appendix F.3).
    • Fixed candidates: among 2,516 attacked turns, always choosing the obtained P0 gives JSU 2,066, always choosing P1 gives 2,034, and PROVISO gives 2,147 (Table 19).
    • In-fold selection restores the margin: attack success remains 7, clean completion remains 85, and attacked turns that both complete and fail the attack change from 2,147 to 2,141 (Appendix F.4).
    • Confusion counts: on ToolSafe, PROVISO is 339/1/5/362 (TP/FN/FP/TN, Table 12); on SkillsBench, 124/26/15/1,485 (Table 13).
    • Source-group confirmation for the payment task totals 4,096 cases: mean coverage is 88.85% with full retrieval and 78.93% with the second retrieval suppressed; unsafe rates are 0.37% and 0.39% (Table 18). This setting is not the same number as the 88.48% to 78.54% in Section 5.3.
  5. What are some points that can be further explored?

    The authors note that it must be fit separately per benchmark, depends on recoverable rollouts, and does not evaluate adaptive attacks.

    Limitations and future directions noted by the authors

    • Per-benchmark fitting: each benchmark is fitted separately, so labeled results on the target setting are required; the authors write that no study has transferred a trained controller to other tasks (Limitations).
    • Two-sided rollout: in candidate selection, both continuations are first rolled out in a recoverable environment; deployment must support such rollouts and bear the cost of both sides (Limitations).
    • Adapters: roles are specified by benchmark-specific adapters. On AgentDojo, external content enters inside the candidate trajectory rather than as a separate G row; some ToolSafe inputs are role summaries generated by the proposing model, incurring generation cost (Limitations).
    • Agents: the agents on ANCHOR and AgentDojo are a single LLM (Limitations).
    • Attacks: evaluation uses published benchmarks and their attacks; attacks rewritten against PROVISO itself were not evaluated (Limitations).

    Scope of experiments

    • Tasks: the main results are malicious skill detection, agent misuse, unsafe tool calls, and indirect prompt injection; MalSkillBench, SkillVetBench, and MaliciousSkillBench are also reported (Section 4–5, Table 5). The appendix also reports native execution results on AppWorld and AgentTrap.
    • Models: the agents on ANCHOR and AgentDojo are Gemma 4 31B IT; ToolSafe role summaries are written by gpt-oss-20b; the encoder is ModernBERT-large or E5-large-v2. The appendix also reports controller results with Granite-3.3-8B, Phi-4, Mistral-7B, and Muse-Glimmer-30B as evidence or agent backbones.
    • Comparisons: the baselines for each task are the published defenses listed in Section 4, scored on the same test population; trained baselines use released checkpoints (Appendix B).
    • Splits and tests: the main splits are in Table 7, including grouped five-fold and the released splits. SkillsBench and ToolSafe report permutation tests paired by test group with Holm correction (Appendix E).
    • Not reported in the paper: the main experiments only state that the released Harm & Risk judge was used; the main text does not give the model name of that judge; the appendix's Granite comparison states that Gemma 4 31B IT served as the judge. The paper does not report the agreement between that judge and human judgments.
  6. Summarize the main content of the paper

    At the commitment point, the controller compares P0 and P1 by source and commits the one with higher calibrated value; the authors state it need not trade safety for utility.

    PROVISO is a controller inserted between the LLM agent's policy and executor, making a choice before an action is actually executed.

    • Problem: Whether an action should be executed depends on whether its operation and goal come from the user or the deployment's authorization, while parameters may come from tool returns. Judging only the whole text, or not comparing with alternative actions, means blocking interrupts benign tasks.
    • Approach: Write trusted observations, external observations, the proposed action and the alternative action into memory with source roles, retrieve and compare them with five queries, then commit the value learned from native outcomes and calibrated on the development set. At admission, the alternative is refusal. On AgentDojo, the commitment point is at the task start, and the controller chooses between an already-rolled-out agent trajectory and a DRIFT recovery trajectory, rather than generating a new action on the spot.
    • Skills and calls: On SkillsBench, F1 is 85.81, which the authors report as 13.84 points above SkillGate, while recall is lower than the semantic scanner. On ToolSafe, F1 is 99.12, which the authors report as 7.07 points above TS-Guard; the test confusion is 339 hits, 1 miss and 5 false blocks.
    • Misuse and injection: With Gemma 4 31B IT as the agent, on ANCHOR H&R is 0.00 and benign completion is 100.00. On AgentDojo, attacked utility is 85.41 and targeted ASR is 0.28%; BIPIA's ASR is 0.20 and SecAlign's benign utility is 88.66.
    • Authors' conclusion: The abstract says no baseline, and the conclusion says no published defense, exceeds PROVISO on both safety and utility simultaneously. The authors also state F1 of 88–99 on three other skill benchmarks, and link the gains mainly to evidence selection, role queries and second retrieval.

Related papers

Show 3 more papers
Read originalarxiv.org