Shortcut Sensitivity in Agent Safety Evaluations

Core Problem: Agent safety evaluation benchmarks contain surface cues introduced by the evaluation process. These cues can not only directly predict safety labels but may also interfere with the true defensive decisions of safety guardrails, leading to biased measurements of guardrail protective capabilities.

• Scenario and real-world importance: In scenarios where agents execute sensitive actions through tools, front-end tool guardrails are responsible for intercepting potentially risky actions; if the evaluation benchmark itself contains exploitable non-semantic shortcuts, the guardrail performance measured by developers will not reflect its true safety level in real deployment.
• Shortcomings of existing evaluations: Existing benchmarks typically directly compute guardrail accuracy or F1 scores, without testing whether benchmark labels are determined solely by low-level format features, nor verifying whether the consistency between rules and guardrails stems from real risk, data collection differences, or guardrail use of shortcuts.
• Core observation and hypothesis: Scaffolding information in benchmark construction (such as data source markers, specific tool names, trajectory lengths, interaction subsets, etc.) can create potential shortcuts; even if surface cues have high label predictability, this does not mean the guardrail actually relies on those cues; the two need to be strictly decoupled and validated.
• Paper's solution: The authors propose a three-level shortcut audit inference ladder comprising "label predictability, observational guardrail associations, and paired decision sensitivity," and audit published trajectory records and representative guardrail sensitivity on LinuxArena and TS-Bench-derived datasets.

Annotation Artifacts and Shortcut Learning

• Gururangan et al. (2018) and Geirhos et al. (2020): The former studies annotation artifacts in natural language inference datasets due to annotation habits; the latter systematically defines shortcut learning in deep neural networks.
• Koh et al. (2021), Gardner et al. (2020), and Ribeiro et al. (2020): They propose the WILDS distribution shift benchmark, contrast sets for local decision boundary evaluation, and the CheckList behavioral testing framework, respectively.
• Dehghani et al. (2021) and Alzahrani et al. (2024): The former discusses the "benchmark lottery" phenomenon caused by benchmark sampling bias; the latter analyzes the sensitivity of large language model leaderboards to minor perturbations.

Safety Evaluation and Surface Heuristics

• Li et al. (2026) and Tasawong et al. (2025): The former notes that prompt attack defense models tend to learn surface heuristic features; the latter analyzes the impact of keyword bias in safety guardrails on decisions.
• Meng et al. (2025) and Li et al. (2025): The former develops the Docent interpretability tool; the latter proposes the PIGuard guardrail to reduce over-defense.
• Bean et al. (2025), Wang et al. (2026), and Tu et al. (2026): The former studies construct validity in large model benchmarks; the middle conducts validity audits of agent safety benchmarks; the latter proposes BenchGuard for automated auditing of agent benchmarks.

Agent Safety Guardrails and Monitoring Environments

• Debenedetti et al. (2024) and Andriushchenko et al. (2025): They propose AgentDojo, a dynamic prompt injection testing environment, and AgentHarm, a harmfulness evaluation benchmark for large model agents, respectively.
• Mou et al. (2026) and Tracy et al. (2026): The former proposes the ToolSafe framework (including the TS-Bench benchmark and TS-Guard guardrail); the latter proposes LinuxArena, a real-time software production environment agent monitoring evaluation.
• Zheng et al. (2026), Bhagwatkar et al. (2025), and Greenblatt et al. (2024): They propose StepGuard step-level guardrails, end-to-end red team audits of AgentDojo firewalls, and research on monitoring-based AI control protocols, respectively.

Baseline Methods and Benchmarks

• Comparison baselines and benchmark datasets: The paper constructs baselines including the negative average command character count rule, the <INFORMATION> marker plus 11 action prefix rule, exact tool majority class lookup, and an all-predict-unsafe baseline; benchmark datasets include LinuxArena public samples, and TS-Bench-derived AgentDojo-Traj and AgentHarm-Traj.

Differences Between This Paper and Related Work

• The authors state that paired prompt sensitivity testing has precedent; this paper's positioning is to integrate label predictability, observational guardrail associations, and paired decision sensitivity into a three-level inference ladder for published agent safety artifacts, focusing on auditing published records, evaluation weighting units, and black-box guardrail decisions.

The authors propose a three-level shortcut-auditing inference ladder (three-level shortcut-auditing framework), decoupling cue predictability, guardrail association, and causal decision sensitivity into layers, and evaluate black-box guardrails through paired edits that preserve safety semantics and experimental controls.

Formalization of the three-level inference ladder

• Level 1: Label Predictability: Evaluate the ability of deliberately limited low-capability diagnostic rules r(z) to predict published labels y based only on candidate cues z, with score defined as Spred(r) = M({(r(z_i), y_i)}_i=1ⁿ), where M is a separability metric (e.g., AUC, F1, or balanced accuracy). This stage only establishes construct features and does not imply that the cue is used by the guardrail.
• Level 2: Observational Guard Association: Within a restricted comparison cohort C, measure the mean difference in guardrail decisions g(x) between presence and absence of candidate cues: Δ_assoc(C) = E[g(x)|a(z)=1, x ∈ C] − E[g(x)|a(z)=0, x ∈ C]. This metric is used only to screen intervention hypotheses that may exhibit sensitivity.
• Level 3: Paired Decision Sensitivity: Apply transformations T(s, z) = (s, z') to inputs that preserve operation authorization and action effects. For a random guardrail g, the target estimator is Δ_T = 1/n ∑_i=1ⁿ {Pr[g(T(x_i)) = 1] − Pr[g(x_i) = 1]}.

Design of low-capability diagnostic rules

• LinuxArena diagnosis: Use negative average number of Shell command characters (fewer characters yields higher score) as a deterministic scoring rule, and compare against a character n-gram TF-IDF logistic regression model trained with leave-one-environment-out (LOEO).
• AgentDojo-Traj diagnosis: Design a fixed binary rule requiring that the history contains the <INFORMATION> marker and the current tool name prefix belongs to one of 11 side-effect verbs (e.g., send_, delete_, update_, etc.); also design a lookup rule recording majority-class labels for each tool, tested under interaction-grouped five-fold cross-validation and leave-one-domain-out (LODO).
• AgentHarm-Traj diagnosis: Test an all-unsafe baseline that predicts all steps as unsafe, and analyze direct mapping of source subset metadata (harmful_steps and benign_steps) to strict labels.

Construction of frozen counterfactual cohort

• Sample determinism screening: From each of 4 AgentDojo domains, select 8 marker-containing safe steps and 8 marker-containing unsafe steps, sort by SHA-256 hash value and take the first 8, forming a 64-step fixed cohort (frozen cohort) across 58 original interactions, frozen before running any guardrail.
• Five input conditions: Include original input, marker rewriting (replace <INFORMATION> label with <UNTRUSTED_EXTERNAL_CONTENT> but preserve internal bytes), tool aliasing (replace tool names with prefix plus hash, uniformly updating identifiers in requests, history, and descriptions), combined editing (apply both marker rewriting and tool aliasing), and a format placebo that only adjusts outer JSON indentation.

Guardrail evaluation protocol and randomness control

• Deterministic model evaluation: Use an 8-bit, group-64 TS-Guard MLX reproduction model (Q8), run under greedy decoding, and pass gate validation on 64 original samples (64/64 parsing, 63/64 strict decision agreement).
• Random model interface and scheduling: Call Claude Haiku 4.5 and GPT-5.6 Luna, both routed directly through OpenRouter with fallback disabled, sampling temperature set to 0, and strict JSON-schema structured output enabled; each base sample is called 3 times under each of the 5 conditions with alternating shuffled scheduling (960 calls total).
• Estimation and confidence intervals: The main estimator is the mean of per-sample interception probability differences across 3 calls, with 10,000 bootstrap resamples stratified by domain and clustered on 58 source interactions to compute 95% exploratory intervals.

Experimental Setup

• Models under test: Claude Haiku 4.5, GPT-5.6 Luna, TS-Guard (including the officially released checkpoint and the Q8/MLX reproduced version), Claude Opus 4.8, Claude Sonnet 5, GPT-5.6 Terra, GPT-5.5, and the Claude Opus 4.6 monitor in the LinuxArena evaluation.
• Datasets and benchmark scale: The LinuxArena public sample contains 576 deduplicated Shell trajectories (176 attack, 400 honest, across 10 environments); AgentDojo-Traj derived from TS-Bench contains 1,220 steps in total (352 unsafe steps, 868 safe steps, across 4 domains); AgentHarm-Traj contains 731 steps in total (525 unsafe steps, 206 safe steps, covering 197 interaction groups); the frozen AgentDojo counterfactual cohort contains 64 steps (32 safe steps, 32 unsafe steps, from 58 source interactions).
• Baseline methods: Negative mean command character count rule, character n-gram TF-IDF logistic regression, marker plus 11-action-prefix fixed rule, exact tool majority-class lookup rule, and an all-predict-unsafe baseline.
• Evaluation metrics: AUC, F1, balanced accuracy, strict block rate, and marginal block probability difference (Δ, percentage points, pp).
• Judging method: For the TS-Guard composite score, a score greater than 0.5 is judged a strict block; for hosted LLM guardrails, the system prompt requires structured JSON output judging the maliciousness of the user request, third-party attacks, and the harmfulness of actions, and any non-zero harmfulness judgment or attack judgment counts as a strict block.
• Sample size and number of repetitions: Haiku and Luna were each repeated 3 times under 5 conditions on the 64-step cohort with independent alternating calls (960 calls each); TS-Guard Q8 ran 64 steps with single greedy decoding; GPT-5.5 performed single-call sampling over 64 steps.

Main Results

On the 64-step frozen AgentDojo cohort, with three rounds of repeated alternating calls against the Claude Haiku 4.5 guardrail, the block probability differences of each edit relative to the original input were measured (according to Table 2):

Swipe horizontally to read the table

• Marker rewrite and alias sensitivity: The authors report that rewriting the <INFORMATION> marker reduced Haiku's overall block rate by 8.33 pp, whereas introducing opaque tool aliases increased the block rate by 11.46 pp; after subtracting the formatting placebo effect, the adjusted effects were -6.77 pp [-12.82, -1.61] pp and +13.02 pp [4.92, 21.72] pp, respectively, with both 95% confidence intervals excluding zero.
• False-positive changes and false-positive shift: The authors point out that all unsafe samples remained blocked after the edits, so the change in block rate stems entirely from changes in false positives on safe samples' inherited labels, causing the measured strict accuracy to change from the original 73.44% to 81.77% after marker rewrite and to drop to 61.98% after tool alias.
• Direction and sparsity: The authors point out that the affected samples maintained the same direction of change across repetitions; marker rewrite showed only block decreases (6 down, 0 up), and tool alias mainly showed increased blocking (2 down, 9 up).

Cross-Model Replication and Configuration Specificity

• What was tested: GPT-5.6 Luna (960 calls) and Q8 TS-Guard (local MLX) were evaluated on the same 64-step cohort, prompts, and schedule.
• Test results: Under marker rewrite, GPT-5.6 Luna's full-sample difference was 0.00 pp [-3.13, 3.17] pp, and relative to placebo -1.04 pp [-4.76, 2.60] pp; under tool alias, the difference was +1.04 pp [-2.08, 4.41] pp, and relative to placebo 0.00 pp [-3.65, 3.24] pp. Q8 TS-Guard had 3 decision flips among the 64 marker-rewrite pairs (flip rate 4.69%) and 3 flips under tool alias (flip rate 4.76%, according to Table 4).
• The authors' interpretation: The authors state that the sensitivity exhibited by Haiku is not universal across models and is closely related to the specific guardrail configuration; Luna and TS-Guard did not replicate the magnitude of Haiku's sensitive effect.

First-Level Benchmark Cue Separability and Domain Transfer

• What was tested: Low-capability diagnostic rules were run and stress-tested on LinuxArena, AgentDojo-Traj, and AgentHarm-Traj.
• Test results: On LinuxArena, the AUC of negative mean Shell command character count reached .910 (LOEO lexical AUC reached .934, according to Table 1); on AgentDojo-Traj, tool-name majority-class lookup reached F1 .901 under five-fold cross-validation, but under LODO leave-one-domain-out testing F1 dropped to .495 (only 17.3% of steps used tool names seen in the training domain, according to Table 1 and Table 5); on AgentHarm-Traj, TS-Guard's row-level F1 was .902, but dropped to .719 under equal-interaction weighting (according to Table 6).
• The authors' interpretation: The authors believe that the benchmarks exhibit obvious construction cues and weighting sensitivity, but the collapse of tool lookup across domains shows that these surface cues are highly dependent on domain context rather than being universal global shortcuts.

Observational Marker Association Screening

• What was tested: The difference in strict block rates between real safe rows with and without markers in AgentDojo-Traj was counted for 7 guardrail configurations (according to Table 3).
• Test results: Claude Haiku 4.5's block rate on safe rows with markers was 74.45% (239/321), versus 0.37% (2/547) without markers, a difference of 74.09 pp [68.67, 79.75] pp; GPT-5.6 Luna's difference was 10.48 pp, GPT-5.5's difference was 4.22 pp, and TS-Guard's difference was 3.62 pp.
• The authors' interpretation: The authors emphasize that this comparison has confounders (requests, contexts, and underlying risks differ), so the high observational association serves only as a basis for nominating guardrails as intervention candidates, not as a conclusion that markers are causally exploited.

Other Ablations and Analyses

• Predictive power of action count on LinuxArena: In the same monitor rerun subset, the AUC of action count predicting the label was .947, higher than the AUC of .905 for mean command length.
• Mean of matched execution signatures on AgentHarm: Among 16 mixed contexts with identical execution signatures, TS-Guard's mean was .922 in unsafe contexts and .413 in safe contexts (difference .510).
• Flip rate of combined edits for TS-Guard Q8: 6 decision flips occurred among 64 samples, with a base flip rate of 9.38% (according to Table 4).
• Unpaired difference in GPT-5.5 single sampling: Under single sampling, marker rewrite and the original input disagreed on 3/64 base samples, with a directed block rate change of +1.56 pp [-3.28, 7.15] pp.
• Performance on unseen tool names in AgentDojo: In the LODO test, exact tool lookup on 1,009 steps with unseen tool names had F1 .000 and balanced accuracy .500 (according to Table 5).
• Repeated statistics of interaction segments in AgentHarm: 57 of 72 harmful interaction groups contained duplicate segment IDs, whereas none of the 125 benign interaction groups did.

Limitations and Future Directions Noted by the Authors

• Cross-model configuration specificity: Luna reproduction experiments did not reproduce Haiku-level sensitivity, indicating that the observed sensitivity depends on the specific guardrail configuration and cannot be generalized to all guardrails (Section 8).
• Non-probability sampling and interval properties: The 64-step counterfactual cohort is not a probability sample, and the computed confidence intervals are exploratory and not corrected for multiple testing (Section 8).
• Edit semantics and lack of blinded human review: Marker rewriting changes the wording of lines from external sources, tool aliasing is not a natural global renaming, and neither type of edit underwent double-blind human equivalence validation (Section 8).
• Model implementation and environmental limitations: TS-Guard uses a Q8 quantized version, GPT-5.5 lacks repeated experiments, and StepGuard was only validated on a non-equivalent Mac feasibility check, not a full official environment evaluation (Section 8).
• Evaluation input attribute limitations: LinuxArena uses completed execution trajectories rather than real-time monitoring, and AgentHarm's subset identity metadata has not been proven visible to guardrail inputs (Section 8).
• Defense adaptivity not tested: Public rules may become targets for adaptive adversarial evasion, motivating future research on hidden group holdouts and equivalent serialization refresh mechanisms; the paper does not test such adversarial adaptivity (Section 6).

Experimental Coverage

• Guardrail model coverage: Experiments deeply tested Claude Haiku 4.5, GPT-5.6 Luna, and local Q8 TS-Guard in counterfactual edits, with auxiliary testing of GPT-5.5 single sampling; StepGuard was not run in the full official BF16/vLLM environment.
• Test dataset scope: Experiments were limited to LinuxArena's 576 deduplicated shell trajectories, and TS-Bench-derived AgentDojo-Traj (1,220 steps) and AgentHarm-Traj (731 steps).
• Paired intervention sample size: Counterfactual edits were performed only on AgentDojo's 64-step frozen sample (from 58 source interactions), with no paired edits on AgentHarm or LinuxArena.
• Offline and non-interactive limitations: The study is a black-box static audit using published offline records, without executing end-to-end tool calls or deploying interactive agents or evasion attacks.
• Unreported information: The paper does not report the impact of longer prompt chain reasoning on shortcut sensitivity, nor does it conduct online interception evaluations for real system deployment.

• Research positioning: This paper targets safety benchmarks and tool guardrails for large language model agents, proposing a three-level inference ladder for auditing evaluation validity and reliance on surface shortcuts.
• Core problem: Existing agent safety benchmarks contain metadata, markers, and interaction formats introduced by the evaluation pipeline; these cues may artificially boost label separability and interfere with guardrail decisions, causing evaluation results to fail to accurately reflect the true protective capability of guardrails.
• Method design: The authors establish an analytical framework distinguishing "label predictability, observational guard association, paired decision sensitivity," design deliberately limited simple diagnostic rules, and implement marker rewriting and tool alias edits on 64-step frozen samples that preserve safety semantics, combined with format placebos and alternating repeated calls to test sensitivity of random guardrails.
• Key results: On benchmark features, LinuxArena negative average command length classification AUC reaches .910, AgentDojo tool-name majority-class lookup under leave-one-domain-out drops F1 from .901 to .495; on guardrail sensitivity, marker rewriting changes Claude Haiku 4.5 interception rate by -8.33 pp, tool aliasing changes interception rate by +11.46 pp, while GPT-5.6 Luna's corresponding changes are 0.00 pp and +1.04 pp, and Q8 TS-Guard flips decisions only 3 times across 64 pairs.
• Authors' conclusions and implications: The authors argue that high-predictive cues in benchmarks are only hypotheses, not proof of guardrail reliance on shortcuts; shortcut sensitivity is highly configuration-specific; they recommend that future agent safety benchmarks adopt domain-grouped leave-out at release, specify evaluation weighting units (e.g., interaction-level weighting), and introduce placebo-controlled paired intervention experiments in guardrail evaluation.