Study reveals shortcut sensitivity in agent safety evaluations
Start here
An arXiv paper released on October 16, 2026 proposes a three-level shortcut audit framework, checking published Agent safety evaluations through three tests—label predictability, observational guardrail association, and paired decision sensitivity—to determine whether they rely on benchmark construction rather than genuine safety signals. The authors conduct black-box audits of LinuxArena, AgentDojo-Traj, and AgentHarm-Traj, finding that in LinuxArena, shorter average shell commands strongly predict trajectory labels, with an AUC of .910; AgentDojo-Traj's exact tool lookup drops in F1 from .901 to .495 under domain holdout.
Generated by AI from reports · Updated 28 days ago
Attention and developments
Attention, hourly
Now 39, 0% in 24 hours·Peak 39 (Nov 6 03:00)
Related research
This event's paper
- Shortcut Sensitivity in Agent-Safety Evaluations
arXiv · 2026-10-15 · Deep reading
Progress timeline
- arXiv: Dangerous Capabilities & Safety EvalsShortcut Sensitivity in Agent Safety Evaluations
The paper proposes a three-level shortcut audit framework, using three checks—label predictability, observational guardrail association, and paired decision sensitivity—to examine whether published agent safety evaluations rely on benchmark construction rather than genuine safety signals. The authors black-box audit LinuxArena, AgentDojo-Traj, and AgentHarm-Traj: in LinuxArena, shorter average shell commands strongly predict trajectory labels, with AUC of .910; AgentDojo-Traj's exact tool lookup drops from F1 .901 to .495 under domain holdout; AgentHarm-Traj's released TS-Guard drops from F1 .902 to .719 under equal interaction weighting, while balanced accuracy rises from .754 to .776. Based on this, the authors recommend domain-grouped holdout, per-evaluation-unit scoring, and placebo-controlled paired interventions.