Judging when to intervene: causal evaluation of AI agents

The researchers propose a framework for estimating the causal effects of randomized interventions in AI agent trajectories, to determine whether agent behavior should be verified or guided at deployment time. The paper notes that static benchmarks cannot reflect the value of audit interventions at deployment time, and the authors claim the framework is the first methodologically grounded method of its kind, using smaller-scale trajectory data collected under randomized interventions such as verification and screening to estimate the dynamic causal effects of a class of contextual intervention policies. The estimated structural parameters can be used to construct safety leaderboards characterizing the risk of various agent behaviors, for practitioners to prioritize audits. Applications on tau-bench Retail, ToolSandbox and ImpossibleBench show that intervention effects vary significantly with agent actions and states. The authors believe this work points to a research agenda at the intersection of causal inference and agent evaluation, and is important for reliable and safe deployment of agents in the real world.