Organizing Trajectory Evidence for Language Model Agent Assurance: Fragments, Methods, and Residuals

The paper proposes a unified framework that combines the different-strength claims made by methods such as log rule checking, skill coverage analysis, support checking, prefix monitoring, execution gating, and rare event estimation about language model agent trajectories, and reveals obligations that remain uncovered. It requires writing as a two-layer logical formula: the outer layer is a finite-trace temporal logic over recorded events, and the inner layer judges whether the argument structure supported by the recorded context holds when the agent makes a decision; the information available to the evaluator, the agent, and the execution gate defines fragments of this language, each checking method decides one fragment and returns exact, limited-to-test-set, risk-bound, or descriptive conclusions, and the assurance ledger classifies each obligation as established, addressed but not established, or unaddressed. On 456 published τ²-bench telecom trajectories, the benchmark oracle flags 231 runs; after adding a rule checker, a support-checking stand-in, and a prefix-monitoring baseline, the union rises to 391, 401, and 407 runs respectively, each method contributes flags missed by the others, and gating makes one prohibition an exactly decidable conclusion on the gated deployment.