Write Once, Check Many Times: A Reusable Ledger for Post-Hoc Policy Compliance Auditing of Agent Trajectories
Write Once, Check Many: Reusable Ledgers for Post-Hoc Policy Compliance Auditing of Agent Trajectories
Translation in progress — untranslated text is omitted.
The paper proposes a "write once, check many times" framework, in which a writer model converts each Agent trajectory into a policy-agnostic ledger, and a reader model checks a new policy using only the ledger, without re-reading the original trajectory. The authors test on 900 trajectories from SHADE-Arena, τ-bench, and τ²-bench: when a 12B reader model reads narrative ledgers, its AUC matches or even exceeds that of the same model reading full trajectories; on SHADE-Arena policies that require combining multiple facts, the method achieves an AUC of .777, exceeding the .713 of a 31B model reading full trajectories. When checking 8 SHADE-Arena policies individually, the ledger still uses about 5 times fewer tokens even counting the writing cost. The analysis also finds that rule-based extraction methods perform better on some single-action policies, and that ledgers may miss evidence in longer tool outputs and message content.
- When re-checking stored agent trajectories afterward with new natural-language policies, existing monitors often must re-read long trajectories per policy. The authors ask whether a factual record can be written once without knowing the policy and reused for later policies.
- The Writer does not see the policy and writes the facts into a ledger (structured or narrative) using Gemma-4-12B-it, without giving an overall verdict. The Reader sees only one policy and the ledger, and gives a violation verdict and a suspicion score from 1 to 10. The main set labels are computed by a script from successful tool results.
- On the eight policy sets, 12B reading the narrative ledger is not lower than the same model rereading. SHADE combined AUC .777, 31B reread .713. Eight SHADE policies, Figure 1 marks 4.9× fewer tokens. On CUA, narrative .770, reread .896.
- The authors state that the main policy sets are mainly about recorded actions, and the combined set is small and mostly written by the authors; the cost assumption is that policies arrive separately and there is no caching, and weighting input and output tokens equally would underestimate generated tokens. No comparison was made with extract-then-judge, policy-aware summarization, or policy compilation. Agents that make the writer omit evidence were not tested.
Related events and news
1 events · 0 news mentions
- This event's paper
可复用账本提升Agent轨迹策略审计效率