Probing agent over-proactiveness in real-world settings with Continuation Audits
Probing agent overeagerness in realistic settings with continuation audits
AI summary
The researchers propose Continuation Audits, an audit pipeline for probing overeagerness in coding agents in realistic settings. The method builds on prefixes of real coding agent session records and guides an audit model to continue the trajectory toward a preset behavior, so it only needs to maintain realism rather than manually construct synthetic scenarios. Validation on SWE-Chat prefixes shows that in 90 paired audits the judge could not reliably distinguish continuations from original sessions (realism win rate 0.42 vs. 0.43), and across all 900 audits the verbally expressed evaluation awareness did not exceed the level observed in unaudited real sessions.