Rethinking Verification in Agent Frameworks
Rethinking Verification in Agent Harnesses
Researchers treat verification as a dimension of test-time scaling, comparing 17 verification designs across seven frontier coding models, and find that independent review by an independent reviewer provides a more favorable trade-off between accuracy and cost. The framework covers four layers of the agent lifecycle, namely initial instructions, mid-turn reminders, completion gating, and independent review, and is evaluated on Terminal-Bench and SWE-bench Verified. On Terminal-Bench and SWE-bench Verified, independent review brings average accuracy improvements of 5.9-9.5 and 3.9-6.6 percentage points at 1.13-1.25× and 1.00-1.04× baseline cost, respectively; initial instructions and execution-time reminders bring only modest, model-dependent improvements with no large increase in average output token usage on average, while completion gating brings larger improvements but may substantially increase cost. Adding review rounds or independent reviewers can further improve Terminal-Bench average accuracy by up to 3.5 percentage points.
Related events and news
1 events · 0 news mentions
- This event's paper
研究:独立评审在智能体验证中权衡更优