Parsewave 审计 AutomationBench 评估器发现问题

Parsewave released AutomationBench Verified V0.0, which audits and applies fixes to the grader of Zapier's AutomationBench (commit 4a8e106), confirming that 210 tasks have grader issues, of which 197 come from the 600 scored tasks and 13 from the 200 unscored simple tasks. The audit first used an AI agent to write adversarial test answers for all 600 scored tasks, then manually reviewed the flagged tasks. Among 16,683 test answers that should have failed, the original grader let through 9,797 (59%); among 5,949 answers that should have passed, the original grader misjudged 1,919 (32%). The dataset includes fixed task files, grader/fixed_rubric.patch, 210 evidence directories, and records of 1,236 runs of Kimi K3 on 206 fixed tasks, three runs each of the original and fixed versions.

Parsewave 对 AutomationBench 的 600 个公开任务做了验证器审计，用智能体辅助的对抗性提交筛查出约 300 个可疑任务，人工复核后确认 206 个存在严重问题并全部修复，其中 100 个案例公开了详细推理。该基准测试 AI 智能体在销售、营销、人力资源、运营、客服和财务等业务流程上的表现，出现在 Gemini 4 Argon、Claude Opus 5.5、GPT-6 Astra 等模型卡中。用开源前沿模型 Kimi K3 复测显示，修复后 16 个任务变宽松、57 个任务变严格，平均分均朝修复方向变化；用相反验证器重判时 344/1235 条判定改变，占 27.9%。审计仅覆盖公开任务，作者表示希望与 Zapier 合作审计私有集。