Microsoft FARA 7B success rate on WebVoyager drops from 74% to about 38%
Microsoft's FARA 7B: The Web Agent That Scored 74% — Until a Better Judge Looked Closer
Microsoft researcher Corby Rosset and Browserbase's Miguel González Fernández point out that Microsoft's FARA 7B web agent achieves a 74% success rate on the WebVoyager benchmark as scored by the official judge, but drops to about 38% when evaluated with a general-purpose verifier calibrated against human annotations, with a Cohen's Kappa of 0.58 against human labels. The article's analysis notes that existing LLM judges exhibit systematic flaws in long-trajectory evaluation, such as ignoring scoring rubrics and multimodal context overflow, and that evaluation reliability needs to be improved through rubric alignment and process scoring.
同一模型同一基准，官方 GPT-4o 判分给出 74%，经人工校准的通用验证器只有约 38%，差距来自判分器设计而非模型本身。