Skip to content
WeChat · Weijiqi Lixin·Indexed by this site · Originally published· AI updates →· Its place in the scan record →

Shanghai AI Lab paper: switch to a different agent execution framework, and model score rankings may reverse

上海AI Lab|大模型Agent翻车,到底怪模型还是怪框架?这篇论文戳破Agent跑分的真相

AI summary

Shanghai AI Lab's paper, through cross-controlled experiments on 52168 agent run trajectories, breaks agent scores into two variables: the model and the execution framework (Harness), finding that many failures cannot be attributed solely to the large model. The experiments cover four frameworks—OpenCode, Codex CLI, Pi CLI, and DeepAgents—and four models—DeepSeek V4 Flash Preview, MiniMax M3, Kimi K2.7, and Qwen3.6-35B-A3B—totaling 111 evaluation settings and 6 major agent benchmarks. The paper proposes five metrics including VCR effective completion rate, distinguishing two types of failure: doing the task wrong versus the program crashing; results show that switching to a different framework may reverse model rankings, some model-framework combinations have clear interaction effects, full token consumption including failures is inflated by abnormal loops, and taking the best of four attempts is significantly higher than the average score. The paper recommends that evaluation reports note the framework version, and select frameworks by cross-model adaptability or paired testing.

Read originalmp.weixin.qq.com