Fable 5.2 costs 17 times more and still loses! 12 AIs join a group chat as colleagues, OpenAI takes the top two spots
Fable 5.2贵17倍还输了!12 个AI进群当同事,OpenAI包揽前二
Data collaboration platform Deepnote released the Agent Initiative Benchmark, placing 12 frontier models into a fictional company's Slack channels as AI colleagues to test when they should help and when they should stay silent. OpenAI took the top two spots: GPT-6 Astra scored 8.97, and GPT-6.1 Sol scored 8.65 at $0.11 per task; third place Anthropic Fable 5.2 scored 8.17 at $1.87 per task, about 17 times that of Sol, which translates to a monthly bill of $281 for Sol and $4,684 for Fable 5.2 for a 70-person company. The evaluation used the fictional company PageMantle, with 71 employees and 11 business systems, and 50 workplace tasks, 10 of which were scenarios where the model should not act; two blind-review models independently scored 0 to 10, and the dataset and code are open-sourced. Even the best-performing model held back only 8 times when it should have stayed silent, and butting in cost on average 1.8 times more time and money than staying silent; Gemini and Grok called tools 35 times per task on average, and 27 and 18 tasks respectively timed out without submission.