Skip to content

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Hugging Face Daily Papers· arXiv:2611.20218· Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan·Indexed by this site · 2 days ago· Originally publishedAI updates· Its place in the scan record →
AI summary

The paper's authors release the SWE-CoWork benchmark, containing 101 development tasks covering 101 real repositories, 11 programming languages and 12 application domains; each task places the agent in a simulated engineering organization with 26 colleagues of differing responsibilities and knowledge, requiring it to recover requirements from colleagues and work records, reconcile conflicting evidence, and deliver evaluated by private executable tests. Evaluation of 11 models shows that two leading models satisfy about 63% of fail-to-pass (F2P) tests, but fully complete fewer than 8% of tasks; providing complete requirements in advance improves F2P for Seed Evolving and GPT-6 Astra by about 16 percentage points, with the full task success rate unchanged.

Read originalhuggingface.co