SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
The paper's authors release the SWE-CoWork benchmark, containing 101 development tasks covering 101 real repositories, 11 programming languages and 12 application domains; each task places the agent in a simulated engineering organization with 26 colleagues of differing responsibilities and knowledge, requiring it to recover requirements from colleagues and work records, reconcile conflicting evidence, and deliver evaluated by private executable tests. Evaluation of 11 models shows that two leading models satisfy about 63% of fail-to-pass (F2P) tests, but fully complete fewer than 8% of tasks; providing complete requirements in advance improves F2P for Seed Evolving and GPT-6 Astra by about 16 percentage points, with the full task success rate unchanged.