Scale AI releases SWE-Bench Pro V2: a software engineering agent leaderboard that is harder to game(original, opens in a new tab)
Scale AI and Reflection jointly release SWE-Bench Pro V2, refreshing the public leaderboard with a modified benchmark and a network-locked evaluation protocol. V2 retains 642 tasks across 11 repositories, removes 89 invalid tasks found during review, and adds a Hard subset of 51 high-difficulty tasks. The hardened evaluation closes four cheating channels: mirrors exclude fix commits and hidden files, the agent-stage sandbox can only access model endpoints, scoring re-executes on clean mirrors and publishes both scores, and the verifier restores hidden test paths and rolls back changes to fixtures. On the public leaderboard, Claude Opus 5 reaches 99.4%, Kimi K3 97.7%, and GPT-Astra 96.9%, but there is a clear gap between public and private sets; for example, Claude Opus 5 solves 638/642 on the public set and 222/272 on the private set, a 17.8 percentage point difference, which the authors attribute more likely to training-period exposure than evaluation-time leakage.