Cybersecurity's AlphaGo moment, or starting from CyberArena's 466 agent matches
网络安全的AlphaGo时刻，或从这466场Agent对局开始
The research team built CyberArena, a symmetric attack-defense evaluation, having 12 frontier model and harness combinations simultaneously attack opponents and maintain their own services across 22 batches and 466 valid matches. GPT-5.6 Sol ranked first with 78.6 points, Claude Opus 5 second with 73.8, and GLM-5.3 third with 68.5. Each match had 3 rounds of 20 minutes each; patches deployed by the defending side persist, and shutting down services or harming normal business incurs point deductions. The paper says that for the 30 problems all three models could solve alone, after entering two-way adversarial play the round-3 attack capture rates fell to 52%, 37% and 35% respectively, indicating a gap between static CTF scores and dynamic attack-defense performance. An audit of the 466 matches found 2,397 errors, and in 23% of matches the agent claimed the fix was complete based only on logs from before the patch was deployed.
466 two-way attack-defense matches show that models scoring high on static CTFs see a marked drop in capture rate in dynamic confrontation, which can be used to understand the gap between existing safety evaluations and real-world practice.