CyberArena: A benchmark for real-time attack-defense confrontations among AI agents
Start here
The researchers propose CyberArena, a dynamic benchmark where AI agents compete in real-time symmetric attack-defense confrontations: two agents in isolated sandboxes act concurrently as attacker and defender under the same rules on continuously changing services, turning static targets into evolving semi-open challenges, with an automated pipeline that continuously generates, reviews, and validates new environments to sustain benchmark longevity. The benchmark uses a dynamic model to map matches into multidimensional scores for offense, defense, and domain capability, evaluating 12 frontier LLM agents across 40 challenges and 466 matches.
Generated by AI from reports · Updated 29 days ago
Attention and developments
Attention, hourly
Now 37, 0% in 24 hours·Peak 37 (Nov 6 03:00)
Related research
This event's paper
- CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
arXiv · 2026-10-13 · Deep reading
Progress timeline
- XiaohongshuXiaohongshu · Liu Dongrui (Shanghai AI Lab)· DiscussionCyberArena: A benchmark for evaluating agents in real-time attack-defense confrontation(opens the original post in a new tab)
- arXivCyberArena: Evaluating AI Agents in Real-Time Symmetric Attack-Defense
The researchers propose CyberArena, a dynamic benchmark that pits AI agents against each other in real-time symmetric attack-defense confrontations, evaluating agents' ability to both attack and defend simultaneously. Two agents act concurrently as attacker and defender in isolated sandboxes under identical rules on continuously changing services, turning static targets into evolving semi-open challenges; an accompanying automated pipeline continuously generates, reviews, and validates new environments to sustain benchmark longevity. The benchmark uses a dynamic model to map matches into multidimensional scores for offensive, defensive, and domain capabilities, evaluating 12 frontier LLM agents across 40 challenges and 466 matches. The authors state that match trajectories resemble fights more than exams, with defenders patching one vulnerability but missing the next, certifying fixes based on stale evidence, and more capable agents planning ahead, implanting backdoors for later rounds, and in one case writing their own packet sniffer to monitor the opponent.
Related discussion
- Older than 72 hours · Not included in the 72-hour figures
- XiaohongshuXiaohongshu · Liu Dongrui (Shanghai AI Lab)CyberArena: A benchmark for evaluating agents in real-time attack-defense confrontation(opens the original post in a new tab)