CyberArena: Evaluating AI Agents in Real-Time Symmetric Attack-Defense

Static, single-role cybersecurity benchmarks cannot measure an agent's ability to attack and defend simultaneously when the opponent adapts in real time.

• Scenario: The authors argue that in real operations, one side exploits a vulnerability while the other patches it, and the patch then forces the attacker to change course. Fixed evaluations can be sustained by leaked solutions or memorized exploits, and they also saturate as models become stronger.
• Existing shortcomings: The authors point out three issues. The target does not adapt; single-role setups separate attack and defense, with defense evaluations replaying pre-recorded exploits rather than a real-time opponent (Figure 1); and the fixed tasks as the authors define them lose discriminative power (Section 5.4).
• Core setup: The opponent's attacks and patches change the conditions faced by each side, and the service and sandbox are a shared environment. The authors call the co-evolution of attack and defense driven by competitive interaction Recursive Adversarial Improvement (RAI), and call CyberArena a first step toward RAI.
• This paper proposes: a symmetric multi-round attack-defense platform, a dynamic latent-capability model that maps sparse matches onto a unified reference, an agentic curation pipeline that continuously generates and reviews new environments, and an evaluation of 12 frontier contestants, 40 challenges, and 466 valid matches.
• Adversarial setup:
  • Objective: Each contestant simultaneously captures rotating read, write, or execute targets on the opponent's service and patches and maintains their own service.
  • Knowledge: Ranking matches give no vulnerability hints. The briefing describes the application, ports, source code paths, and functionality that must be preserved, without disclosing vulnerabilities or fixes.
  • Capabilities: Root SSH access to their own service, with the ability to edit, rebuild, and restart it; access to the opponent's service via the attack network. Initial access is defined by the challenge, and defensive modifications persist across rounds.
  • Environment and adjudication: Identical vulnerable services in isolated sandboxes. The arena uses MCP tools to verify flags and write markers, and health checkers check availability and specified functionality. In ranking, at most one successful capture counts per round.

Single-role vulnerability and CTF benchmarks

• Capture-the-flag and penetration: Shao et al. (2024)'s NYU CTF Bench, Zhang et al. (2025b)'s Cybench, and Yang et al. (2023)'s InterCode let isolated agents capture flags from fixed targets. Penetration testing directions include Deng et al. (2024)'s PentestGPT, Gioacchini et al. (2024)'s AutoPenBench, and Lin et al. (2026).
• Real-world software vulnerabilities: Fang et al. (2024), Zhu et al. (2025, 2026), Wang et al. (2026a, 2026b), Lee et al. (2025), Liu et al. (2026), Zhang et al. (2025a), and Shi et al. (2026) cover one-day or zero-day exploits, Web CVEs, PoCs, exploitation under mitigations, bug bounties, and end-to-end patching, respectively.
• The authors' comparison: In Table 1, these benchmarks evaluate predefined tasks without a real-time opponent. The authors specifically note that BountyBench splits Exploit and Patch into separate tasks targeting known vulnerabilities. This paper follows the CTF convention of using flags as evidence of successful exploitation, but places both roles in the same match.

Capability estimation and ranking

• In-game points are insufficient for cross-match comparison: Vigna et al. (2014) and Zafar et al. (2024) use points to decide CTF winners. The authors note that cross-competition comparison must also account for opponent strength and problem difficulty; raw win rates depend on the opponent set (Balduzzi et al., 2018; Omidshafiei et al., 2019).
• Paired scoring and item response: Bradley–Terry, Elo, Glicko, TrueSkill, and Chatbot Arena's pairwise comparisons (Zheng et al., 2023; Chiang et al., 2024). Item response theory links success to ability and difficulty (Lord and Novick, 1968; Reckase, 2009).
• Sports models separating attack and defense: Maher (1982) and Dixon and Coles (1997) estimate attack and defense strengths separately. The authors state that this paper combines Bradley–Terry's attack-defense difference with challenge-specific item response terms.

Match selection under limited budgets

• Information-gain scheduling: Maia Polo et al. (2024) and Hofmann et al. (2025) discuss evaluating with fewer samples; Heckel et al. (2019) and Chiang et al. (2024) actively select pairwise comparisons. The authors accordingly schedule matches by expected information gain rather than exhaustively pairing.

Experimental comparisons and baselines

• The comparison is not a re-run of the Table 1 benchmarks. The authors compare single-player attacks against three-round adversarial play on the same 40 challenges (Section 5.4), and check the expected-score ranking against a Bradley–Terry model using only wins, draws, and losses on the same 466 valid matches (Appendix D.4).

The authors position CyberArena as extending exploit-based evaluation to simultaneous attack and defense against an adaptive opponent. The Table 1 caption states that, among the existing work compared, it is the first benchmark to let autonomous agents face a real-time, adaptive opponent while attacking and defending simultaneously.

CyberArena has two model–harness competitors host the same vulnerable service in isolated sandboxes, attacking and defending simultaneously over multiple rounds; a latent capability model with round dependence then maps sparse matches to expected scores on a common scale.

Match protocol

• Each match has three rounds of 20 minutes, with no vulnerability hints. Competitors use fixed versions and fixed settings. The paper states each match requires two agents and lasts one hour.
• The attacker starts from the challenge-defined baseline access and crosses the security boundary to complete a read, write, or execute objective. Objectives rotate each round, and even if old paths remain usable, the new objective must be targeted. Defensive modifications persist across rounds.
• Periodic health checks penalize patches that break specified functionality. Ranking points come from captures (at most one success per round), preventing the opponent's capture, and keeping the service healthy (Appendix A).
• One agent per team runs throughout, polling match state and deciding on its own whether to attack or patch. Round timers rotate objectives and do not reset agents or services between rounds.

Capability estimation and ranking

• A match is g=(i,j,c). Y indicates whether i captured at least one of j's objectives in that round; Z is the number of health probes passed, normalized to m=20.
• Before accounting for round dynamics, the log odds of capture are η_i,j,c=α_c+λ_c[a_i−d_j+q_c^⊤(u_i−v_j)]. αc is the challenge difficulty intercept, λc>0 controls sensitivity to capability differences, qc is the curator-specified mechanism profile, ai and dj are attack and defense capabilities, and ui and vj are mechanism expertise.
• The three rounds of the same attack direction share a random effect; there are also round effects and a previous-round capture term. Availability is estimated from health checks via a parallel beta-binomial. Dynamics and hyperparameter priors are calibrated on 53 held-out matches, which do not enter the evaluation likelihood.
• The single-match score equally weights average captures per round, rounds not captured by the opponent, and uptime: S_i(g)=100/3[Ȳ_g,i+(1−Ȳ_g,j)+Z̄_g,i/m]. The ranking is the expectation of this score over all opponents, all reference challenges, and posterior samples. After each batch, a Laplace approximation is refit, and 5–95% credible intervals are given conditional on the model. Expertise sums to zero across mechanisms and competitors; log λc sums to zero across challenges.

Adaptive match selection

• Exhaustive pairing is infeasible. After uniform piloting, scheduling follows approximate expected information gain of the parameters, retaining 25% uniform exploration to guard against model misspecification (Appendix D.3). In ranking activities, each batch after piloting plans 24 matches.

Agent challenge curation

• Sources are public vulnerability advisories and attack-defense archives from saarCTF, FAUST, and DEF CON CTF finals. Advisory items prioritize entries with public exploits in CISA KEV or Metasploit, and OSV is used to locate vulnerable and patched versions. Candidates are excluded for withdrawal, missing evidence, incompatible licenses, or inactive repositories.
• A valid attack must achieve the stated objective through a real application vulnerability; score shortcuts introduced by the evaluation environment are invalid, and vulnerabilities not covered by reference exploits also count. A valid defense must block the tested exploit and preserve specified functionality (Figure 2). The briefing does not disclose the vulnerability or fix and is appended to the system prompt.
• Advisory candidates are first reproduced in isolation: the same proof of concept achieves the objective only on the vulnerable instance, and functional checks pass on both instances; then editing, restart, and clean-environment reruns are done. Packaging does not change the verified security assumptions.
• Before admission, a validator checks build, run, and interface. Three reviewer agents perform static review, cheat review, and exploitability review; the cheat review must execute the reported shortcut under competitor permissions. A final decision agent checks against source code and runtime evidence. Defective items are returned, with up to four rounds of automated review. Humans reviewed the security assumptions, objectives, and design of 27 challenges. Appendix C gives the full three-layer prompts: arena rules, run instructions, and challenge briefing.

Verification facility

• Each challenge is a Python package with a unified interface: fixed image, objective planting and retrieval, functional checks, restart handling, and briefing. Write and execute objectives must publish new identifiers each round, and the framework rejects reuse.
• Read and execute check submitted values; the execute flag must be retrieved by a helper program in the service execution context. Write objectives are read back by the facility from the declared area for the team's own round marker, without executing competitor-submitted commands inside the defended container. Scoring observations are produced by the facility, not self-reported by agents.
• Of the 46 packages, 40 have network checkers inside the health-polling container, outside the defender's root; 4 legacy packages check inside the service container; and 2 framework controls have neither type of functional checker.

Experimental setup

• Contestants tested: 12, spanning ten model families, each with its official harness, or OpenCode where no official harness exists (Table 6): GPT-5.6 Sol, Claude Opus 5, GLM-5.3, Gemini 3.7 Flash, Grok 4.6, Claude Opus 4.8, Qwen3.8 Max, Gemini 3.5 Flash, DeepSeek V4 Pro 0813, MiniMax M3, Kimi K3, HY4 Preview.
• Matches and challenges: 760 submitted matches yielding 466 valid matches; a reference set of 40 challenges (14 CTF, 26 open source). The pool has 46 challenges, 40 with six-mechanism profiling. The protocol is in Section 3.1, and prompts in Appendix C.2.
• Calibration and batches: 53 matches, six challenges, four reference models (Claude Opus 4.8, GPT-5.5, Gemini 3.5 Flash, Kimi K3), not entering the evaluation likelihood. 22 batches: three uniform pilots totaling 46 valid matches, then 24 matches per batch planned per Section 3.3.
• Metrics: Expected score is averaged uniformly over opponent and challenge configurations, on a 0–100 scale, with 90% model-conditional credible intervals. Round capture rates, flag-holding rates, and health probe pass rates are also reported. Bradley–Terry uses wins, draws, and losses from the same 466 matches to check the ranking.
• Behavioral audit: GPT-6 Sol and GPT-6.1 Sol audited all 466 matches, producing 2,397 findings, 673 candidate patterns, and 249 repeated across matches. The paper states this does not constitute independent manual verification of every finding. GPT-6 Sol is not a contestant.
• Exclusions: 294 matches do not count toward scores: 254 failed before completion, 25 canceled, 4 with unparseable records, 1 startup failure, and 10 completed matches with insufficient health probes. Failed tasks are replayed.

Main results

Swipe horizontally to read the table

Model	Expected score	90% interval	Valid matches	USD/match
GPT-5.6 Sol	78.6	76.8–80.2	83	17.7
Claude Opus 5	73.8	72.1–75.4	62	30.6
GLM-5.3	68.5	67.2–69.9	78	4.7
Gemini 3.7 Flash	67.1	65.9–68.2	76	7.3
Grok 4.6	65.5	63.9–66.9	86	5.4
Claude Opus 4.8	65.0	63.7–66.1	73	13.0
Kimi K3	57.0	55.2–58.9	77	4.7
HY4 Preview	56.3	54.7–58.2	81	1.2

Per Table 2. One match counts two contestants; Games is the number of valid matches for that contestant. Omitted: Qwen3.8 Max (62.3), Gemini 3.5 Flash (60.6), DeepSeek V4 Pro 0813 (59.5), MiniMax M3 (59.1); all numbers per Table 2.

• The authors state that in cumulative refits across 22 batches, GPT-5.6 Sol was consistently first, and in the final update all contestant scores changed by less than 1 percentage point (Figure 4). Gemini 3.7 Flash joined after batch 15 and Claude Opus 5 after batch 18, starting from a shared prior.
• Standard Bradley–Terry and expected score have Kendall τ=0.85, but the top two swap: Claude Opus 5 is first and GPT-5.6 Sol second. The authors explain that the former counts only the winner, while the expected score also accounts for margin. The authors report that these swaps are not statistically significant differences in latent capability (top pair p=0.41, others 0.64–0.99). There is also a swap between Grok 4.6 and Gemini 3.7 Flash, and a reversed order for Gemini 3.5 Flash, DeepSeek V4 Pro 0813, and MiniMax M3.
• The authors report Pearson r=0.85 between attack and defense capability across the 12 agents (Figure 3b); the same holds on five of six mechanism axes, while attack and defense are nearly uncorrelated on protocol and state (Figure 5). In Figure 6, GPT-5.6 Sol holds its flag in 83% of rounds and Claude Opus 5 in 81%, with the rest at 43–61%. After losing the flag in the previous round, the former loses it again in the next round 44% of the time, versus 62% for the remaining agents combined. The authors argue this is more consistent with between-round defensive adaptation.

Behavior in real-time adversarial play

• Event logs plus full trajectory audits. GPT-5.6 Sol's median time to first completed service restart in round 1 is 2.3 minutes, versus 3.9–12.3 minutes for the rest (Figure 9). In one Table 4 example, it captures a flag in 0.7 minutes and deploys a patch in 1.2 minutes, while Claude Opus 4.8 does not deploy until 18.9 minutes. In audited matches, agents from the other nine models identified a defensive fix but delayed deployment in 9% of matches.
• After losing the flag in rounds 1–2, the defender queries attacks against itself in 67% of cases and restarts before the next round in 57%. Among those that restart, 60% hold the flag in the next round, versus 9% for those that do not. The authors state this is a correlation and does not establish that the defense was effective.
• Table 3 (share of 466 audited matches; categories can overlap): partial patch leaving another observed path 21%; fix claims exceeding recorded tests 23%; actively sleeping through rounds 14%; over-blocking legitimate flows 8%; identified fix not deployed 9%; still using old targets or old proofs after rotation 8%; attacker-installed code still executable after fix or restart 4%. The authors state that Grok 4.6's action records are sparse, and its low recording frequency is hard to interpret (Figure 10). Cases also include bypassing blocked endpoints, reusing a rotated token after writing one's own packet capture, and a plugin still executable after ingress patching and restart (Figure 11, Table 4).

Single-player versus adversarial

• For GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3, single-player attack is two single-round attempts against an unhardened service (initial plus one retry), and a flag captured in either attempt counts as solved. Of the 40 challenges, 30 are solved by all three. The main text does not give per-model single-player percentages across all 40 challenges; Figure 12a plots this set of single-player results.
• In adversarial matches on these 30 challenges, round-3 flag capture rates are 52%, 37%, and 35%, respectively (Section 5.4). The introduction states a combined capture rate of 42% by round three. The authors speculate that this dynamic interaction can address benchmark saturation without continuously retiring problems.

Arena integrity

• An empty sandbox exposes only a static health endpoint and still places and rotates flags normally. In three initial matches, 6 captures were accepted through an unauthenticated spectator endpoint. After restricting the panel and excluding affected matches, three pairings were replayed, and none of the 22 submissions across 18 attacker rounds were accepted (Appendix E.2).
• A second probe kept the application vulnerabilities in etcd (CVE-2026-33343) and NATS (CVE-2026-58209) but moved the scoreable flag out of the data exposed by those vulnerabilities. No submissions were accepted in eight matches.
• Part of the ranking campaign occurred before the panel restriction. Record screening found no calls to or mentions of that route; one harness's transcript omits tool calls, so 121 of 1,336 accepted captures could only be screened against model text. The audit directly attributed 6 accepted captures to the shortcut, or 0.45%, spread across 3/466 matches (0.64%), which remain in the ranking.

Other ablations and analyses

• Model token costs for valid matches total $8,368, averaging $18.0 per match; including excluded runs, the campaign cost $10,542. Of 190,893 requests, 95% of prompt tokens came from provider cache; cached input accounts for 62% of cost and output for 13% (Figure 8).
• Across 466 matches there are 53,511 game tool calls, averaging 115 per match; get_game_state 25,066 times (53.8/match) and get_attacks_received 14,739 times (31.6/match) (Figure 13).
• nginx-ui-management's checker leaks memory on the runner; that challenge has no valid matches, remains in the reference set, and its score is predicted by shared parameters and priors (Section 5.2).
• In calibration, M3 with a previous-round capture term has the highest log score under batch, pairing, and challenge leave-outs, at -158.0, -158.4, and -186.1 (Table 8). On the held-out batch, it predicts 18.2 for the "100" sequence versus 31 observed, and 6.4 for "101" versus 1 observed (Appendix D.2).
• Appendix D.4: 72 draws. After Holm correction across all 66 pairs, seed 0 reports 25 pairs and seed 20260925 reports 27 pairs reaching the 5% level; the corrected p for GPT-5.6 Sol versus Claude Opus 5 is 1.0 under both seeds.
• Direct SSH login probes appear in 141 matches (30.3%), and the authors state there is no evidence these direct probes succeeded. Seven matches (1.5%) contain attempts to identify or favor the health checker by network address or account (Appendix E.3).

Limitations and future directions noted by the authors

• Incomplete functional checks: Automated functional tests cover only part of the expected behavior and may miss some degradation or failures on untested paths (Section 6).
• The checker can be treated differently: The audit found attempts in seven matches to identify the health checker by network address or account; some defenses let the checker through while denying other clients. Running checks over SSH on the agent's host also carries a state-forgery risk. The authors propose a shared ingress proxy, broader behavioral tests, and randomized checker identity (Section 6).
• Ecosystem coverage: Only 12 fixed model–tool combinations and a limited challenge domain have been explored so far; broader configurations and specialized vulnerability types are unmeasured. The authors state they open-source the orchestration, curation, challenge suite, and execution logs to invite distributed auditing and community submissions (Section 6).
• Attack scope: Paired, single-service matches omit volumetric vectors such as denial of service and multi-agent swarms. The authors argue that the latent model aggregates over discrete outcomes and is independent of agent count, so extending to distributed multi-service setups mainly requires orchestration updates (Section 6).
• Capabilities are not yet optimized: Currently they are only measured, not optimized. The authors propose applying offline policy optimization to these trajectories and then returning the improved contestants to the arena to close RAI (Section 6).
• Calibration scope: It covers only four contestants and six challenges, and does not establish nominal coverage for Laplace intervals; the intervals are reported as model-conditional in Section 3.2 (Appendix D.2).

Experimental coverage

• The ranking is a single campaign: 12 contestants, 40 reference challenges, 760 submitted matches, 466 valid matches, 22 batches; 294 excluded runs do not count toward scores (Section 5.2).
• The single-player comparison includes only GPT-5.6 Sol, Claude Opus 4.8, and GLM-5.3, with one initial attempt plus one retry per challenge (Section 5.4).
• Trajectory audits cover all 466 valid matches and were performed by GPT-6 Sol and GPT-6.1 Sol. The paper states that the process is traceable but does not constitute independent manual verification of every finding (Section 5.3). The paper does not report a numeric agreement between the audit and human judgment.
• Integrity probes include an empty sandbox, as well as etcd (CVE-2026-33343) and NATS (CVE-2026-58209); ranking records were additionally screened for panel routing (Section 5.5).
• Of the 46 challenges in the pool, 6 were played but not yet profiled and are not in the ranking reference; nginx-ui-management is in the reference set but has no valid matches (Section 5.2, Appendix B). Humans reviewed 27 challenges (Section 4).

CyberArena is a symmetric, multi-round, simultaneous attack-and-defense dynamic benchmark for cybersecurity agents.

• Problem: Isolated agents capturing flags or patching against static targets cannot measure speed and precision when an opponent changes conditions in real time; fixed tasks also saturate, and leaked or memorized exploits can still work.
• Approach: Two competitors host the same vulnerable service in isolated sandboxes, with three rounds of 20 minutes each, simultaneously capturing flags and patching their own side. The facility verifies flags, write markers, and health rather than accepting self-reports. An automated pipeline generates, reviews, and validates environments from CTF archives and public vulnerabilities. A latent model maps attack-defense differences, challenge difficulty, mechanism expertise, and round dependence to expected scores on a common scale, and allocates matches using expected information gain.
• Ranking: Among 12 frontier competitors, 40 challenges, and 466 valid matches, GPT-5.6 Sol has an expected score of 78.6, and Claude Opus 5 has 73.8 (Table 2). A win-loss-only Bradley–Terry model swaps these two, with Kendall τ=0.85 against expected scores; the authors report p=0.41 for this swap and do not treat it as a statistically significant difference in latent capability.
• Attack-defense and saturation: The attack and defense capabilities of the 12 agents have Pearson r=0.85. On the 30 challenges all three solved solo, round-3 flag capture rates are 52%, 37%, and 35%. The introduction gives a combined rate of 42%.
• Behavior: In audited matches, partial patches account for 21%, and fix claims exceeding recorded tests account for 23%. There are also attacker codes that remain executable across rounds, and credential rotation after self-written packet captures. The authors state these trajectories resemble sparring more than a single exam.
• Authors' conclusion: Treating the opponent as a measurement instrument can record behaviors that static benchmarks miss; static puzzles saturate, and competitive arenas can serve as a foundation for measuring and guiding the co-evolution of attack and defense. The authors also state that functional check coverage is incomplete, calibration is narrow, and current measurements only cover capability, without yet using these trajectories to optimize strategy.