CyberGym: Large-scale evaluation of AI agents' real-world cybersecurity capabilities

1. 01What problem does this paper attempt to solve?

   Existing AI cybersecurity evaluations are small in scale and only test static results; the authors propose CyberGym, a benchmark of 1,507 real-vulnerability instances.

   How to evaluate AI agents' cybersecurity capabilities at scale and realistically, and measure their actual impact on the current security ecosystem.

   • Problem setting: The authors believe LLM agents' capabilities on software engineering tasks are increasing, bringing both opportunities and risks to cybersecurity, so rigorously evaluating their security capabilities is both critical and urgent.
   • Existing shortcomings: The authors point out that existing benchmarks have two limitations: (i) small scale (at most 200 instances, see Table 1), relying on extensive manual construction or fragile data sources, making evaluation unstable; (ii) only focusing on static benchmark instances, making it difficult to judge agents' impact on the evolving real security landscape.
   • Core observation: OSS-Fuzz has discovered more than 13,000 vulnerabilities since 2016, each with PoC and patch information; the authors use it as a data source of large-scale real historical vulnerabilities, and use the sanitizer as a reliable vulnerability detection oracle.
   • Paper proposes: The CyberGym benchmark, containing 1,507 instances covering 188 software projects; the main task is, given a textual vulnerability description and the pre-patch codebase, to have the agent generate a PoC that reproduces the vulnerability; 4 difficulty levels are also set, and open-ended zero-day vulnerability discovery is supported.
   • Task nature: This is evaluation/benchmark work and does not involve an attacker threat model; the subjects under test are AI agent frameworks and backbone LLMs, and the victims are real open-source projects containing historical vulnerabilities (executed within containerized environments).

   Collapse this answer

2. 02What related research exists?

   Related work falls into two categories: cybersecurity benchmarks and coding benchmarks; the authors state that CyberGym is seven times the scale of other security benchmarks and can discover zero-days.

   Cybersecurity benchmarks (CTF-type)

   • NYU CTF Bench (Shao et al., 2024) and Cybench (Zhang et al., 2025b): Benchmarks based on CTF challenges, with 200 and 40 instances respectively (Table 1). The paper points out that CTF challenges are idealized settings and often fail to capture real-world complexity.

   Cybersecurity benchmarks (real-project type)

   • AutoAdvExBench (Carlini et al., 2025), CVE-Bench (Zhu et al., 2025), BountyBench (Zhang et al., 2025a), SEC-Bench (Lee et al., 2025): Benchmarks that turn to real software projects, with 40–200 instances covering 26–41 projects (Table 1). The paper points out that these works only focus on known historical vulnerabilities, whereas CyberGym can also discover zero-day vulnerabilities.
   • ARVO (Mei et al., 2024): Infrastructure that collects OSS-Fuzz vulnerabilities and provides reusable Docker images; the paper points out that ARVO itself does not define evaluation tasks and metrics and cannot serve as a benchmark on its own, and that 1,368 of CyberGym's instances originate from ARVO.

   Coding benchmarks and coding agents

   • SWE-bench (Jimenez et al., 2024) and SWT-bench (Mündler et al., 2024): Have agents generate a PR from an issue, or write unit tests to verify a PR, respectively. The paper points out that they focus on functional fixes, mostly local code changes, whereas CyberGym requires repository-wide reasoning.
   • OpenHands (Wang et al., 2025), Codex (OpenAI), and the specialized models SWE-Gym (Pan et al., 2024) and R2E-Gym (Jain et al., 2025): Agents and fine-tuned models spawned by coding benchmarks; this paper uses them as subjects under test.

   Baselines and comparison setup

   • This is a benchmark paper with no method baseline; the subjects under test are 4 agent frameworks (OpenHands, Codex CLI, EnIGMA, Cybench agent) and 11 LLMs, the comparison benchmark is CyberGym itself, and Table 1 compares scale with the above 6 security benchmarks.

   Authors' positioning: CyberGym has 1,507 instances and 188 projects; the authors state that its scale exceeds other cybersecurity benchmarks sevenfold, and it is the only one among them that can discover zero-day vulnerabilities (Table 1); it is complementary to, not a replacement for, SWE-bench.

   Collapse this answer

3. 03How does the paper address this problem?

   1,507 real vulnerability instances were built from OSS-Fuzz, success of PoC reproduction is judged by executing both versions with the sanitizer, and 4 difficulty levels are set.

   Overall approach: package real vulnerabilities discovered and already patched by OSS-Fuzz into containerized evaluation instances, have agents generate PoCs to reproduce the vulnerabilities, and automatically judge success or failure based on execution results; the framework is named CyberGym.

   Task setup and success criteria

   • Main task (Level 1): Input is a textual vulnerability description (including approximate location, type, and root cause) plus the pre-patch codebase and executable files; the agent outputs a PoC input file, which can be submitted via submit.sh to obtain execution feedback and iterate.
   • Success criteria: The PoC must trigger a crash on the pre-patch version with sanitizer enabled, and must not trigger any sanitizer crash on the post-patch version; only then is it considered successful. The metric is success rate (proportion of successful instances).
   • Reason for choosing reproduction tasks (per the authors): Human experts need about 5 hours to reproduce a known vulnerability (Mu et al., 2018), and automated fuzzing tools have a median time of 324 days to discover vulnerabilities in OSS-Fuzz projects (Keller et al., 2023), indicating that the task itself is difficult.

   Four difficulty levels

   • Level 0: Only the pre-patch codebase is given, with no description, i.e., open-ended vulnerability discovery; also used for zero-day discovery in Section 5.
   • Level 1: Codebase plus textual description, i.e., the main task, simulating a scenario where a CVE has only a textual report.
   • Level 2: Adds the crash stack trace of the ground truth PoC on top of Level 1.
   • Level 3: Adds the ground truth patch in diff format and the post-patch codebase on top of Level 2, simulating a real one-day scenario.

   Data construction and quality control

   • Source: 1,507 vulnerabilities disclosed between 2017-01-01 and 2025-04-21; 1,368 of them from ARVO (as of 2024-07-31), plus 139 additionally collected newer vulnerabilities. Binary search is used to locate the patch commit, obtaining the pre-/post-patch codebases, ground truth PoC, and patch.
   • Description generation: GPT-4.1 is used to rewrite the patch commit message into a vulnerability description; the filtering and rewriting prompts are in Appendix C (Figure 9, 10).
   • Quality filtering: GPT-4.1 acts as a reviewer to remove commits with insufficient information or that fix multiple issues; the ground truth PoC is rerun to verify reproducibility; duplicate instances are removed by crash stack trace. Manual verification of 300 stratified samples shows a filtering precision of 96%, with Cohen's κ of 0.82±0.03 between two annotators (Appendix C).

   Scale and statistics

   • Descriptions and PoCs: The median description length is 24 words, with a maximum of 158 words; the median ground truth PoC is 259 bytes, with a maximum of 1,048,576 bytes (Table 3).
   • Codebases and patches: The median codebase has 1,117 files and 387,491 lines; the median patch changes 1 file and 7 lines, with a maximum of 40 files and 3,456 lines (Table 3).
   • Coverage: 188 projects and 28 sanitizer crash types (Table 4, 5); 62.4% of instances come from projects outside the top 10.

   Evaluation infrastructure

   • A modular containerized environment; the submission server uses about 10TB of disk to store task environments; a randomly sampled subset of 300 instances (about 20%) is also provided for low-cost evaluation; full evaluation requires about $3,000 in API credits.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluation of 11 LLMs and 4 agent frameworks: OpenHands+Claude-Sonnet-4 reaches 17.9%, and GPT-5 with thinking enabled reaches 22.0%.

   Experimental setup

   • Models under test: 11 LLMs—GPT-4.1, GPT-5, o4-mini, Claude-3.7-Sonnet, Claude-Sonnet-4, Gemini-2.5-Flash (closed-source general); Qwen3-235B-A22B, DeepSeek-V3 (open-source general); SWE-Gym-32B, R2E-Gym-32B, OpenHands-LM-32B (SWE-specific).
   • Agent frameworks: OpenHands, OpenAI Codex CLI, EnIGMA, Cybench agent, with a uniform cost budget of about $2.0 per task and at most 100 iterations.
   • Benchmark and metric: The full CyberGym set of 1,507 instances (some experiments use the 300-instance subset), with success rate as the metric; Level 1 by default unless noted otherwise.
   • Judgment method: Execution-based judgment—the PoC triggers a sanitizer crash on the pre-patch version and does not crash on the post-patch version; no LLM review participates in scoring.
   • Default configuration: The main experiments disable thinking (o4-mini does not support disabling it, and GPT-5 uses minimal reasoning), and the agent scaffold is OpenHands.
   • Number of repetitions: The paper does not report the number of runs per instance.

   Main results

   Success rates of each LLM with OpenHands at Level 1 (Figure 3):

   Model	Success rate (%)
   Claude-Sonnet-4	17.9
   Claude-3.7-Sonnet	11.9
   GPT-4.1	9.4
   GPT-5 (minimal)	7.8
   Gemini-2.5-Flash	4.8
   DeepSeek-V3	3.6
   o4-mini	2.5
   R2E-Gym-32B	2.0
   Qwen3-235B-A22B	1.9
   OpenHands-LM-32B	1.7
   SWE-Gym-32B	0.1

   • The authors state that SWE-specific models perform strongly on SWE-bench but achieve no more than a 2.0% success rate on CyberGym, showing the complementarity of the two benchmarks.
   • The success rate of the union of results across all models reaches 27.2%, and the authors state that the tasks succeeded by different models overlap little.
   • The authors report that o4-mini often conservatively requests user confirmation and terminates execution early, a pattern not observed in other models.

   Thinking mode and agent framework comparison

   • Thinking experiment: 4 models are compared on the 300-instance subset (Figure 4). GPT-5 rises from 7.7% (minimal reasoning) to 22.0% (high reasoning), surpassing Claude-Sonnet-4; Qwen3-235B-A22B rises from 2.7% to 4.7%, Claude-3.7-Sonnet from 14.3% to 17.3%, and Claude-Sonnet-4 from 17.7% to 19.3%. The authors state that thinking brings a large improvement to GPT-5 and limited improvement to other models.
   • Framework comparison: With GPT-4.1 as the backbone at Level 1 (Figure 5), EnIGMA 7.2%, Codex 7.4%, Cybench 9.0%, OpenHands 9.4%; the union of results across the four frameworks reaches 18.4%. The authors state that the success rates of the frameworks are close but their success sets overlap little, and their behavior patterns differ (OpenHands uses more Bash command chains, while CTF frameworks rely more on Python scripts, Figure 17).

   Data contamination and difficulty analysis

   • Data contamination: Instances are split into before and after each model's knowledge cutoff; for the 4 models with more than 50 post-cutoff samples, Fisher's exact test and two-proportion Z-test are performed (Table 2), and all p-values are greater than 0.1 (e.g., Claude-3.7-Sonnet 11.9% before vs. 12.5% after, p=0.87); the authors state that no statistically significant difference was found.
   • Difficulty levels: OpenHands+GPT-4.1 achieves success rates of 3.5%, 9.4%, 13.1%, and 17.1% at Level 0/1/2/3 respectively (Figure 6); Level 0 reproduced 5 of the 142 vulnerabilities after GPT-4.1's cutoff date.
   • PoC length: The longer the ground truth PoC, the lower the success rate (Figure 7); GPT-4.1 achieves 43.5% and Claude-Sonnet-4 55.3% in the [0,10) byte range, while instances with PoCs longer than 100 bytes (65.7% of the full benchmark) have a success rate of only about 10%.

   Other ablations and analyses

   • Crash type breakdown: OpenHands+Claude-Sonnet-4 achieves success rates of 10.6%–25.0% on the top 10 crash types (Table 8).
   • Balanced resampling: Conclusions remain unchanged after balanced resampling by project or crash type (Figures 15, 16).
   • Failure modes: For OpenHands+GPT-4.1, the early-termination proportion is about 0.26–0.30, and the long plaintext PoC proportion is 0.14–0.19 (Table 9).
   • Execution steps: Successes are mostly concentrated in 20–80 steps, and nearly half of the runs fail near the 80–100 step limit (Figure 18).
   • Security impact: Of the PoCs generated during evaluation, 759 crash on the post-patch version, confirming 9 zero-days (existing on average 969 days) and 18 incomplete patches; additionally, Level 0 was run on 431 projects, with GPT-4.1 confirming 7 and GPT-5 confirming 22 zero-days (4 overlapping), totaling 34 zero-days, for which 4 CVE IDs have been obtained and 10 have been patched.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors state that, limited by the dependence on sanitizers, they only cover C/C++ memory safety vulnerabilities and PoC tasks, and list two directions for future work: benchmarks and agents.

   Limitations and future directions noted by the authors

   • Vulnerability types and language scope: CyberGym currently mainly covers memory safety vulnerabilities in C/C++ projects, which stems from its dependence on sanitizer detection and may fail to capture the full picture of security vulnerabilities (Section 3.1, Section 6).
   • Extending vulnerability categories and platforms: The authors plan to extend to types such as logic vulnerabilities and cryptographic weaknesses, and to cover web and mobile platforms and more programming languages (Section 6).
   • Extending task types: Currently only PoC generation is done; the authors plan to support tasks such as patching (defense) and exploitation (offense); they note that patch evaluation needs to address challenges such as determining functional preservation and standardizing heterogeneous test systems, and that exploitation evaluation requires finer-grained oracles (Section 6).
   • Directions for agent improvement: The authors suggest strengthening LLM long-context reasoning, designing ensemble frameworks that combine the complementary strengths of various agents, developing specialized security tools, and optimizing tool-use patterns (Section 6).

   Scope of experiments

   • The models tested are 11 LLMs and 4 agent frameworks; the main experiments with backbone models all use OpenHands as the scaffold, and the framework comparison experiments all use GPT-4.1 as the backbone (Section 4).
   • The thinking mode comparison is conducted only on the 300-instance subset with 4 models (Figure 4).
   • The data contamination analysis performs statistical tests only on the 4 models with more than 50 post-cutoff samples (Table 2).
   • The zero-day discovery experiments use OpenHands with GPT-4.1 and GPT-5, covering 431 projects and 1,748 entry executables (Section 5).
   • The paper does not report the number of repeated runs per instance; the manual verification for quality filtering uses 300 stratified samples (Appendix C).

   Collapse this answer

6. 06Summarize the main content of the paper

   CyberGym evaluates agents with 1,507 real vulnerabilities, with a highest success rate of 22.0%, and incidentally discovered 34 zero-day vulnerabilities.

   CyberGym is a large-scale evaluation benchmark for AI agents' cybersecurity capabilities, built to address the problems that existing benchmarks are small in scale and only test static results.

   • Problem to solve: The authors believe existing security benchmarks have at most 200 instances and only focus on known historical vulnerabilities, making it difficult to stably evaluate agents' impact on the real security ecosystem.
   • Method highlights: 1,507 real vulnerabilities were collected from OSS-Fuzz (188 C/C++ projects); the agent generates a PoC based on the vulnerability description and the pre-patch codebase; success is judged by the sanitizer crashing on the pre-patch version and not crashing on the post-patch version; four difficulty levels, Level 0–3, are set, all executed in containers.
   • Main results: OpenHands+Claude-Sonnet-4 achieves a 17.9% success rate at Level 1 (Figure 3); GPT-5 rises from 7.7% to 22.0% after enabling thinking (Figure 4); SWE-specific models do not exceed 2.0%; the four agent frameworks each achieve 7.2%–9.4%, with a union of 18.4% (Figure 5).
   • Security impact: During evaluation, 18 historically incomplete patches were found; the open-ended discovery experiment confirmed a total of 34 zero-day vulnerabilities, which were responsibly disclosed, obtaining 4 CVE IDs, with 10 patched.
   • Authors' conclusion: The authors state that CyberGym can effectively distinguish the security capabilities of agents and models, forms a difficulty ladder, and can serve as a proxy for agents' real security capabilities; the current strongest combination also achieves only about a 20% success rate, indicating that the benchmark is overall highly difficult.

   Collapse this answer

Related papers