ExploitGym: can AI agents turn security vulnerabilities into real attacks?

The core question is: can an agent progress from "triggering a vulnerability" to a complete exploit with real security impact? The authors separate vulnerability reproduction from exploitation, and focus on measuring unauthorized code execution.

• Application scenarios: the authors argue that exploitation capability can help defenders assess severity, prioritize patching, and verify mitigations, and may also lower the barrier for attackers to develop exploits.
• Existing evaluations: the authors point out that some benchmarks mainly test source-level reasoning; existing benchmarks that include exploitation tasks focus on CTF or user-space software, with at most 200 items (Table 1).
• Capability requirements: the authors emphasize that exploitation requires handling runtime memory layout, instruction-level control flow, constrained inputs, and multi-stage exploit chains, and may also require bypassing system defenses.
• Proposed benchmark: ExploitGym provides real vulnerabilities, PoVs, source code, and reproducible environments, covering user space, V8, and the Linux kernel, with defense toggles.
• Success boundary: the agent interacts with a controlled remote target and must obtain a dynamic flag beyond the authorized scope, after which an agent judge checks whether the specified vulnerability was actually exploited; obtaining the flag alone does not count as final success.

Related research falls into four groups: vulnerability discovery infrastructure, automated exploitation, agent security benchmarks, and dangerous capability evaluation.

Vulnerability discovery and archive infrastructure

• AFL++, libFuzzer, and domain fuzzers—discover vulnerabilities through fuzzing; the paper also discusses fuzzers for JavaScript engines and operating system kernels.
• OSS-Fuzz, ClusterFuzz, syzbot, kernelCTF—continuously discover, reproduce, or archive vulnerabilities and related inputs, versions, and exploit materials. The authors point out that these systems were not originally standardized evaluation benchmarks.

Exploitation techniques and automated exploit generation

• Stack smashing, ROP, DOP—the paper lists these as exploitation techniques for countering system mitigations.
• AEG, Mayhem, Gollum, MAZE, etc.—study automated exploit generation, heap layout manipulation, and related problems. The authors summarize that such methods often make strong assumptions about vulnerability class, exploit structure, or runtime environment.

Agent cybersecurity benchmarks

• NYU CTF, Cybench, CVE-Bench, BountyBench—include exploitation tasks; Table 1 lists 200, 40, 40, and 40 items respectively, in the CTF or user-space software domain.
• CyberGym, PatchAgent, etc.—cover complementary capabilities such as vulnerability reproduction, patch generation, and secure code generation; Table 1 also lists BaxBench, SEC-bench, SeCodePLT, SecRepoBench, and SecureAgentBench.
• Experimental comparison—the main experiments compare the performance of different models and agent frameworks on ExploitGym; the above benchmarks are not used as experimental datasets, and no comparison results for traditional automated exploit generators are reported.

Dangerous capability evaluation

• AgentHarm, WMDP, and work by Phuong et al.—the paper positions itself as part of frontier model dangerous capability evaluation, and emphasizes that a higher ExploitGym score does not mean the model is safer overall.

The distinction the authors emphasize is: turning real vulnerability materials from multiple software domains into unified tasks, combined with configurable defenses, reproducible environments, and verification of the specified vulnerability, rather than only checking whether a security impact was obtained.

ExploitGym packages trigger materials for existing vulnerabilities into remote exploitation tasks, then separately verifies the security impact and the exploitation path. Vulnerability hints are provided by default, but patches are not.

Task inputs and interaction

• Build information: provides source code, build configuration, scripts, compiled binaries or kernel images, and runtime configuration.
• Vulnerability information: provides the PoV and a textual description; information such as patches can be included or hidden separately, and patches are hidden by default.
• Remote target: the agent develops locally and interacts with the remote target over multiple rounds through a restricted entry point, and can request that the controller reset the target. All components are packaged in a containerized environment (Figure 1).

Three types of security boundaries

• Userspace: the target runs as an unprivileged user, with sanitizers disabled at compile time; obtaining the flag requires achieving code execution in the target process. Defense toggles are ASLR+PIE and stack canary.
• Browser (V8): uses a standalone d8 shell of the vulnerable version, with convenience interfaces that can directly perform system operations or read files removed, keeping only the entry points needed for the PoV. Defense toggles are ASLR and the V8 heap sandbox.
• Linux kernel: each connection enters a QEMU/KVM virtual machine, and the agent process is confined by nsjail; the flag is located on the raw block device, and obtaining UID 0 within a user namespace alone is still not enough to read it. Configuration items are KASLR and user-namespace access (Table 2).

Data construction and quality control

• User-space filtering: OSS-Fuzz vulnerabilities are obtained from CyberGym, supplemented with vulnerabilities in OSV that were not found by fuzzing. The latter have PoVs generated by Claude Code with Claude Opus 4.6, then manually verified; after rebuilding targets with sanitizers disabled, 520 items across 161 projects are retained.
• V8 filtering: collects ClusterFuzz and manual reports submitted after 2024, and recovers PoVs from patch tests or attachments. Candidate vulnerable versions are built and triggering is verified, retaining 185 items from 403 candidates.
• Kernel filtering: kernelCTF complete exploits are reduced to PoVs that only trigger the vulnerability; syzbot x86/x86_64 high-severity memory safety and data race reports are deduplicated and reproduced, ultimately retaining 193 items.
• Excluding shortcuts: user-space filtering removes entry points that can directly achieve code execution through script interfaces; the V8 shell interface is reduced. Appendix B gives details on filtering, defense configuration, and permission control.

Two-stage success verification

• Flag verification: each target generates a dynamic flag placed where normal authorized interfaces cannot access it; submitting the correct value is a necessary condition.
• Agent-as-a-judge: the judge reads the full trajectory, task inputs, and agent artifacts, checks whether the specified vulnerability was exploited, and rules out shortcuts caused by other vulnerabilities or known public exploits.
• Judge review process: typically GPT-5.5/Codex CLI and Claude Opus 4.6/Claude Code review separately; agreement is accepted, and disagreements go to a human. Experiments run by Anthropic use only the Claude Code judge (Section 4 footnote).

The main experiments compare the number of successful exploits of the specified vulnerability across model-framework combinations, supplemented by defenses, time budget, and judge validation.

Experimental setup

• Models and frameworks tested: Claude Mythos Preview, Claude Opus 4.6, Claude Opus 4.7, GLM-5.1 with Claude Code; GPT-5.4, GPT-5.5 with Codex CLI; Gemini 3.1 Pro with Gemini CLI.
• Data and budget: ExploitGym has 898 items in total: 520 Userspace, 185 V8, 193 kernel. The main experiments have system defenses off, one attempt per task, and a wall-clock limit of 2 hours.
• Model configuration: Table 6 lists 7 models, although the beginning of Appendix C says "six models." Reasoning settings are, in order, max, max, xhigh, auto, xhigh, xhigh, high; each agent is limited to 4 CPU cores and 8 GB of memory.
• Prompts and access: the main experiments disable deployment-time content filters under the trusted access program, and refusals formed during training may still occur. The same user prompt is generally used, with Mythos Preview additionally using a CLAUDE.md with adjusted instructions; external network access is restricted by an allowlist.
• Metrics and determination: Flag means obtaining the dynamic flag; Success additionally requires passing the specified-vulnerability review. The judges are GPT-5.5 and Claude Opus 4.6, with disagreements going to a human; Anthropic experiments use only the latter.
• Other statistics: Table 3 reports the average estimated cost, time, and number of LLM calls per task for the successful subset and for the full benchmark; no repeated-run statistics for the main results are reported.

Main results

According to Table 3, all with system defenses off and 2 hours per task; U, B, K are Userspace, V8, kernel respectively, and the numbers are Success counts.

Swipe horizontally to read the table

Model	Framework	Total	U	B	K
Claude Mythos Preview	Claude Code	157	107	38	12
Claude Opus 4.6	Claude Code	15	12	2	1
Claude Opus 4.7	Claude Code	7	4	3	0
Gemini 3.1 Pro	Gemini CLI	12	10	2	0
GLM-5.1	Claude Code	4	4	0	0
GPT-5.4	Codex CLI	54	38	15	1
GPT-5.5	Codex CLI	120	71	27	22

• Authors' interpretation: the authors believe that complete exploitation remains challenging, but some frontier configurations can already complete real vulnerability exploitation; kernel task successes are a capability signal they focus on.
• Exceptions and refusals: Claude Opus 4.7 has fewer successes than 4.6. The authors observed that 4.7 and Gemini often end early after judging the vulnerability unexploitable; GPT-5.4 and GLM-5.1 had 36 and 23 safety refusals respectively.
• Safety filter: when GPT-5.5 restores the default filter and uses the default prompt, 88.2% of tasks are blocked before any tool call; the remaining tasks average 4.4 valid LLM requests, and the authors report that all remain in the reconnaissance stage.

Rerun with system defenses enabled

Table 5 reruns only tasks that previously succeeded. The table below shows successes with defenses off → successes with defenses on, and cannot be treated as an independent evaluation of the full benchmark with defenses enabled.

Swipe horizontally to read the table

Model	Userspace	V8	Kernel
Claude Mythos Preview	107→25	38→17	12→3
Claude Opus 4.6	12→0	2→0	1→0
Claude Opus 4.7	4→0	3→0	0→0
Gemini 3.1 Pro	10→0	2→0	0→0
GLM-5.1	4→0	0→0	0→0
GPT-5.4	38→2	15→0	1→1
GPT-5.5	71→10	27→3	22→8

• Authors' interpretation: the authors believe that common mitigations prevented most exploits, but some agents could still adapt to defenses. The table does not break down the independent effect of each defense toggle.
• Failure case: the GPT-5.4 V8 case in Figure 4 achieved code execution with defenses off; the authors report that it no longer succeeds after the relevant defenses are restored.

Obtaining the flag vs. exploiting the specified vulnerability

• Metric difference: in Table 4, Mythos Preview has 226 Flags and 157 Successes, an alignment rate of 69.5%; GPT-5.5 has 210, 120, and 56.7%. The authors state that some trajectories turned to neighboring vulnerabilities or other attack surfaces.
• Expert validation: 59 trajectories were independently labeled by two authors with disagreements adjudicated, excluding 1 trajectory on which both were uncertain, leaving 58. The GPT-5.5 judge matched 58/58, κ=1.00; Claude matched 56/58, κ=0.931.
• Production review: excluding Anthropic experiments, of 313 reviewed tasks the two judges agreed on 294 (93.9%). Both of Claude's errors in expert validation judged exploitation of another vulnerability as exploitation of the specified vulnerability.

Time budget and success set

• Extended runs: Figure 2 uses a 6-hour limit and compares only Mythos Preview and Opus 4.6. The text figure shows endpoint labels of 204 and 16, and 127 and 15 near 2 hours; 127 differs from the 157 in Table 3, and the paper does not explain the discrepancy.
• Curve interpretation: the authors state that Opus 4.6 plateaus after the first 30 minutes, while Mythos Preview continues to gain successes after 2 hours. The curve shapes are not visible in the supplied text, so the trend cannot be independently verified.
• Complementary results: Figure 3 marks 56 items unique to Mythos Preview, 26 unique to GPT-5.5, and 4 unique to other models; the main text reports that the first two share 91 items, and the total union including extended runs is 239 items.

Other ablations and analyses

• Cost and calls: in Table 3, GPT-5.5 averages $34.55 and 375.4 LLM calls per task on the full benchmark; the successful subset is $22.99 and 256.8.
• Case trajectory: Figure 4 labels the GPT-5.4 V8 case as taking 71 minutes and costing $15.80; only the result is reported here, without restating the exploit construction.

The authors explicitly distinguish the effects of task failure, unexploitable vulnerabilities, and evaluation budget.

Limitations and future directions noted by the authors

• Target scope: does not cover Windows, iOS, Android, or their applications (Section 5, Limitations).
• Success metric: the code execution metric does not record arbitrary read/write primitives, sandbox escapes without code execution, or partial exploit progress (Section 5).
• Failure interpretation and ground truth: failures may stem from safety refusals, tool misuse, or the vulnerability being unexploitable; not every task has a known complete exploit. The authors also believe that the fact that complete solutions are not widely public helps mitigate data contamination (Section 5).
• Attempts and resources: only one attempt per task, constrained by time and cost; more attempts or resources may increase the success rate (Section 5).
• Instruction setup: a single set of instructions may favor a particular model; customized instructions or additional task context may increase the success rate (Section 5).
• Specialized tools: no vulnerability analysis or exploitation-specific tools are provided; the authors believe that integrating such tools may increase the success rate (Section 5).

Experimental coverage

• Software and scale: 898 tasks divided into 520 Userspace, 185 V8, and 193 Linux kernel (Table 2).
• Configuration comparison: the model-framework pairings in Table 3 are fixed; Mythos Preview uses additional instructions, and the review process also has an Anthropic experimental exception (Section 4).
• Defense evaluation: Table 5 reruns with standard defenses enabled on tasks that had already succeeded with defenses off; no independent ablation results for each toggle are reported.
• Budget analysis: the main experiments use 2 hours per task; the 6-hour analysis involves only Mythos Preview and Opus 4.6 (Figure 2).
• Human agreement: judge validation is based on 58 trajectories with definite expert labels; the production phase separately reports dual-judge agreement on 313 non-Anthropic experimental items (Section 4).

ExploitGym is a benchmark for agents' real-world vulnerability exploitation capability, focused on distinguishing "triggering a bug" from "achieving unauthorized code execution".

• Task design: The benchmark provides source code, PoV, vulnerability descriptions and a runtime environment, with the patch hidden by default. After obtaining the dynamic flag, it must also be reviewed whether the specified vulnerability was used, to avoid counting successes caused by other vulnerabilities in the main metric.
• Main experiment: Under the conditions of 898 tasks, system defenses off, and 2 hours per task, Claude Mythos Preview / Claude Code succeeded on 157 items, and GPT-5.5 / Codex CLI succeeded on 120 items (Table 3).
• Kernel results: Under the same setting, GPT-5.5 completed 22 kernel tasks; the authors regard such successes as a signal of capability to handle complex exploitation environments (Table 3).
• Defenses and metrics: Rerunning previously successful tasks with defenses on, Mythos Preview's V8 success count changed from 38 to 17 (Table 5). GPT-5.5's Flag count is 210 and specified-vulnerability Success is 120; the two cannot be conflated (Table 4).
• Evidence boundaries: The authors did not provide complete exploitation ground truth for each task, and the main evaluation allows only one attempt within the budget; therefore failures cannot be directly interpreted as a lack of exploitation capability, and success counts are not ground truth for the exploitability of all vulnerabilities.
• Authors' conclusion: The authors believe that agents autonomously developing exploits has become a capability that needs to be incorporated into threat modeling; current success rates cannot serve as a long-term security guarantee, and traditional hardening measures still need to be combined with evaluations oriented toward autonomous exploitation.