Skip to content

DoGNAVY-Exploit: Stage-Aware Vulnerability Exploitation with Parallel Open-Weight Agents

DoGNAVY-Exploit: Harnessing Parallel Open-Weight Agents for Stage-Aware Vulnerability Exploitation

arXiv· arXiv:2611.04196· Yuxuan Wang, Dadi Guo, Weijie Guo, Zikai Xu, Chen Lu, Yizhe Zhuang, Peng Huang, Yuejin Xie, Guopeng Lin, Xuangan Xiao, Shipei Qu, Dongrui LiuShanghai Artificial Intelligence Laboratory; HKUST; DARKNAVY; Tsinghua UniversityIndexed by this site · 16 days ago· Originally publishedAttacks

Quick read

Attacks

Based on the paper PDF (AI-generated); refer to the original

The authors propose a multi-agent framework, DoGNAVY-Exploit, and state that it enables the open-weight model GLM-5.2 to reach 97.5% Pass@1 on Cybench and 97.5% (zero-day)/100% (one-day) on CVE-Bench.

Threat modelTask setting: given a vulnerability description and PoC (proof-of-vulnerability), the agent interacts with the target in an executable environment and progressively develops the known vulnerability into a complete exploit, with goals such as obtaining a flag, arbitrary code execution, or privilege escalation; evaluation is conducted in controlled benchmark environments (Cybench, CVE-Bench, ExploitGym).

Questions
Frontier closed-source models can turn known vulnerabilities into usable exploits, but at high cost and often refuse cybersecurity requests due to safety guardrails; open-weight models are cheap and available, yet struggle to maintain progress in long exploit chains and to apply security knowledge at the appropriate stage.
Methods
The authors propose the multi-agent framework DoGNAVY-Exploit: it decomposes the exploitation process into three milestone stages; within a stage, a route planner proposes multiple candidate routes, parallel executors verify them, and a controller adjudicates based on runtime evidence; expert knowledge is progressively disclosed by stage and role, and a shared structured memory accumulates verified facts and failed routes.
Experiments and results
With GLM-5.2 as the backbone, Cybench Pass@1 is 97.5%, CVE-Bench zero-day 97.5% and one-day 100%; on ExploitGym, GLM-5.2 solves 79/225 problems and GLM-5.3 solves 157/485 problems, versus 39 and 130 problems for the corresponding public Claude Code baseline; the GLM-5.2 configuration costs $4.80 per case and produces 18.94 exploits per $1,000.
Limitations and future work
The authors do not have a dedicated limitations section; due to resource constraints, ExploitGym evaluates only a subset (225/485 tasks, selected by the team's security experts), and the task coverage and time budget differ from the public baselines (8 hours versus 6 hours), so the authors switched to cost comparison; the public frontier numbers come from their respective original settings and are not controlled comparisons.
Experimental setupGLM-5.2、GLM-5.3 · Cybench、CVE-Bench et al. · Pass@1、on-target exploits et al.
Evaluated models
GLM-5.2GLM-5.3
Benchmarks
CybenchCVE-BenchExploitGym
Metrics
Pass@1on-target exploitscost per case (USD)successful exploits per $1,000
AI summaryThe researchers propose DoGNAVY-Exploit, a multi-agent framework that enables open-weight models to complete long-chain vulnerability exploitation, with GLM-5.2 reaching 97.5% Pass@1 on Cybench. The framework uses configurable milestones to break long-chain exploitation into short, verifiable capability jumps, disclosing relevant expert knowledge only at the corresponding stage; hierarchical orchestration lets planners propose multiple attack routes and executors test them in parallel, with the controller accepting progress only when runtime evidence supports it, and a shared structured memory aggregates verified facts, artifacts, reproduction steps, and failed routes. On CVE-Bench, GLM-5.2 achieves 97.5% zero-day and 100% one-day vulnerability Pass@1; on ExploitGym, GLM-5.2 solves 79 tasks and GLM-5.3 solves 157, compared with Claude Code public baselines of 39 and 130 respectively. Ablation results show that stage-aware orchestration, expert knowledge, parallel exploration, and the memory design all improve the vulnerability exploitation capability of open-weight agents.

The researchers propose DoGNAVY-Exploit, a multi-agent framework that enables open-weight models to complete long-chain vulnerability exploitation, with GLM-5.2 reaching 97.5% Pass@1 on Cybench. The framework uses configurable milestones to break long-chain exploitation into short, verifiable capability jumps, disclosing relevant expert knowledge only at the corresponding stage; hierarchical orchestration lets planners propose multiple attack routes and executors test them in parallel, with the controller accepting progress only when runtime evidence supports it, and a shared structured memory aggregates verified facts, artifacts, reproduction steps, and failed routes. On CVE-Bench, GLM-5.2 achieves 97.5% zero-day and 100% one-day vulnerability Pass@1; on ExploitGym, GLM-5.2 solves 79 tasks and GLM-5.3 solves 157, compared with Claude Code public baselines of 39 and 130 respectively. Ablation results show that stage-aware orchestration, expert knowledge, parallel exploration, and the memory design all improve the vulnerability exploitation capability of open-weight agents.

Related events and news

1 events · 1 news mentions

News that mentions this paper

大模型安全周报:智能体安全边界在签名、授权与上下文

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    How to give low-cost open-weight models strong vulnerability exploitation capability, to replace expensive and restricted closed-source frontier models.

    Core question: how to give open-weight, low-cost models strong vulnerability exploitation capability.

    • Scenario and motivation: LLM agents can already perform vulnerability exploitation in real execution environments—iteratively exploring multiple attack directions, discovering exploitable primitives, and composing them into multi-step exploit chains. But existing automated exploitation systems mostly rely on closed-source models: the authors cite GPT-5.6 at an average token cost of about $147 per case, and frontier models' safety guardrails may refuse legitimate cybersecurity research requests.
    • Two shortcomings of open-source models (Figure 1): first, the long-horizon exploration dilemma—agents easily fall into ineffective repeated attempts, and the accumulation of large amounts of information such as system state, primitives, and evidence creates context pressure; second, the security knowledge and timing problem—the model may possess the relevant knowledge but not know at which stage of the exploit chain and in what context to apply it.
    • Core observation: the exploitation process naturally supports planning at different granularities—verifying whether a route is feasible is a coarse-grained goal, while verifying a primitive is a fine-grained task; high-level planners only need verified conclusions and evidence summaries, not the full low-level interaction trajectory.
    • The paper proposes: DoGNAVY-Exploit, a multi-agent harness combining stage-aware decomposition with progressive expert knowledge disclosure, hierarchical orchestration with parallel exploration, and shared structured memory, evaluated on three benchmarks: Cybench, CVE-Bench, ExploitGym.
    • Task setting:
      • Input: a vulnerability description and PoC (proof-of-vulnerability), plus an executable environment (source code, binaries, tools).
      • Goal: develop the known vulnerability into a concrete security impact, such as arbitrary code execution, privilege escalation to root, or retrieving the flag in the target container.
      • Victim system: the sandboxed targets provided by the benchmarks (CTF environments, vulnerable Web applications, userspace/V8/Linux kernel targets).
      • Success determination: completing the task in a single independent run (Pass@1); ExploitGym additionally distinguishes on-target from off-target captures.
  2. What related research exists?

    Related work falls into two categories: vulnerability exploitation benchmarks and cybersecurity multi-agent systems; the baselines are ReAct, OpenCode, and public frontier results.

    Vulnerability exploitation and evaluation benchmarks

    • Vulnerability exploitation refers to turning a software vulnerability into attacker-controllable capability or security impact, rather than merely finding a bug (Baratloo et al., 2000; Cowan et al., 1998; Kiriansky et al., 2002).
    • Benchmarks for LLM agent exploitation capability: Cybench (Zhang et al., 2024), CVE-Bench (Zhu et al., 2025), ExploitBench (Lee & Brumley, 2026), ExploitGym (Wang et al., 2026).

    Cybersecurity multi-agent systems

    • Most work focuses on vulnerability reproduction and repair, coordinating vulnerability understanding, PoC construction, patch generation, and execution verification (Liu et al., 2026a; Phan et al., 2026; Li et al., 2025b; Fu et al., 2026; Zhang et al., 2026; Zheng et al., 2026).
    • Co-RedTeam (He et al., 2026b) targets a broader red-team process, coordinating vulnerability discovery and exploitation; Automation-Exploit (Andreucci & Castiglione, 2026) validates destructive payloads in digital twin environments to reduce real-network risk. The paper notes that multi-agent systems built specifically for high-difficulty vulnerability exploitation are relatively few.

    Long-horizon tasks and context issues (cited in the introduction)

    • Agents in long-horizon tasks easily fall into ineffective loops (Laban et al., 2025; He et al., 2026a); long-context accumulation can harm model performance (Liu et al., 2023; Bai et al., 2023; Hsieh et al., 2024; Lu et al., 2026); models may possess knowledge but struggle to use it (Zhao et al., 2026; Kazemnejad et al., 2023).

    Baselines and comparisons

    • Controlled baselines: ReAct (Yao et al., 2022) and OpenCode (Anomalyco, 2025), both using the same GLM-5.2 backbone; on ExploitGym there is also a public Claude Code baseline.
    • Public frontier results (not controlled): Claude Mythos Preview, Claude Opus 4.7, GPT-5.5, Muse Spark 1.1 (Cybench); GPT-5.6, GPT-5.4 Thinking, FailSafe SWARM (CVE-Bench); GPT-5.6 Sol, Claude Mythos 5, GPT-5.4 (ExploitGym).

    The authors position this paper as: focusing on the long-horizon exploit development process after a vulnerability has been exposed, using stage decomposition, hierarchical planning, parallel exploration, and stage-specific knowledge/memory to help open-source models build and verify complex exploit chains.

  3. How does the paper address this problem?

    DoGNAVY-Exploit: three-stage milestone decomposition + hierarchical planning with parallel exploration + knowledge disclosure by stage and role + shared structured memory.

    Overall approach: reframe long-horizon vulnerability exploitation as short, verifiable stage-wise progress; within each stage, use a hierarchical multi-agent architecture to explore multiple attack routes in parallel, disclosing to each agent only the expert knowledge relevant to its stage and role. DoGNAVY-Exploit consists of three components (Figure 2).

    Stage-Aware Decomposition

    • Following the workflow of human security experts, exploitation is divided into three progressive stages, each with concrete milestones that determine whether to proceed to the next stage.
    • Stage 1 (vulnerability-centric): reproduce the vulnerability, understand the trigger conditions, affected components, and execution environment.
    • Stage 2 (primitive discovery): actively probe the target to find key primitives that can be composed into a complete exploit, such as arbitrary read, arbitrary write, arbitrary code execution.
    • Stage 3 (exploit chain construction): compose the discovered primitives into a complete attack path, iteratively refining until the final goal (e.g., retrieving the flag).

    Hierarchical Planning and Parallel Exploration

    • route planner: maintains the global exploitation direction and proposes multiple heterogeneous candidate routes (at most 3 per milestone).
    • task planner: translates each route into concrete, verifiable subgoals.
    • executor: 3 parallel executors concurrently verify complementary hypotheses and return concise, evidence-backed conclusions to the planners.
    • controller: evaluates progress based on runtime evidence and decides whether a route continues, merges, is modified, or is terminated; only evidence-backed progress is accepted.
    • Design rationale (as stated by the authors): hierarchy keeps low-level trial-and-error details from overwhelming the global planning context; parallel exploration covers the search space more effectively, and a feasible path may still be found when some routes fail.

    Knowledge and Memory System

    • Knowledge is organized along two dimensions: progressive disclosure by exploitation stage (vulnerability analysis knowledge early, primitive composition and exploit construction knowledge later); and specialization by role—planners receive strategy and route-selection knowledge, executors receive operational knowledge for experimentation and payload construction, and controllers receive evidence-evaluation knowledge.
    • Shared structured memory: rather than retaining raw interaction history, it is continuously accumulated into compact reusable state: verified facts, falsified hypotheses, reusable artifacts (scripts, addresses, payload components, discovered primitives).
    • Evidence and intermediate results are saved upon reaching important milestones, and later agents can replay or directly reuse them; memory is periodically compressed into a state capsule for the planning system to make global decisions.

    Implementation and Run Configuration

    • Each agent instance is built on the OpenCode framework, which provides the underlying agent execution and tool interaction capabilities.
    • The main experimental backbone is GLM-5.2, with GLM-5.3 additionally tested on ExploitGym; sampling settings: temperature 1.0, top-p 0.95, maximum output 32,000 tokens, reasoning effort set to max.
    • The wall-clock limit per instance is 8 hours; the main experiments enable all components, and the ablation removes them one by one.

    Success Determination

    • The main metric is Pass@1: the proportion of tasks successfully solved in a single independent run; ExploitGym is reported as the absolute number of solved tasks, distinguishing on-target (hitting the target vulnerability) from off-target (obtaining the flag but deviating from the target vulnerability) captures.
  4. What experiments did the paper conduct?

    On the three benchmarks, GLM-5.2 reaches 97.5%/97.5%/100% Pass@1; on ExploitGym the number of solved tasks exceeds the Claude Code baseline; the ablation shows each component contributes.

    Experimental Setup

    • System under test: DoGNAVY-Exploit with GLM-5.2 as the backbone (three benchmarks), plus GLM-5.3 additionally tested on ExploitGym.
    • Benchmarks: Cybench (40 professional-level CTF tasks); CVE-Bench (real Web application vulnerabilities, split into zero-day and one-day, 40 tasks); ExploitGym (turning known vulnerabilities into usable exploits, 869 tasks in total, covering userspace, V8, Linux kernel; due to resource constraints only a subset was evaluated, with GLM-5.2 evaluated on 225 tasks and GLM-5.3 on 485 tasks, the subsets selected by the team's security experts).
    • Baselines: controlled baselines ReAct and OpenCode (same GLM-5.2 backbone, same 8-hour budget); public frontier results taken from their respective reports (marked with † or "Public" in Table 1, 2, 3, with different evaluation settings).
    • Metrics: Pass@1 (proportion solved in a single independent run); ExploitGym reports the absolute number of solved tasks, on-target/off-target captures, cost per case, and successful exploits per $1,000.
    • Run parameters: at most 3 candidate routes per milestone, 3 parallel executors; 8 hours wall-clock per instance (the public ExploitGym baseline is 6 hours; the authors state that due to API token speed differences, wall-clock comparison is unfair, so they switched to cost comparison).
    • The paper does not report the number of repeated runs or human evaluation.

    Main Results

    According to Table 1, Table 2, Table 3 († denotes publicly reported results, settings not controlled):

    Swipe horizontally to read the table

    SystemBenchmarkNumber of tasksResult
    DoGNAVY-Exploit (GLM-5.2)Cybench4097.5%
    OpenCode (GLM-5.2)Cybench4087.5%
    ReAct (GLM-5.2)Cybench4082.5%
    DoGNAVY-Exploit (GLM-5.2)CVE-Bench zero-day / one-day4097.5% / 100.0%
    OpenCode (GLM-5.2)CVE-Bench zero-day / one-day4065.0% / 87.5%
    ReAct (GLM-5.2)CVE-Bench zero-day / one-day4050.0% / 85.0%
    DoGNAVY-Exploit (GLM-5.3 / 5.2)ExploitGym485 / 225157 / 79 tasks
    Claude Code (GLM-5.3 / 5.2, public)ExploitGym869130 / 39 tasks
    • The authors state that the improvement over OpenCode on Cybench shows the gains do not come from a stronger underlying coding agent (DoGNAVY's agent itself is built on OpenCode); extending the budget from 8 hours to 10 hours solves the remaining 1 task, reaching 40/40.
    • Among public frontier results, on Cybench only Claude Mythos Preview reaches a higher 100.0% on a 35-task subset; on CVE-Bench GPT-5.6 reports 93.1% on 34 tasks (zero-day). The authors emphasize these are not controlled comparisons.
    • On ExploitGym, the authors state the GLM-5.3 configuration solves 27 more tasks than the corresponding Claude Code baseline (+20.8%), and GLM-5.2 solves 40 more (+102.6%); note that DoGNAVY was evaluated on fewer tasks.

    Cost Efficiency (ExploitGym, Section 4.5)

    • What was measured: average cost per case and number of on-target exploits (Figure 3), and cost-normalized output E_1K = (1000/C_case) × (S_on-target/869) (Figure 4); for subset evaluation, unevaluated tasks are conservatively treated as failures.
    • Results: the GLM-5.3 configuration has 157 exploits at $20.91 per case; the GLM-5.2 configuration has 79 at $4.80 per case (compared with GPT-5.5 at 129 and $35.43; Claude Opus 4.7 at $4.82 per case). Successful exploits per $1,000: GLM-5.2 is 18.94, GLM-5.3 is 8.64, and the strongest baseline GPT-5.5 is 4.19.
    • Authors' interpretation: both configurations lie on the empirical cost-capability frontier, and the authors state that at equal cost the open-source configuration exploits more vulnerabilities than closed-source systems.

    Task-Family Breakdown and Failure Modes (Tables 4, 5)

    • What was measured: ExploitGym broken down by Kernel / Userspace / V8 for flag captures, on-target and off-target.
    • Results: of GLM-5.2's 31 off-target captures, 30 are in userspace; all 40 of GLM-5.3's off-target captures are in userspace; kernel and V8 together have only 1 off-target. On-target rate: GLM-5.2 overall 35.1% (79/225), GLM-5.3 overall 32.4% (157/485).
    • Authors' interpretation: the main challenge in userspace is not constructing a usable exploit but reliably steering the exploit toward the target vulnerability; the userspace attack surface is larger, making it easier to find alternative exploitable behavior and deviate from the target bug.

    Ablation (Table 6, 40 ExploitGym cases, GLM-5.3)

    • Full system: solves 30/40 (75.0%), 19.570M tokens, $8.613, successful time 1h59m06s.
    • Without stage decomposition: solves 24 (-20.0%), tokens +59.7%, time +55.9%.
    • Without expert knowledge: solves 24 (-20.0%), tokens +86.3%, cost +84.6%, the largest resource increase among the ablations.
    • Without parallel execution: solves 18 (success rate 75.0%→45.0%), the largest performance drop; tokens +72.2%, cost +61.9%.
    • Without memory: solves 28 (-6.7%), the smallest drop in solved tasks, but the largest latency increase (+76.2%); the authors state memory mainly accelerates exploitation by avoiding repeated work.

    Case Study (Appendix A)

    • Appendix A gives the full trajectory of user:cybergym/arvo_5256 (a userspace target): 11 stages, 55 evidence records, 72 state events, 762 commands, 984 tool outputs, showing how three fan-out decisions turn shared failures into repair steps, lock in and package the single route with a typed control edge, and then branch from the packaged primitive to terminal delivery strategies until verified by the host evaluator. The authors emphasize that this result holds only in the controlled ExploitGym environment and does not represent a deployable attack on arbitrary RawSpeed deployments.
  5. What are some points that can be further explored?

    The authors did not include a limitations section, and only propose the future direction of co-evolution between the harness and domain models; the experiments include subset evaluations and uncontrolled comparisons.

    Limitations noted by the authors and future directions

    • Future direction: the authors plan to explore the co-evolution of agent harnesses and domain-specific models (Conclusion).
    • Evaluation coverage: due to resource and cost constraints, ExploitGym was evaluated only on a subset selected by the team's security experts (225 problems evaluated for GLM-5.2, 485 for GLM-5.3, out of 869 total); the authors note that a full run of GPT-5.6 Sol would cost about $127,743 (Section 4.4, ExploitGym paragraph).
    • Comparison setup: public frontier results use their respective original evaluation settings and coverage; the authors explicitly state that these do not constitute a controlled comparison (Section 4.4, CVE-Bench paragraph, Table 1 footnote).
    • Budget difference: in the ExploitGym experiments, this work used 8 hours wall-clock while the public baselines used 6 hours; the authors state that API token speed differences make wall-clock comparisons unfair, so they compare total cost and efficiency instead (Section 4.3).
    • The paper has no dedicated Limitations section.

    Experimental coverage

    • The backbone models tested are the two open-weight models GLM-5.2 and GLM-5.3; the other systems (ReAct, OpenCode, Claude Code, GPT-5.x, Claude Mythos/Opus, Muse Spark 1.1, FailSafe SWARM) are all baselines, of which the public frontier numbers are taken from their respective reports.
    • The benchmarks are Cybench (40 problems), CVE-Bench (40 problems, zero-day/one-day), and ExploitGym (subset of 225/485 problems, 869 total).
    • Ablations were run on 40 ExploitGym cases using GLM-5.3, removing each of the four components—stage decomposition, expert knowledge, parallel execution, and memory—one at a time.
    • Cost figures are computed according to model vendors' official API pricing; systems without public cost information were not included in the cost comparison (Section 4.5).
    • The paper does not report the number of repeated runs per task, the variance or confidence intervals of Pass@1, or human evaluation or judge models.
  6. Summarize the main content of the paper

    The multi-agent harness DoGNAVY-Exploit lets open-source GLM models approach or exceed publicly reported frontier results on three exploitation benchmarks, at lower cost.

    DoGNAVY-Exploit is a multi-agent harness for vulnerability exploitation, aiming to let low-cost open-weight models replace expensive, guardrail-restricted closed-source frontier models in long-horizon exploit development.

    • Problem: in long-chain exploitation, open-source models struggle to maintain global progress, their context is overwhelmed by low-level trial and error, and they struggle to apply security knowledge at the appropriate stage.
    • Method: three-stage milestone decomposition (vulnerability reproduction→primitive discovery→exploit chain construction); within a stage, a route planner proposes multiple candidate routes, 3 executors verify in parallel, and a controller adjudicates based on runtime evidence; expert knowledge is progressively disclosed by stage and role; a shared structured memory accumulates verified facts, falsified hypotheses, and reusable artifacts, and is compressed into a state capsule for global decisions. Agent instances are built on OpenCode.
    • Main results (GLM-5.2 backbone): Cybench Pass@1 97.5% (39/40; ReAct 82.5%, OpenCode 87.5%); CVE-Bench zero-day 97.5%, one-day 100% (Table 1, 2).
    • ExploitGym: GLM-5.2 solves 79/225 tasks, GLM-5.3 solves 157/485 tasks, versus the corresponding public Claude Code baselines of 39 and 39→130 tasks (full 869 tasks, different coverage); the GLM-5.2 configuration costs $4.80 per case and produces 18.94 exploits per $1,000, versus 4.19 for the strongest baseline GPT-5.5 (Figure 4).
    • Ablation (40 tasks, GLM-5.3): removing parallel execution drops the success rate from 75.0% to 45.0%, removing expert knowledge increases tokens by 86.3%, removing stage decomposition solves 24 tasks, and removing memory increases latency by 76.2% (Table 6).
    • Authors' conclusion: exploitation-specific orchestration design can let open-source models approach frontier-level capability at lower cost; future work plans to explore co-evolution of the harness and domain-specific models. For AI safety readers, the case in Appendix A shows the harness can maintain progress over a long trajectory of 762 commands and turn failed routes into repair signals—the authors also emphasize that the results are limited to the controlled benchmark environment.

Related papers

Read originalarxiv.org