VulnGym: A Benchmark for Repository-Level Vulnerability Detection with Coding Agents

How to evaluate and diagnose where programming agents fail at vulnerability detection in real repositories.

• Scenario: The authors state that programming agents extend detection from preselected snippets to the full repository, requiring them to search files and trace dependencies on their own. As an example, a stored XSS in Open WebUI passes through multiple components before reaching an unsafe innerHTML assignment.
• Existing shortcomings: The authors state that existing benchmarks mainly classify preselected code and do not require agents to autonomously explore and locate the relevant code. Without fine-grained traces, a correct label can still be matched to the wrong vulnerability or irrelevant evidence, and the evaluation cannot reveal the failing step.
• Question to answer: How to evaluate the detection process in real repositories and diagnose where failure occurs.
• Proposed benchmark: VulnGym aligns reviewed GitHub advisories to vulnerable-version repositories, containing 184 advisories, 408 entries, and 23 repositories. Each entry is annotated with row-level Entry Point (EP), Critical Operation (CO), and vulnerability Trace, and defines an end-to-end task and three oracle subtasks.

The authors discuss vulnerability detection benchmarks in terms of evaluation scope and evidence granularity, and compare them in Table 1.

Function-level classification benchmarks

• Early data: Lin et al. (2019) framed detection as function-level binary classification. BigVul (Fan et al., 2020) and Devign (Zhou et al., 2019) provide large-scale function annotations; the paper states they have label noise and duplicates, citing Croft et al. (2021), Tamberg and Bahsi (2025), and Liu et al. (2024).
• Later cleaning: PrimeVul (Ding et al., 2024) uses automated policies to improve annotation accuracy and removes exact duplicates with MD5. SecVulEval (Ahmed et al., 2026) refines annotations to statement level and extracts multiple code contexts. The paper states these works still lack interprocedural context and the complete execution chain from entry point to key operation.

Repository scope but still classification-oriented

• VulEval (Wen et al., 2026): Supplements target functions with repository-level dependency context and performs classification in a broader context; it also localizes vulnerable files and lines from patches, providing cross-function call dependencies.
• CWE-Bench-Java (Li, Dutta, and Naik, 2025): 120 manually verified vulnerabilities from compilable Java projects, each associated with a fixing commit.
• The paper's distinction: The authors state that both primarily evaluate classification accuracy rather than autonomous repository exploration, and neither provides vulnerability execution chains.

Agent benchmarks and comparison table

• JITVul (Yildiz et al., 2025): The authors state this is the first repository-level vulnerability detection benchmark for ReAct agents. It frames detection as pairwise classification of vulnerability-introducing and fixing commits; the agent uses tool calls to retrieve interprocedural context and uses pAcc to avoid inflating F1 by overpredicting vulnerability classes. The paper notes it still targets preselected functions and lacks chain-level annotations to evaluate whether reasoning is complete.
• Table 1: The five dimensions are evaluation scope (Level), file and line localization (Location), natural language defect cause (Cause), cross-file line-level execution chain (Chain), and manual verification (Val.). The authors state that no previous benchmark covers all five simultaneously, especially lacking Chain and Cause; in the table, VulnGym has all five.
• Experimental comparison: The paper does not run these benchmarks as detector baselines. The experimental comparison is among LLM and scaffolding combinations on the same task, with data from VulnGym.

The authors position this work as: using repository-level line annotations, natural language defect causes, and manually verified cross-module Traces to evaluate autonomous exploration and complete evidence chain construction.

VulnGym first aligns advisories to vulnerable version snapshots and generates reachable entries, then undergoes manual verification and revision, and then uses EP, CO, and Trace to define four tasks.

Data Construction

Figure 1 shows three phases. The authors state that time filtering is intended to reduce data leakage.

• Phase I: Starting from 3,632 reviewed high-severity GitHub Advisories from January 2025 to April 2026, covering 1,579 repositories. Only repositories from November 2025 to April 2026 with more than 10,000 stars are kept, yielding 184 advisories. Advisories where the repository cannot be located or the vulnerable snapshot cannot be reliably determined are excluded.
• Reachable entries: On the vulnerable snapshot, data flow and control flow analysis is used to find externally reachable EPs, security-relevant COs, and Traces connecting them. Each independent EP–CO path is stored as a separate record.
• Definitions: EP is an externally reachable location that can trigger the vulnerability, such as an API handler or event callback. CO is a security-critical location where the vulnerability takes effect, such as an unsafe operation or a missing security check. Trace is an ordered execution path, with each step corresponding to a file location and code snippet.

Verification and Revision

• Phase II: First, locations are compared line-by-line with the source code, classified as exact match, line offset, content mismatch, or missing file. Two reviewers independently label each as Correct, Repair, Incorrect, or Unsolved based on the vulnerability description, checking external reachability, security relevance of CO, and Trace order. Disagreements or Unsolved cases are adjudicated by a third senior reviewer. Cohen's κ = 0.87, which the authors state indicates high agreement.
• Phase III: Based on labels and line-level corrections, source files and lines are relocated and fields updated. Those requiring no modification are marked as Verified; corrections with source evidence are marked as Update and return to Phase II. Entries where the file or line number cannot be reliably identified are excluded.
• Scale: The final set includes 408 reachable records from 184 advisories across 23 repositories. Figure 2 shows the proportion of entries per repository and the distribution of vulnerability categories; the text does not provide global proportions for each category. The authors state that the benchmark includes natural language vulnerability causes, but the text does not separately describe the annotation steps for Cause.

Four Tasks

Figure 3. A single advisory can contain multiple independent reachable paths, and entries are written as EP → Trace → CO.

• T1: Input is the vulnerable repository along with the advisory title and description. Output is one or more EPs, COs, and their Traces, simultaneously testing exploration, localization, reachability, and evidence construction.
• T2: The input listed in the paper is an oracle CO, requiring finding externally reachable EPs that can lead to it.
• T3: The input listed in the paper is an oracle EP, requiring locating all COs reachable from it.
• T4: The input listed in the paper is an oracle EP–CO pair, requiring constructing an ordered evidence Trace between them. Figure 3 does not restate whether T2–T4 also provide the advisory text.

Matching and Metrics

Location matching requires that the normalized repository-relative path is the same and the line number difference does not exceed k = 5.

In T1, a prediction must match both the reference EP and CO for an entry to be considered hit. ER = N_entry^hit / N_entry, i.e., the proportion of reference entries that are hit. An advisory is considered hit if at least one entry is hit, corresponding to AR. ES(P, G) = 1 − D_Lev(P, G) / max(n, m), using the Levenshtein distance between the predicted Trace and the reference Trace to measure similarity in location and call order, computed only for hit entries.

T2 and T3 share FR (proportion of correct files), LR (proportion of line-level hits within k lines), and FP (proportion of predictions that hit the reference within k lines). T4 reuses ES and reports TNR, TNP: one-to-one hits of intermediate nodes divided by the number of reference intermediate nodes and the number of predicted intermediate nodes, respectively.

End-to-end AR for nine frontier combinations on Hard, with the highest in Table 2 being 22.58%, from OpenHands with DeepSeek-V4-Flash.

Experimental Setup

• Models: The paper selects models with context of at least 256K tokens. Scale experiments use Qwen3.5-2B, 4B, 9B, 27B. Frontier models are DeepSeek-V4-Flash, MiniMax-M3, GLM-5.2.
• Scaffolds: Claude Code, OpenHands, MiniSWE. Agents can only call read-only commands; the paper lists cat, find, grep, head, ls, nl, pwd, rg, sed, tail, wc.
• Difficulty: Divided by Trace length into Easy (≤7), Medium (=8), Hard (≥9), and makes Medium and Hard similar in size. Advisory is 110/31/31, entries are 278/56/46.
• Task assignment: The four Qwen3.5 sizes run T1 on three difficulties with Claude Code. Nine frontier combinations run T1 (Table 2) and T2–T4 (Table 3) on Hard.
• Evaluation: Against human-verified reference entries, k = 5. The paper does not use LLM judges, nor does it report repetition count, sampling temperature, or turn limits.

Main Results

The table below shows T1 on Hard, numbers from Table 2. Due to row limits, omitted are Claude Code+MiniMax-M3 (12.90 / 8.70 / 6.25) and MiniSWE+MiniMax-M3 (16.13 / 10.87 / 6.95). The table already includes OpenHands+GLM-5.2 with the lowest AR.

Swipe horizontally to read the table

Scaffold	Model	AR(%)	ER(%)	ES(%)
Claude Code	DeepSeek-V4-Flash	9.68	6.52	3.37
Claude Code	GLM-5.2	12.90	8.70	6.94
OpenHands	DeepSeek-V4-Flash	22.58	15.22	11.63
OpenHands	MiniMax-M3	16.13	10.87	7.93
OpenHands	GLM-5.2	6.45	4.35	2.15
MiniSWE	DeepSeek-V4-Flash	16.13	10.87	7.67
MiniSWE	GLM-5.2	16.13	10.87	8.65

• Comparison with scale models: The authors state that frontier combinations outperform Qwen3.5-27B with Claude Code on Hard, which has 12.90% AR, 8.70% ER, 4.55% ES. The authors still state that even the best combination detects fewer than a quarter of Hard entries.
• Opposite combination directions: The authors state that DeepSeek-V4-Flash is best on OpenHands and drops when switched to Claude Code; GLM-5.2 rises from OpenHands to Claude Code. The numbers in the table are exactly these two pairs of comparisons.
• MiniSWE: All three LLMs have AR of 16.13% and ER of 10.87%. The omitted row's ES is 6.95%; the authors give an ES range of 6.95%–8.65%. The authors state that this scaffold is relatively stable, and combination compatibility is key to end-to-end results.

Scale and Difficulty

• What was tested: Four Qwen3.5 sizes, scaffold fixed to Claude Code, metrics are T1's AR, ER, ES (Figure 4 and main text).
• Results: The authors state that 2B and 4B are nonzero only on Easy; 9B extends nonzero results to Medium; among these four sizes, 27B leads on all difficulties and metrics, and only it is nonzero on Hard. Qwen3.5-9B's AR, ER, ES drop from 6.36%, 4.32%, 2.52% on Easy to 3.23%, 1.79%, 1.67% on Medium, and are all 0 on Hard. 2B and 4B are all 0 on Medium and Hard. 27B on Hard is 12.90%, 8.70%, 4.55%.
• Authors' interpretation: The authors state that 4B is higher than 9B on all three metrics on Easy, and believe scale gains may vary with difficulty. The paper does not give specific percentages for 2B, 4B, and 27B on Easy and Medium.

Oracle Conditional Capabilities

• What was tested: T2–T4 for nine combinations on Hard (Table 3). T2 gives oracle CO, T3 gives oracle EP, T4 gives oracle EP–CO.
• Results: The authors state that except for OpenHands+GLM-5.2, CO localization FR is higher than EP identification. For that exception, the LR for EP and CO are 9.30% and 2.33%; according to Table 3, their FR are 18.60% and 6.98%. The authors give the highest CO localization FR and LR as 51.16% and 30.23%, corresponding to MiniSWE+DeepSeek-V4-Flash and Claude Code+GLM-5.2, respectively. The highest EP identification FR and LR are 30.23% and 18.60%, with the highest FP being 11.11% for EP and 5.22% for CO. TNR ranges from 30.60% to 53.96%, TNP from 14.61% to 26.71%, and the highest ES is 36.37% (MiniSWE+GLM-5.2).
• Authors' interpretation: The authors believe EP identification is generally harder than CO localization, but EP has higher FP, making predictions more selective with lower coverage. The authors state that after removing endpoint uncertainty, the evidence Trace remains incomplete and noisy: it recovers some reference intermediate nodes while also including many unmatched nodes.

Failure Composition

• What was tested: Relation-level failures on T1 on Hard, totaling 347 cases (Figure 5 and main text).
• Results: Both endpoint files missed in 93 cases (26.8%); at least one endpoint reached the correct file but none located the correct line in 103 cases (29.7%). The authors state these two categories total 56.5%. Only CO hit, EP missed in 80 cases (23.1%); only EP hit, CO missed in 32 cases (9.2%). No usable findings in 39 cases (11.2%), of which OpenHands+GLM-5.2 accounts for 18.
• Authors' interpretation: The authors believe that in T1 outputs, EP identification is more often unresolved. The authors state these statistics show where predictions fail, but are only a coarse-grained view.

Other Ablations and Analyses

• Efficiency (Figure 7, Hard, T1): The authors state the token–AR Pareto frontier extends from Claude Code+GLM-5.2 to OpenHands+DeepSeek-V4-Flash; the latter has the highest AR and is among the cheaper configurations, while OpenHands+GLM-5.2 has the highest cost but the lowest AR. The authors believe higher consumption does not necessarily lead to more effective vulnerability discovery. The main text does not give token counts or dollar amounts.
• Trajectories (Figure 6, DeepSeek-V4-Flash, Hard, T1): Actions are divided into Explore (find, ls), Search (rg, grep), Read (cat, head, sed, tail), Other, Failed. The authors state Read has the largest share, Explore is concentrated in the initial rounds, and Search runs throughout; Claude Code has fewer Failed and Other, MiniSWE has more Other, and OpenHands has more Failed concentrated early. The main text does not give counts for each action.
• Non-monotonic scale: On Easy, 4B's three metrics are higher than 9B; the paper does not give specific percentages for this comparison.

The paper does not specifically discuss limitations.

Limitations and future directions noted by the authors

The paper does not specifically discuss limitations. The attached text has no Limitations, Future Work, or limitation notes in the appendix; the Discussion and Conclusion also do not frame anything as a limitation or future work.

Experimental coverage

• The final data consists of 23 open-source repositories, 184 advisories, and 408 reachable entries; Phase II was independently annotated by two reviewers, with disagreements or Unsolved cases adjudicated by a third, Cohen's κ = 0.87.
• The tested models are Qwen3.5-2B, Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-27B, DeepSeek-V4-Flash, MiniMax-M3, and GLM-5.2, 7 in total; the scaffolds are Claude Code, OpenHands, and MiniSWE.
• Scale comparisons ran T1 with Claude Code on Easy, Medium, and Hard. T1 and T2–T4 values for nine frontier combinations are reported on Hard; this subset comprises 31 advisories and 46 entries (Table 2, Table 3).
• Agents may only call the read-only commands listed in the paper. Position matching uses k = 5. T1 reports AR, ER, ES; T2/T3 report FR, LR, FP; T4 reports ES, TNR, TNP.
• The paper does not report the number of repetitions, sampling temperature, or interaction round limits; the per-run token and cost in Figure 7 have no specific values in the main text.

VulnGym uses line-level evidence to evaluate whether programming agents can detect vulnerabilities in vulnerable-version repositories and construct code locations and execution traces that support the conclusion.

• Problem: Function-level classification does not require the agent to find the relevant code itself, and it is impossible to tell whether failure occurs in localization or evidence construction.
• Approach: Aligned reviewed high-risk advisories from November 2025 to April 2026 with more than 10,000 stars to vulnerable snapshots; after two-person verification and revision, 184 advisories, 408 entries, and 23 repositories were obtained. T1 gives only the title, description, and repository; T2–T4 give oracle CO, oracle EP, or oracle EP–CO, respectively. A location match requires the same path and a line distance no greater than 5.
• End-to-end: On Hard (Trace length ≥9; 31 advisories, 46 entries), OpenHands with DeepSeek-V4-Flash achieves AR, ER, and ES of 22.58%, 15.22%, and 11.63% (Table 2). Qwen3.5-27B with Claude Code achieves 12.90%, 8.70%, and 4.55%. The same series' 2B and 4B models score 0 on all three metrics on Medium and Hard, and the 9B scores 0 on Hard; on Easy, the 4B's three metrics are higher than the 9B's.
• Diagnosis: Among 347 Hard failures, the authors state that 56.5% are file misses, or hits on the file but misses on the line. Under oracle conditions, EP identification achieves the highest FR and LR of 30.23% and 18.60%, and CO localization achieves 51.16% and 30.23%; T4 has the highest ES of 36.37%, with TNR ranging from 30.60% to 53.96% and TNP from 14.61% to 26.71% (Table 3).
• Authors' conclusion: The authors state that challenging vulnerabilities remain largely unresolved, and results vary with LLM and scaffolding combinations; higher token or cost does not necessarily correspond to higher AR. The authors state that the benchmark provides a foundation for advancing repository-level vulnerability detection.