HoF-Bench: Rediscovering Real CVEs Found by AI without Frontier Models

Core question: after one LLM vulnerability analyzer has found a CVE, can another (cheaper) analyzer find it again from the vulnerable-version source code alone—existing benchmarks do not directly test this.

• Scenario and motivation: the authors state that AISLE's LLM analyzer has found more than 280 CVEs in projects such as OpenSSL, curl, and GnuTLS that have undergone years of human review, fuzzing, and commercial static analysis (as of July 2026, the Hall of Fame lists 287, covering 78 projects), raising the evaluation question of "can it be found again."
• Shortcomings of existing benchmarks: the authors state that many security benchmarks test snippet classification, patch generation, exploit reproduction, or application of artificial vulnerabilities, and do not directly measure whether a scanner's report can recognize a CVE-level bug in real complex software.
• What the paper proposes: HoF-Bench v1—95 publicly disclosed AI-found CVEs, 8 repositories fixed at the vulnerable commit; the task is source-conditioned (target files are provided, testing analysis and rediscovery rather than fully autonomous whole-repository localization); it also includes a strict rediscovery protocol and a controlled study of 10 detection models.
• Task setting (not a threat model): the scanner receives the repository source code and the scope of 1–7 target files, but not the CVE number, description, fix commit, or expected vulnerability snippet; scoring requires strict vulnerability identity (same code path, root cause, attack conditions, impact), and file proximity or vulnerability category alone does not score.
• Quantitative questions: what proportion of these CVEs can non-frontier LLM analyzers rediscover, and which system choices (repeated scanning, generating context, triage depth, model combination) affect the results.

Snippet/function-level benchmarks

• Devign (2019), Big-Vul (2020), PrimeVul (2025)—focus on vulnerability detection at the function or code-change level; Juliet (2012) provides synthetic test cases; SARD (2025) covers everything from synthetic programs to production applications.
• Snyk VulnBench JS 1.0 (2026)—tests the repeatability of LLM review on 10 small JavaScript fixtures, against a SAST tool's own reference findings. The authors say such targets make evaluation cheap and checkable, but rarely require the evidence that review in production code actually has to trace (cross-file state, configuration, trust boundaries visible only at the call site).

Repository-level benchmarks

• RepoPairBench (cited in the paper as DREA, 2026)—100 Python vulnerability/fix pairs, checking whether model rationales match the documented mechanism; JITVul (2025)—associates functions with introducing and fixing commits, covering 879 CVEs.
• RealVuln (2026)—compares rule-based SAST, general-purpose LLMs and security-specific scanners on 26 Python repositories with synthetic vulnerabilities; BountyBench and SEC-bench (both 2025)—extend to lifecycle stages such as exploitation, PoC generation and patching.

Census-style work

• Berkeley Vulnerability Initiative's Agentic Vulnerability Coverage Map (Villa et al., 2026)—a daily multi-vendor census of CVEs attributed to agentic systems. The authors say the census answers "what agentic systems have found in the wild," while HoF-Bench answers "whether a given scanner can find a fixed set again," and the two are different questions.

Baselines and comparison setup: the paper does not run external tools as scored baselines; the comparison substitutes 10 detection models within the same reference scanner scaffold (Table 1), and compares conditions such as repeated passes, context on/off and triage depth.

Positioning of this paper: the authors say HoF-Bench occupies the gap between snippet classification and full vulnerability exploitation: pre-fix production snapshots, scoring only strict CVE identity, and repeated passes making reliability measurable; and the repository-level benchmarks above are all single-language, whereas HoF-Bench spans three languages, C, PHP and JavaScript.

Overall approach: turn "can the same CVE be found again" into a rerunnable benchmark, HoF-Bench, then use a deliberately minimal reference scanner (derived from AISLE's open-sourced nano-analyzer) as a fixed instrument, changing only the detection model and evaluation conditions.

Benchmark construction

• Topic source: From CVEs attributed in AISLE's public Hall of Fame, select repositories with at least 5 disclosures that can be mapped to the same vulnerability snapshot; records that cannot be linked to a repository, cannot be verified at a common commit, or require different snapshots are excluded.
• Scale and composition: 95 CVEs, 8 repositories (OpenSSL, curl, GnuTLS, Apache httpd, pearweb, FOGProject, OpenEMR, WeKan), one scan commit per repository; 31 multi-file cases; covering 39 CWEs, the most common being CWE-79 (9 cases), CWE-89 (8), CWE-863 (8), CWE-284 (7), CWE-476 (6).
• Field separation: scanner-visible (task ID, repository, commit, language, target files, scope mode) and evaluator-only (CVE ID, expected path, line range, mechanism, impact, code snippet, CWE/severity, fix metadata) are separated; the manifest was leak-tested against ground truth (Appendix A, Table 4).

Task definition

• Input: the entire repository at a fixed commit + 1–7 target files (derived from ground truth, including the main vulnerability file, which the authors call a substantial localization oracle); the median target file is 911 lines, truncated at a 300,000-character limit.
• Not provided to the scanner: CVE number, vulnerability description, fix commit, expected vulnerability snippet.
• Primary result: strict rediscovery—a finding must hit the same vulnerability code path/component, root cause, attacker-controllable condition or trust boundary, and security impact across four dimensions.

Reference scanner workflow

• Three stages: optional context construction (generating a security brief: component purpose, externally reachable entry points, trust boundaries, security-relevant data/control flow, with up to 3 narrow repository searches allowed) → detection (outputting structured candidate findings) → skeptical triage (3 sequential review rounds + 1 arbiter; the arbiter sees the finding, source code, and all reviews and makes a final valid/invalid judgment based on code evidence rather than vote counting).
• Constraints: at most 5 raw findings per pass, bounded retries for structured output; every triage round's judgment is persisted so that shallower triage strategies can be replayed.
• Localization: the authors state the scaffold is a controlled comparison tool and not a contribution claim; external tools (SAST or agentic) can also be run on the 8 snapshots and scored with the same review.

Evaluation protocol

• Run scale: each model runs 4 repeated passes over 95 CVEs, with context on/off, totaling 7,600 pass records.
• Review: a fixed automated LLM reviewer (GPT-5.6 terra, used only for review, never using frontier models in the detection stage), which sees ground truth but is blind to detector identity, context condition, and pass number; outputs four levels: match / partial / no match / insufficient evidence.
• Metrics: the primary metric is strict recall; success@k is defined as the expected number of CVEs found at least once in a random subset of k passes, averaged over all subsets to avoid pass-order dependence; pass@4 is the union of four passes. Because every task is a positive example and unmatched findings were not independently annotated, the authors emphasize this is only a recall metric and do not call unmatched findings false positives.
• Review burden estimate: for post-triage findings, semantic deduplication by "the same review-actionable issue" (deterministic similarity rules + LLM review adjudicating ambiguous pairs), with the number of deduplicated clusters divided by the number of strictly rediscovered CVEs, counted separately by model and context condition.

实验设置

• 被测模型：10 个检测骨干，5 个开放权重 MoE（GPT-OSS 20B、GPT-OSS 120B、DeepSeek V4 Flash、Qwen 3.6 35B A3B、Mistral Small 4；总参数 21B–284B、激活 3–13B）和 5 个专有 small/flash 档（GPT-5.4 nano、Gemini 2.5 Flash、GPT-5.4 mini、GPT-5.6 luna、Gemini 3.5 Flash；尺寸未披露），输入价 $0.03–$1.50/百万 token（Table 1）。
• 基准：HoF-Bench v1，95 个 CVE、8 个仓库、三种语言（C 45、PHP 26、JavaScript 24）。
• 评审：GPT-5.6 terra 作为盲评审（对检测器身份、context、pass 编号盲），四档判定；strict match 才计入主指标。
• 指标：strict recall / pass@4（95 个 CVE 上）、success@k、每个重发现 CVE 的去重发现簇数；Wilson 95% 区间仅作描述。
• 运行量：7,600 条 pass 记录（95 × 10 × 2 × 4）；triage 深度变体通过回放已存评审判定评估，不重跑。
• 重复次数：每模型每条件 4 次 pass；论文未报告评审判定的人工一致性审计。

主结果

据 Table 2（pass@4 严格重发现数，满分 95；"Best result"列为最佳百分比及 95% Wilson 区间）：

Swipe horizontally to read the table

模型	Ctx	No ctx	Best result
GPT-5.6 luna	65	65	68.4% (58.5–76.9)
DeepSeek V4 Flash	62	56	65.3% (55.3–74.1)
GPT-5.4 mini	58	57	61.1% (51.0–70.2)
Gemini 3.5 Flash	56	53	58.9% (48.9–68.3)
GPT-OSS 120B	52	48	54.7% (44.7–64.4)
Qwen 3.6 35B A3B	50	49	52.6% (42.7–62.4)
GPT-5.4 nano	46	47	49.5% (39.6–59.4)
Gemini 2.5 Flash	47	42	49.5% (39.6–59.4)
GPT-OSS 20B	40	40	42.1% (32.7–52.2)
Mistral Small 4	38	36	40.0% (30.7–50.1)

• 作者解读：6 个模型在某一四 pass 条件下达到至少 50 个 CVE；从 1 个 pass 到 4 个 pass 平均每模型多发现约 12–20 个 CVE；单次 pass 平均只恢复四 pass 所覆盖 CVE 的 68%。
• 不稳定性：两种 context 条件下，62% 的成功模型–CVE–条件单元只在 4 次相同 pass 中的 1–3 次被发现（Figure 3 给出 success@k 曲线）。

模型多样性与组合

• 测了什么：把同样 4 次 pass 分给不同模型，与单模型重复比较。
• 结果：四个领先模型各取 1 个 pass 的组合期望覆盖 68 个 CVE（Table 2 portfolio 行，对 256 种组合取平均，作者标注为探索性分析），高于最强单模型 GPT-5.6 luna 四 pass 的 65；全部 40 次 context 运行的并集达 84/95（两种条件合计 85）；12 次 pass 分给三个便宜模型以 $66 找到 70 个 CVE，对比四次 GPT-5.6 luna pass 的 65 个、$128。
• 作者解读：不同模型发现不同的 bug，盲区只部分重叠，多样性优于重复。

context 与 triage 深度的影响

• context：pass@4 下有 context 的严格重发现为 514/950 个模型–CVE 单元（54.1%），无 context 为 493/950（51.9%）；配对差 +21/950（2.2 个百分点，CVE-bootstrap 95% 区间 [−6, 47]）；被所有模型漏掉的 CVE 从 16 降到 11，但去重后发现数从 1,664 升到 2,088，平均每个重发现 CVE 的发现数从 3.4 升到 4.1。作者称大部分新增发现只增加评审工作量。
• triage 深度：从单轮评审改为三轮加 arbiter，重发现的模型–CVE 结果总数仅从 974 变到 1,007；回放中 645 个被完整协议拒绝的"模拟幸存"发现经同一盲评审补判，70 个 strict match、28 个 partial（附录 B）。作者列出两点保留：后轮评审是在看到前轮内容后生成的，且未为两轮规则模拟 arbiter，回放估计的是已记录评审的过滤行为而非完整反事实重跑。

语言与难度结构

• 按语言（Table 3，context 条件）：JavaScript 24 个 CVE 的中位模型召回 81%、最佳 100%、0 个被所有模型漏掉；PHP 26 个中位 63%、最佳 73%、2 个全漏；C 45 个中位 31%、最佳 62%、9 个全漏。有 context 时 21 个 CVE 被所有模型发现、11 个无一发现（无 context 为 22 和 16）；21 个全中发现里 18 个是 JavaScript 或 PHP，11 个全漏里 9 个是 C。
• 类内对照：37 个授权/认证类 CVE（唯一跨三种语言的大类）中，平均每个 JavaScript CVE 被 8.1 个模型发现、PHP 5.8、C 3.0；二项回归中语言解释的偏差是漏洞类别的两倍以上；源码规模独立预测难度（总体 ρ = −0.47，C 内部 −0.53）。
• 作者解读：作者认为大型、有状态的 C 基础设施代码与漏洞机制无关地更难；并明确说原因只能推测——JavaScript/PHP 的信任边界通常在目标文件内可见，而 C 案例依赖跨多次调用组装的状态，证据常在检测器视野之外。

其他消融与分析

• 可靠性模型（附录 C，Table 5）：Rasch Beta-Binomial 为优选模型；Beta-Binomial 变体比二项变体 ELPD 好 100 以上，2PL 判别参数无增益（∆ELPD = 5.6，dSE = 6.3）。
• 分析器强度排序（Figure 5）：Rasch logit 尺度上 GPT-5.6 luna（无 context）最高，Mistral Small（无 context）最低，区间为 95% 可信区间。
• 成本–重发现（Figure 6）：按每基准 CVE 的归属 API 成本（对数轴）对 pass@4 严格重发现作图，作者称 context 通常增加的成本超过其带来的召回，且注明这是实验性记账视角而非生产定价主张。
• 失败案例：作者称共同漏掉的包括 curl 的证书验证状态 bug，全中发现的包括一个 OpenEMR SQL 注入。

Limitations and future directions noted by the authors

• Source and task condition (Discussion): HoF-Bench is AISLE-conditioned and source-conditioned, and the 8 repositories do not represent all software; it samples one vendor's slice of the agentic-CVE population from the census by Villa et al., and the authors suggest readers consult that census to judge representativeness.
• Recall only (Discussion): every task is a positive example and provides target files derived from ground truth; the study estimates recall under the target condition, not whole-repository localization, precision or false-positive rate; unmatched findings are unverified, not false positives.
• Single scaffold (Discussion): the 10 detection backbones share the same scaffold, prompt family, triage design and review, so results should be viewed as 10 configurations of the same analyzer, not a comparison of 10 independent products.
• Contamination and review reliability (Discussion): the CVEs are public, so training-data contamination cannot be ruled out; review and semantic deduplication are automated and not human-audited; the review model is from the same vendor family as several detectors, and blind review cannot rule out same-family scoring preference.
• Cross-task semantic assignment (§4.3): the current implementation does not do cross-task semantic assignment, which the authors list as a limitation.
• Future directions (Discussion): the authors consider the benchmark suitable for future fine-tuning experiments (the raw trajectories contain prompts, responses, findings and review labels), while noting that fine-tuning may reduce variance and schema failures, and may also cause models to overfit prompt style or review preferences.

Scope of experimental coverage

• The detection models tested are 10 (5 open-weight MoE, 5 proprietary flash tier), and the review model is GPT-5.6 terra; frontier models are not used in the detection stage.
• The benchmark covers 8 repositories, 95 CVEs, and three languages C/PHP/JavaScript; 4 passes per model per condition, 7,600 pass records in total.
• Pipeline-variable evaluations include context on/off and triage depth (single round vs three rounds plus arbiter, evaluated via replay); the model portfolio analysis is exploratory.
• The paper does not report agreement between review judgments and human review, nor sensitivity tests of the review prompt.
• The paper discusses using traditional SAST and agentic tools as external tools feeding the same review, but the experiments in this paper did not run such external tools.

A benchmark paper: the authors build HoF-Bench v1 from CVEs attributed in the public AISLE Hall of Fame, answering "can another analyzer without a frontier model find again, from the vulnerable version's source code, a real CVE that one LLM analyzer had found."

• Problem: existing vulnerability benchmarks mostly test snippet classification, patches or synthetic vulnerabilities, and do not directly measure a scanner's ability to recognize CVE-level vulnerabilities in real repository snapshots.
• Method: 95 CVEs fixed at the vulnerable commits of 8 repositories (OpenSSL, curl, GnuTLS, Apache httpd, pearweb, FOGProject, OpenEMR, WeKan); the scanner sees only source code and 1–7 target files, not the CVE ID, description or fix; GPT-5.6 terra acts as blind reviewer, judging strict match on four dimensions: code path, root cause, attack conditions, impact; 10 non-frontier detection models (5 open-weight MoE + 5 proprietary flash tier) each run 4 passes × two context settings under a fixed scaffold, 7,600 records in total.
• Main results: the highest pass@4 strict rediscovery is GPT-5.6 luna's 65/95 (same with/without context, Table 2); 6 models reach at least 50 under some condition; going from 1 to 4 passes finds on average about 12–20 more CVEs; the expected coverage of a portfolio of one pass each from four models is 68, exceeding the strongest single model's 65; context only raises rediscovery among the 950 units from 493 to 514, while deduplicated findings rise from 1,664 to 2,088; triage going from single round to three rounds plus arbiter changes the rediscovery result only from 974 to 1,007.
• Difficulty structure: with context, 21 CVEs are found by all models and 11 by none; of the 11 missed by all, 9 are C, and median recall for C is 31% (Table 3); the bias explained by language in the regression is more than twice that of vulnerability category.
• Authors' conclusion: the authors consider vulnerability discovery a systems problem rather than a single model call, and that scanner evaluations should report cumulative results over multiple runs and the volume of findings submitted for review; the remaining headroom is concentrated in large, stateful C infrastructure code, and the authors say the reason can only be speculated.
• Boundaries: the authors emphasize this is a vendor-built benchmark (all authors affiliated with AISLE), it tests only recall under the target condition, the review was not human-audited, and the published data, manifest and raw run records are provided for independent audit.