Discovery or Memorization? A Faithful Benchmark for Real-World Business Logic Vulnerability Discovery

Are current vulnerability-finding agents actually discovering complex vulnerabilities, or are they memorizing publicly disclosed vulnerabilities.

• Scenario: The authors argue that cybersecurity has become a prominent testing ground for large language models and autonomous agents, motivated in part by the need for better security in AI-generated software. Challenges like CTF competitions saturate as models become stronger, making benchmark scores harder to interpret.
• Existing shortcomings: The authors state that synthetic vulnerability benchmarks are reproducible and have objective success criteria, but may not test real vulnerabilities; benchmarks based on advisory databases like CVE, GHSA, and Snyk are more realistic, but public software identity clues allow models to recognize previously seen vulnerabilities, leaking the answer.
• Core observation: Business logic vulnerabilities are defined by application-specific policies and semantic relationships, unlike low-level memory or threading issues or generic injections that manifest as syntactic patterns or runtime signals (Figure 1). Legitimate operations and violations often return the same successful response; only the relationships among subject, object, and effect can distinguish them.
• This paper proposes: The authors propose a cross-language benchmark AnonVul, containing 132 executable application-level business logic vulnerabilities across nine languages and 36 CWEs. A semi-automated pipeline performs semantics-preserving anonymization, injects oracles from expert specifications, and differentially validates PoCs.
• Access settings: blackbox provides only the running application and task-specific attacker permissions; greybox additionally provides an isolated read-only source snapshot and source inspection tools. Both share scenarios, authorization, and runtime constraints. Target vulnerabilities, reference PoCs, patch instances, and oracle internals are hidden from agents. Success is determined by a hidden relational oracle.

Cybersecurity agent benchmarks

• CTF and interactive tasks: InterCode-CTF (Yang et al., 2023), NYU CTF Bench (Shao et al., 2024), Cybench (Zhang et al., 2025b) evaluate capture-the-flag or cybersecurity capability; RedCode-Exec (Guo et al., 2024) evaluates risky execution and generation by code agents.
• Known vulnerabilities and open discovery: CVE-Bench (Zhu et al., 2025), BountyBench (Zhang et al., 2025a), SEC-Bench (Lee et al., 2025), CyberGym (Wang et al., 2026d), CyberGym-E2E (Shi et al., 2026), HoF-Bench (Simecek et al., 2026) target CVEs, bounties, or end-to-end discovery. Table 3 also lists AutoAdvExBench, SeCodePLT, PatchAgent, and gives sizes, e.g., Cybench 40 CTF, CVE-Bench 40 CVE, CyberGym 1,507, CyberGym-E2E 920, AnonVul 132.
• Gaps named in the paper: the authors say business logic vulnerabilities remain rare in these benchmarks. The Detect tasks of CVE-Bench and BountyBench hide the target vulnerability from the agent; CyberGym-E2E hides ground truth in the end-to-end setting; HoF-Bench hides CVE identifiers from the detector but not from the judge that scores. The authors say none of these four simultaneously has business-logic-centric corpora, matched blackbox/greybox, and paired anonymized and identity-visible evaluation.

Business logic vulnerabilities

• Transactions and workflows: Felmetsger et al. (2010), Sun et al. (2014), Pellegrino and Balzarotti (2014) etc. do Web application logic vulnerability detection; the paper also lists e-commerce, design artifacts, smart contracts, and annotation-based sanitization (Chen et al., 2019; Deepa et al., 2018; Cheh et al., 2022; Sun et al., 2024; Wang et al., 2026b).
• Authorization and permission constraints: Huang et al. (2024b) detect broken object-level authorization in database applications; the paper groups Shi et al. (2025), Li et al. (2026), Cao et al. (2024), Luo et al. (2025), Zhang et al. (2026; 2025c) into permission, privilege escalation, and OAuth-type constraints.
• Invariants and relational abstractions: Wang et al. (2024), Liu et al. (2025), Atlidakis et al. (2019), Li et al. (2021) involve API behavior, contract invariants, and stateful REST exploration; Huang et al. (2021), Li et al. (2023; 2024) provide relational constraints and neuro-symbolic reasoning abstractions. The authors say such defects require verifying security effects across users, objects, and states.

Vulnerability discovery methods

• Analysis and learning-based detection: Dinella et al. (2020), Zhou et al. (2019), Li et al. (2025), Wang et al. (2026a).
• Dynamic exploration, data, and specifications: the paper lists fuzzing (Fioraldi et al., 2020 etc.), vulnerability datasets and empirical evaluations (Fan et al., 2020; Chen et al., 2023; Ni et al., 2024; Ding et al., 2025; Khare et al., 2025), and specification formalization (Zheng et al., 2025; Feng et al., 2026).
• Deployment and conversational measurement: Kang et al. (2022), Yang et al. (2026c) measure real Web applications; Yang et al. (2026b;d) examine conversational context and cross-component security failures. The authors say AnonVul instead evaluates business logic discovery in executable applications with matched access and relational oracles.

Comparison setting: the experiments do not add traditional detector baselines, but compare six agent systems on AnonVul; Scaling adds two more Codex systems. Table 3 compares benchmark design, not experimental scores.

The authors position the paper as: combining matched blackbox/greybox, passive runtime oracles, and paired anonymized and identity-visible evaluation to examine how source access and identifiable software knowledge affect discovery.

The authors build AnonVul with a semi-automated pipeline: large-scale candidate classification uses Claude Haiku 4.5, vulnerability analysis and environment construction use Claude Sonnet 4.6, with human review throughout. Reproduction is performed by a reproduction agent; the paper does not name a model separately there.

Selection, specification, and environments

• Inclusion: vulnerabilities disclosed in public open-source software from 2025-01-01 to 2026-03-01, only business-logic vulnerabilities whose security consequences depend on application-specific semantics (the authors cite OWASP Foundation, 2021). ANOTA's CWE classes only guide retrieval and do not determine inclusion. Cross-layer cases can be included when application behavior makes request context security-relevant.
• Specification: analyze the advisory, patch, and affected implementation, recording affected versions, subject and permissions, protected object, operation or workflow, preconditions and state, forbidden effect, patch behavior, and deployment requirements. The same specification is used to build the environment, verify reproduction, define the oracle, and check anonymization. Humans cross-check against original materials.
• Four packaging types: standalone applications are Direct; those needing databases, caches, or identity services are Compose; those requiring rebuilding host contexts such as CMS, CI, or Kubernetes are Platform; libraries and frameworks are embedded as exact vulnerable dependencies in a scaffold host, i.e., Carrier, with security-critical operations still implemented by the original dependency. The 132 validated instances are Direct 35, Compose 62, Platform 25, Carrier 10 (Appendix A).
• Exclusions and boundaries: of 154 eligible candidates, 7 are excluded for unavailable licenses or proprietary features, 15 for insufficient public upstream evidence. Fixes may change dependencies, configuration, fixtures, startup, and wiring, but may not change vulnerable logic, expand attacker privileges, replace upstream fixes, or add endpoints specifically for exploitation.

Reference PoCs and semantics-preserving anonymization

1. Freeze witness: after successful reproduction, freeze as a reference PoC and replay from a fresh state. The vulnerable version must exhibit the specified violation, and the patched version must not. Humans check the advisory, patch, and implementation. Further replays perturb synthetic identities and objects while preserving roles, permissions, and workflows.
2. Two types of information: preserve roles, permissions, object relationships, and workflows; remove or replace atomic clues such as product names, versions, and advisory IDs, as well as composite clues such as routes, headers, and ports that together identify the software (Figure 4).
3. Two rounds of checking: static analysis scans runtime names exposed to the agent in the greybox source directories and configuration; dynamic checks use synthetic users walking normal workflows rather than the target exploit, inspecting pages, HTTP headers and responses, decoded token claims, logs, and container metadata. Clues are consistently replaced across source and runtime artifacts and re-checked until passing. Finally, the reference PoC is replayed to confirm the vulnerable instance still has the forbidden effect and the patched instance still blocks it.
4. Out-of-sandbox materials: advisories, packages, versions, images, source records, patches, and acceptance records remain outside the experiment sandbox. Greybox exposes only the declared read-only application source directory, excluding build records, evaluator code, patch data, reference PoCs, instrumentation, and hidden fixtures, and blocks snapshots from touching runtime services, the target filesystem, databases, secrets, or container APIs.

Passive relational runtime oracle

• Observation points: determine the subject, protected object, authorization or workflow condition, and resulting effect per the specification. Passive instrumentation is placed where the effect is observable: state diffs, protected responses or content, callbacks or side effects. Primary witnesses are state diffs 59, protected responses or content 59, callbacks or other side effects 14; multi-step witnesses can combine evidence (Appendix B).
• Hidden determination: the verifier only reads recorded evidence and checks the security state or effect specified by the witness. Instrumentation is hidden from the agent. Authentication uses unattacked, legitimate, wrong-subject, wrong-object, and incomplete-workflow controls where applicable, because events alone are insufficient to establish subject, object, permission, and workflow binding.
• Admission: from a fresh state, check service health, normal workflows, and known residual identity clues. The oracle must be negative before exploitation; the reference PoC must trigger the oracle; the matching patched instance must block the violation and the oracle must remain negative. Also check that the oracle does not send requests to the application. Final review confirms that the specification, environment, PoC, oracle, and transformation records point to the same violation.

Evaluation protocol

• Matched settings: blackbox provides the running application; greybox adds an immutable read-only source snapshot and source inspection tools. Greybox does not receive blackbox trajectories, drafts, tool outputs, candidates, trigger results, or human analysis. Runtime isolation denies attackers use of the Docker socket, host mounts, the target filesystem, evaluator files, and unrestricted external network.
• Prompts and hidden items: the system prompt specifies the logical vulnerability discovery goal, authorization scope, and tool constraints; the user prompt gives the target address, authorized identity, task, and resources such as synthetic credentials. The oracle internals, evaluator state, target vulnerability, reference PoC, and patched instance remain hidden.
• Feedback and submission: when oracle feedback is enabled, positive confirmation only indicates the target effect was observed and does not reveal conditions. The submission tool accepts an optional executable PoC, freezes its hash, and replays it on a fresh state with the same hidden oracle, without rewriting or calling the model again.
• Main experiment budget: one independent attempt per system, per task, per setting, starting from a fresh state, limited to 20 minutes and $5. Official attempts are recorded once and not replaced by reruns.

Experimental setup

• Systems under test: six in the main experiments: Claude Opus 4.8, Claude Sonnet 4.6 (Claude Code), GPT-5.4 (Codex), GLM 5.2, Kimi K2.6, MiniMax M2.5 (mini-swe-agent). Scaling additionally adds GPT-5.6 Sol and GPT-6 Astra (both Codex). The paper states these systems come from proprietary and open-weight model families but does not specify which belong to which category.
• Benchmark: AnonVul, 132 vulnerabilities from 154 candidates, nine language or runtime ecosystems, 36 CWEs, with one vulnerability possibly having multiple labels. Figure 5: privilege escalation 40, IDOR 33, authorization 32, business process 17, semantic context 6, authentication bypass 4; languages Go 39, PHP 27, Java 21, JS/TS 19, Python 16, Other (4) 10. Reasoning structures are human-annotated.
• Three experiments: Main covers all 132 tasks in blackbox and greybox. Scaling reruns the six main systems on the 10 vulnerabilities unsolved in both settings. Visible compares anonymized versus identity-restored conditions on 20 vulnerabilities, selected to balance Main successes and failures; Visible restores product, package, and version identifiers without revealing the target advisory or exploit.
• Budget: main experiments 1 run per cell, 20 minutes, $5. Scaling 60 minutes, $50 per cell. 1,584 main experiment cells correspond to at most 31,680 agent-minutes, i.e., 528 agent-hours, and at most $7,920.
• Metrics and determination: verified discovery, i.e., the hidden-relation oracle observes the specified effect. Table 1's Overall is the discovery rate across both settings; Task coverage is the number of vulnerabilities solved at least once. Success is not scored by an LLM judge. Replay results of submitted PoCs must match the final runtime results.
• Sample size: main experiments 132×2×6=1,584 attempts, consistent with the 1,584 stated in the paper. One run per cell, no rerun replacement.

Main results

Swipe horizontally to read the table

System (framework)	Blackbox	Greybox	Overall
Claude Opus 4.8 (Claude Code)	19.70%	26.52%	23.11%
Claude Sonnet 4.6 (Claude Code)	15.91%	21.21%	18.56%
GPT-5.4 (Codex)	11.36%	25.00%	18.18%
GLM 5.2 (mini-swe-agent)	13.64%	21.21%	17.42%
Kimi K2.6 (mini-swe-agent)	4.55%	6.82%	5.68%
MiniMax M2.5 (mini-swe-agent)	1.52%	7.58%	4.55%
Six systems combined	11.11%	18.06%	14.58%

Per Table 1. Task coverage is blackbox 41/132, greybox 58/132, combined 63/132.

• The authors state source access improves every system's performance, but all systems and both settings combined cover only 63 vulnerabilities, leaving 69 unsolved.
• The authors state performance differs across reasoning structures and vulnerability types, with no successes in the semantic context category (Figure 6). In that figure the language axis is 0–75% and the reasoning and logic axis is 0–45%; Rust, .NET, and Elixir/Erlang each contain two vulnerabilities. The caption does not give endpoint readings for each axis.
• The authors state successes concentrate in a relatively small shared subset. Appendix D states: among solved vulnerabilities, 36 have successes in both settings, 5 only in blackbox, 22 only in greybox; 43/63 (68.3%) are solved by at least two systems, and only 20 rely on a single system. The vulnerabilities solved by all six systems are Open WebUI's access control flaw (11 of 12 cells passing) and Keycloak's privilege escalation flaw.

Budget scaling

• What was tested: 10 vulnerabilities unsolved in both settings in the main experiments, 160 scheduled cells, budget 60 minutes and $50, with GPT-5.6 Sol and GPT-6 Astra added.
• Results: 8 cells succeed, distributed across Wagtail, ZITADEL, XWiki. In Table 2, GPT-5.6 Sol is 1/10 and 3/10, blocked 1/20, Rate 20%; Opus 4.8 is 3/10 and 0/10, Rate 15%; GPT-5.4 is 1/10 and 0/10, Rate 5%; Sonnet 4.6, GLM 5.2, Kimi K2.6, and MiniMax M2.5 are all 0/20. GPT-6 Astra's BB/GB/Rate are "–", Blocked 20/20. Provider defenses block 21 cells. Of the eight successes, five still fall within the main experiments' 20 minutes and $5, three exceed both; 7 of the 10 tasks remain unsolved. The final matrix is 8 successes, 131 failures, 21 blocked, with blocks reported separately from the 139 executed attempts.
• Authors' interpretation: the authors state additional budget can sustain stalled search, but exploration itself does not solve most failures. GPT-5.6 Sol's blackbox on Grav uses the full 60 minutes, spending $27.62, without finding the target vulnerability. On n8n, that system's blackbox uses 238 rounds and $13.18, reaching the time limit without submitting a candidate.

Identity clues

• What was tested: 20 vulnerabilities, six systems, two access modes, 240 cells each for anonymized and Visible. Visible does not reveal the target advisory or exploit.
• Results: discovery rate rises from 21.25% (51/240) to 40.83% (98/240). The introduction states an increase of 19.58 percentage points. 64 cells change from failure to success, 17 from success to failure. The introduction states 31.3% of Visible cells invoke specific prior knowledge of the target software; RQ3 states 31.3% of cells contain prior-knowledge clues, 24.2% specifically invoke vulnerability knowledge. Copyparty goes from 2/12 anonymized to 12/12 visible.

Swipe horizontally to read the table

System	BB anonymized	BB visible	GB anonymized	GB visible
Claude Opus 4.8	6	10	3	14
Claude Sonnet 4.6	2	8	5	13
GPT-5.4	5	10	11	13
GLM 5.2	3	6	8	8
Kimi K2.6	2	4	1	7
MiniMax M2.5	1	2	4	3

Per Figure 8, units are number of successes out of 20 vulnerabilities. The authors annotate BB changes from +5 pp to +30 pp and GB from −5 pp to +55 pp; the only negative annotation is MiniMax M2.5's greybox.

• Authors' interpretation: the authors state software identity substantially helps discovery but can also mislead. On Gitea, Kimi K2.6 in Visible blackbox pursues historical vulnerabilities unrelated to the target authorization flaw and fails. The authors state guesses about products or historical vulnerabilities under anonymized conditions are usually incorrect; anonymized successes show discovery is possible through exploration, but the overall discovery rate still has room for improvement.

Failure modes

• What was tested: 1,353 failures among the main experiments' 1,584 runs, all trajectories classified into five categories. Visible's 142 failures each also have one primary category; in thin records, exhausting attempts without a target candidate is conservatively classified as process control and marked low-confidence in the audit ledger (Appendix E).
• Results: C-I policy inference failure 34.59%, C-II process control 24.24%, C-III wrong direction 22.25%, C-IV self-assessment 14.41%, C-V evidence handling 4.51%. Table 4: among Main failures, Opus 4.8's C-I is 59.1% (120/203), GPT-5.4's C-III is 50.9% (110/216), Kimi K2.6's C-II is 55.8% (139/249). On the 63 solved vulnerabilities versus the 69 unsolved ones, C-I is 30.3% versus 37.3%, C-IV is 18.7% versus 11.7%. Among Visible failures, process control is 47.9%, wrong direction 33.8%, policy inference 4.9%.
• Authors' interpretation: the authors state the most common obstacle is failing to infer who, in what state, should not perform which operation on which object. The conclusion states failures come less from interface exploration and more from composing subject, permission, state, and workflow into an effective violating transaction. Figure 9's caption states successes concentrate on the same easier cases, with darker colors indicating more systems passing. In Scaling, Kimi K2.6 on XWiki blackbox makes 138 tool calls and 159 rounds without finding the target vulnerability.

Other ablations and analyses

• Source-code gain (Appendix D.1): GPT-5.4 from 11.36% to 25.00% (+13.64 pp) is largest; Opus 4.8 is +6.82 pp; Kimi K2.6 is smallest (+2.27 pp); the remaining systems range from +5.30 to +7.58 pp. MiniMax M2.5 is lowest overall, but its greybox 7.58% is higher than Kimi K2.6's 6.82%.
• Attribute differences (Appendix D.1, pooled): Authorization +9.0 pp, AuthZ/AccessControl +11.7 pp, object or principal binding +10.8 pp; Workflow/Callback −11.9 pp, SemanticContext −15.3 pp, semantic or context interpretation −15.3 pp; Platform −2.8 pp. Figure 11 additionally gives percentage-point differences by language, CWE, and environment type.
• Submissions: Kimi K2.6 submits candidates in 6.4% of cells, with 88.2% establishing the target effect; MiniMax M2.5 submits 45.5% with 10.0% conversion; GPT-5.4 submits 57.6% with 31.6% conversion; Opus 4.8 submits 48.5% with 47.7% conversion. Among 1,353 failed cells, 976 (72.1%) have no candidate, 369 (27.3%) have a candidate but no observed target effect, and 8 stop at the spending cap.
• Oracle (RQ4): the authors review all trajectories across settings and models and find no false positives or false negatives; the authors state this only bounds observed error and does not establish the absence of error. Replays of submitted PoCs match the final runtime results.
• Passes and reruns: Figure 7 counts 337 passes, of which Main 231, Scaling 8, Visible 98. Appendix E.2 reruns 20 selected cells under the same protocol, with the subset deliberately including 5 observed reversals; of the initial 10 successes and 10 failures, 5 successes become failures and 10 failures remain failures.
• Case study: Section 2's Graylog 6.2.0–6.2.3 (GHSA-3m86-c9x3-vwm9, CWE-285) drops 40% in discovery rate across all models and settings when identity clues are removed. Appendix E.2 states the Keycloak token-exchange task, distinct from the privilege escalation task, is unsolved in all 12 Main cells.

Limitations and future directions noted by the authors

• Time window (Section 7): only vulnerabilities disclosed within the past year are considered, improving timeliness but limiting benchmark size and increasing variance in small subsets.
• Single run (Section 7): each cell is a single run under a fixed budget, and results may vary with attempts.
• Residual clues (Section 7): anonymization removes known identifying clues, but residual signals may remain.
• Human judgment (Section 7): construction includes expert review, and reproducing the construction process may depend on human judgment.
• Scoring scope (Section 7): the oracle verifies the target vulnerability; unrelated valid findings are outside the evaluation scope. The reported ratio measures findings of the specified violation, not findings of arbitrary security flaws.

The paper's Limitations and Conclusion do not list additional experimental plans. Appendix E.2 states that rerun results prompted an examination of how agents select and maintain effective search directions, and states that these results prompted consideration of multi-attempt evaluation.

Experimental coverage

• The main experiment covers six agent systems, with two access types per task, 1 run each; Scaling additionally includes GPT-5.6 Sol and GPT-6 Astra, totaling 10 vulnerabilities and 160 scheduled cells (Section 5, Tables 1–2).
• The corpus is 132 executable business logic vulnerabilities filtered from public open-source disclosures from 2025-01-01 to 2026-03-01, spanning nine language ecosystems and 36 CWEs, with matched blackbox and greybox pairs (Sections 3–4, Figure 5).
• Each admission environment has a manually verified reference PoC; certification includes differential testing of the vulnerable version against the matching patched version, as well as unattacked, legitimate, wrong-subject, wrong-object, and incomplete-workflow controls where applicable (Section 3, Appendix B).
• Failure analysis covers all 1,353 failure trajectories from the main experiment; the 142 Visible failures each have one primary class, with some thin records marked as low confidence (Section 5, Appendix E, Table 4).
• RQ4 reviewed evaluation trajectories across all settings and models, checking for oracle false positives and false negatives; the paper does not report inter-annotator agreement values for failure classification (Section 5, Appendix E).

AnonVul is an executable benchmark for distinguishing whether vulnerability discovery agents discover vulnerabilities from the application security model or recall known vulnerabilities using software identity.

The problem the authors address: CTF and synthetic benchmarks saturate or lack realism, and real advisory benchmarks leak answers through product names, versions, and advisories. Business logic vulnerabilities are defined by application-specific subjects, objects, permissions, and workflows; crashes or status codes cannot separate legitimate operations from violations.

The approach reconstructs 132 tasks from public open-source disclosures between 2025-01-01 and 2026-03-01. The pipeline writes specifications, packages them as Direct, Compose, Platform, or Carrier, removes identity clues while preserving security semantics, and then differentially validates against patched versions using passive relational oracles. Evaluation is matched blackbox and greybox; the main experiment runs each cell once, 20 minutes, $5.

• In Table 1, Claude Opus 4.8's attempt-level overall discovery rate is 23.11%, and the six systems combined is 14.58%; vulnerabilities solved at least once are 63/132, and 69 were not solved by any system.
• Source access raises the combined discovery rate from 11.11% in blackbox to 18.06% in greybox. The authors say the semantic context category had no successes.
• For 10 main-experiment unsolved vulnerabilities extended to 60 minutes and $50, 8 of 160 scheduled cells succeeded, and 7 tasks remained unsolved; GPT-6 Astra's 20/20 were blocked by provider-side guardrails.
• On 20 vulnerabilities, anonymized is 21.25% (51/240), and after restoring product, package, and version identifiers it is 40.83% (98/240). RQ3 writes that 31.3% of Visible cells contain prior-knowledge clues. 64 cells changed from failure to success, and 17 reversed.
• Of 1,353 main-experiment failures, strategy inference failures account for 34.59%. The authors reviewed all trajectories and say they observed no oracle false positives or false negatives, and note this does not establish zero error.

The authors conclude that failures come less from interface exploration and more from composing subjects, permissions, states, and workflows into one effective violation transaction. They also say that evaluating vulnerability discovery requires controlling identifiable software knowledge.