Skip to content

VAST: Compiling Real Vulnerability Scenarios into Verifiable Multi-Step Environments to Evaluate LLM Red-Team Agents

VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents

arXiv: 其他 AI 安全论文· arXiv:2611.19465· Zhen WanSCANTIST PTE. LTDIndexed by this site · 2 days ago· Originally publishedEvaluation· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors propose VAST, which compiles vulnerability scenarios into verifiable evaluation environments with 6-level workflow hardening; the authors state that in a 270-episode prompt comparison, after giving a complete workflow guide, the three models' solve rates rise from 12/90 and 14/90 to 76/90.

Questions
Evaluating LLM security agents requires reproducible environments with adjustable difficulty and diagnosable failures; the authors argue that existing CTF-style benchmarks have four design risks: exposure of training data, exaggerated execution fidelity, reskinned diversity inflation, and overly coarse Pass@1 scoring.
Methods
VAST separates vulnerability scenarios from the compiler-introduced workflow hardening layer: three-layer provenance accounting (37,990 records filtered down to 109 seed edges and 81 topics), a 6-level operator ladder (BASE to L7), a 42:39 topic split, and four deterministic verification gates plus two-stage failure labels.
Experiments and results
On Diagnostic-9, deepseek-v4-pro drops from BASE 7/9 to L6 0/9; in the 270-episode prompt comparison, full_guide raises the solve rate to 76/90; on ranges_v2 the three models solve 88/126; after disclosing the worker module of v2_02, from 0/18 to 18/18; in the LoRA recipe comparison, Recovery SFT has a three-seed mean of 4.67/33, with all solves concentrated in v2_03.
Limitations and future work
The authors note: the audit is conducted post hoc; the four gates only verify a given reference path rather than a complete fix; the controlled evaluation covers only 10 compilation clusters and 6 auxiliary targets rather than all 81 topics; Claude/GPT and other APIs were not tested; failure labels are descriptive coverage rather than causal root causes; the LoRA results solve problems on only a single held-out target.
Experimental setupglm-5.3 (z-ai/glm-5.3)、grok-4.7 (x-ai/grok-4.7) et al. · VAST 主基准(81 topics,42:39 划分,1,387 instances)、ranges_v2 辅助套件(6 个非 TopicServer 目标) · solve rate(服务器端 oracle 判定)、four-gate pass rate(G1–G4) et al.
Threat model
The evaluated agent is an external client that is first unauthenticated and then authenticated, receiving only the base URL and low-privilege tenant credentials; the observable contract is the post-login service catalog (routes, parameter names, object IDs) and structured HTTP error responses. Success is determined only by a server-side oracle, not self-reported by the agent; episode resets are managed by the harness or scheduler.
Evaluated models
glm-5.3 (z-ai/glm-5.3)grok-4.7 (x-ai/grok-4.7)deepseek-v4-pro (deepseek/deepseek-v4-pro)Qwen2.5-Coder-7B-Instruct (Qwen/Qwen2.5-Coder-7B-Instruct)
Benchmarks
VAST 主基准(81 topics,42:39 划分,1,387 instances)ranges_v2 辅助套件(6 个非 TopicServer 目标)
Metrics
solve rate(服务器端 oracle 判定)four-gate pass rate(G1–G4)failure-label coverage(category 1/2/4/5/7/8/9)HTTP requests / turnsWall Wins(recovered_and_solved)
AI summaryThe paper proposes VAST, a declarative range compiler that compiles scenarios based on real vulnerabilities into verifiable multi-step environments for evaluating LLM security agents. After post-hoc code and AST auditing, the main benchmark retains 109 seed edges (45 with strict upstream CVE annotations), 81 topic configurations, and 1,387 instances. In a budget-matched controlled experiment of 270 turns at the TopicServer layer, glm-5.3, grok-4.7, and deepseek-v4-pro solved 12/90 under contract_only and 14/90 under generic_hint, while full_guide, which provides the complete workflow protocol, rose to 76/90. On ranges_v2, a 126-turn evaluation covering 6 workflows solved 88/126 (69.8%); after disclosing the installed worker modules, the asynchronous Redis worker chain scenario rose from 0/18 to 18/18.

The paper proposes VAST, a declarative range compiler that compiles scenarios based on real vulnerabilities into verifiable multi-step environments for evaluating LLM security agents. After post-hoc code and AST auditing, the main benchmark retains 109 seed edges (45 with strict upstream CVE annotations), 81 topic configurations, and 1,387 instances. In a budget-matched controlled experiment of 270 turns at the TopicServer layer, glm-5.3, grok-4.7, and deepseek-v4-pro solved 12/90 under contract_only and 14/90 under generic_hint, while full_guide, which provides the complete workflow protocol, rose to 76/90. On ranges_v2, a 126-turn evaluation covering 6 workflows solved 88/126 (69.8%); after disclosing the installed worker modules, the asynchronous Redis worker chain scenario rose from 0/18 to 18/18.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors build a reproducible environment with adjustable difficulty and diagnosable failures, for evaluating the multi-step exploit capability of LLM red-team agents.

    Core question: how to build a reproducible evaluation environment for LLM security agents whose difficulty can be systematically adjusted, whose failure causes can be localized, and which is not inflated by surface diversity.

    • Application scenario: autonomous penetration testing and vulnerability validation are a frontier direction for LLM agents in security offense and defense; security teams need to evaluate models' reasoning about authorization boundaries, patch behavior, and multi-step exploit paths (Section 1).
    • Four design risks of existing benchmarks (summarized by the authors): training data exposure (public writeups can be replayed literally), exaggerated execution fidelity (a runnable service does not mean the claimed mechanism was executed), reskinning-style inflation (surface rewrites passed off as diversity), and overly coarse Pass@1 scoring (unable to localize where the agent gets stuck). The authors state these are motivational risks, not measured defects of each existing benchmark.
    • Core observation: a bare authorization scenario often requires only a single identifier substitution to reach the sink; while keeping the underlying defect unchanged, compiling a stateful workflow in front of it allows separately studying the additional burden introduced by multi-step workflow composition (Section 1.1).
    • Paper proposes: VAST (Verifiable Adversarial Scenario Training & Evaluation), a declarative range compiler with three-layer provenance accounting, a 6-level operator ladder, a 42:39 topic split, four deterministic verification gates, and structured failure labels; it also provides a Gymnasium environment and curriculum scheduler as infrastructure.
    • Evaluation setup (not an attack threat model): the evaluated agent is an external client that is first unauthenticated and then authenticated, with only the base URL and low-privilege tenant credentials; the observable contract is the post-login service catalog plus structured HTTP errors; success is determined solely by the server-side oracle, not self-reported by the agent (Section 2.2).
  2. What related research exists?

    Related work is divided into four groups: CTF-style benchmarks, verifiable software engineering and vulnerability corpora, REST fuzzing and RL environments; the authors use Table 1 to position VAST's differences.

    CTF-derived benchmarks

    • InterCode-CTF (2023) — wraps picoCTF challenges in a bash execution loop, targeting command-line interaction and short flag tasks; the authors cite its public task count and do not evaluate difficulty.
    • NYU CTF Bench (2024) and Cybench (2025) — curate university and professional CTF challenges; Cybench introduces optional subtask milestones as intermediate progress signals.
    • XBOW's XBEN (2024) — 104 containerized web validation benchmarks, stratified by coarse L1–L3 difficulty; the same group also includes PentestGPT, autonomous web/one-day exploit agents, and CyberSecEval 2. The authors point out three limitations of these suites for evaluating enterprise red-team capability: public solutions are reusable, they emphasize puzzles rather than authorized workflows, and they lack a native mechanism for adjusting multi-step prerequisite ladders on fixed vulnerability primitives.

    Verifiable software engineering, vulnerability corpora, and fuzzing

    • Defects4J, LAVA, EvilCoder, Magma — establish reproducible defect and vulnerability evaluation.
    • RESTler — stateful REST API fuzzing, emphasizing producer-consumer dependencies in multi-step HTTP workflows.
    • SWE-bench (2024) — real GitHub issues paired with fail-to-pass/pass-to-pass unit tests; the authors state that its unit-test verification is similar to but not identical to this paper's patch gate G4.

    Verifiable RL and curriculum design

    • DeepSeek-R1 and GRPO, prioritized level replay, regret-based environment design — the authors state that these works show that deterministic verifiers and adaptive task scheduling can support learning; VAST's corresponding contributions are the Gymnasium-compatible environment VastGymEnv and the scheduler VastCurriculumScheduler, for future experiments.

    Baselines and comparison setup: this paper has no methodological baseline; the comparison objects are three frontier models (glm-5.3, grok-4.7, deepseek-v4-pro) on the VAST main benchmark and the ranges_v2 auxiliary suite, plus two LoRA SFT recipes for Qwen2.5-Coder-7B-Instruct.

    Authors' positioning: Table 1 compares five benchmarks by documented capability; VAST checks the boxes for compiler-introduced multi-step state, 6-level difficulty ladder, per-reset surface randomization, and error-code failure labels, with the G4 patch check marked partial (covering only the tested path).

  3. How does the paper address this problem?

    VAST uses three-layer provenance accounting plus declarative compilation to separate vulnerability primitives from the 6-level workflow hardening layer, then verifies with four deterministic gates.

    Overall approach: decouple vulnerability scenarios (primitive × execution stack × business state machine) from the compiler-introduced workflow hardening layer, render instances via the VAST compiler, and use deterministic gates and structured error codes to support reproducible evaluation and failure diagnosis.

    Three-layer provenance accounting and admission control

    • Four non-interchangeable sets: raw record pool F (|F| = 37,990: GHSA 37,552, Vulhub 334, XBEN 104, all pinned commits) → seed edges S (|S| = 109, 68 from XBEN, 41 from external advisories, 45 with strict CVE annotation) → topic configurations M (|M| = 81, defined as m = ⟨ p, τ, v ⟩, i.e., the combination of primitive, execution stack, and business state machine) → rendered instances I (|I| = 1,387 = 359 canonical + 1,028 variants).
    • Five-question provenance policy: defective code location, real interpreter execution, Entry→Precondition→Sink directed edges, deterministic server-side state evidence, CVE annotation (recorded rather than a hard threshold); additionally a label-based admission heuristic (verif ≥ 0.50, lift ≥ 0.50, real ≥ 0.40), which the authors describe as a policy filter rather than a runtime fidelity certificate.
    • Four descriptive admission controls: single topology share ≤ 40% (actual maximum injection_2nd_order at 34/109 = 31.2%), CVE annotation recorded, shared HMAC relay pattern frozen at 11 families, entry and business object visibility (unlinked secret trap endpoints prohibited). Seed label diversity HHI = 0.160, evenness 0.925.
    • Post hoc audit: after line-by-line code audit of 12 extended topics plus automated AST scanning of all 88 topics, 7 topics were withdrawn and 5 downgraded to bounded execution models; the historical registry (88 topics, 116 edges, 1,597 instances) remains unchanged.

    6-level operator curriculum ladder

    • Ladder definition: R0(BASE) ⊂ R1(P1) ⊂ R2(P2) ⊂ R3(P3) ⊂ R4(P4) = R5(L7), mounting in order the session gate (Op 3), three-dimensional business invariants (Op 4), two-phase draft rebinding (Op 7), and stateful protocol lock (Op 8); P4 and L7 mount the same six server capabilities, L7 is only a historical archive label, and its verification gate additionally exercises canary/quota probes.
    • Property statement: the ladder only adds application-layer preconditions and does not constitute a service, host, or privilege boundary; the nominal step counts (1/2/3/6/6/6) are the number of client workflow stages, not measured solve length.
    • 42:39 split: Track-A's 42 topics expose all six levels (252 canonical specifications), Track-B's 39 expose only some levels (107 specifications); the authors state this is a proposed evaluation split, and the experiments in this paper do not test generalization across splits.

    Four deterministic verification gates and failure labels

    • Verification formula: Validate(I) ⇐⇒ G1(I) ∧ G2(I) ∧ G3(I) ∧ G4(I) = True; G1 benign workflow passes with no marker leakage, G2 black-box reference exploit (without reading ground-truth) satisfies the win predicate, G3 negative probes reach expected outcomes, G4 patch blocks G2 while preserving G1.
    • Boundary statement: the four gates only verify a given reference path and do not prove a complete fix or dynamic taint equivalence; 359/359 canonical specifications rerun and pass on the current source (3 repetitions each, seed 7), and the 1,028 variants were not rerun.
    • Two-stage failure attribution: Stage 1 matches structured server error codes to category 1/2/4/7/8/9; Stage 2, when there is no match, falls back to category 5 or 7 based on trajectory milestones (whether the draft was viewed, whether amend was called). The authors state explicitly: early termination with 0 requests is also given the original label of category 5 and is distinguished separately in the evaluation tables; labels are descriptive coverage rather than causal root causes.

    Online mounting audit and execution stratification

    • Capability graph audit: across 42 Track-A topics × 6 levels, 252/252 instances broadcast a capability graph consistent with expectations, and 210/210 non-BASE missing-step probes were rejected; this is a cumulative mounting check, not a factorial ablation isolating the effect of a single operator.
    • Three-layer verification stratification: 35 declaratively compiled topics go through the shared TopicServer chassis plus AST scanning, 2 handwritten pilots are verified directly, and 39 Track-B topics rely on AST scanning plus rerunning the four gates; the authors state that static AST plus reference-path gates cannot prove dynamic data-flow reachability under arbitrary inputs.

    Evaluation protocol

    • Main-set harness: exposes two tools, http_request and local hmac_sha256; the agent receives only the base URL and low-privilege credentials; completion is determined strictly by the server-side Oracle(response, binding); ranges_v2 uses an independent single-tool harness (max_tokens = 1024, ≤ 4 calls per round).
  4. What experiments did the paper conduct?

    Five groups of experiments: Diagnostic-9 pairing, 500-round stress, four-level sample ladder, 270-episode prompt comparison, ranges_v2 and LoRA recipe comparison.

    Experimental Setup

    • Models under test: deepseek-v4-pro, grok-4.7, glm-5.3 (frontier models, main set and ranges_v2); Qwen2.5-Coder-7B-Instruct (LoRA experiments, Base plus 6 adapters).
    • Benchmarks: VAST main benchmark (81 topics, 42:39 split, 1,387 instances; the experiments actually use Diagnostic-9, 4-topic ladder samples, and 30 P2/P3/P4 instances from 10 compiled clusters); ranges_v2 auxiliary suite (6 non-TopicServer targets, 14 target-level configurations × seeds {7, 19, 31}).
    • Metrics and determination: solving is determined only by the server-side oracle; failures use category 1/2/4/5/7/8/9 labels; statistical tests are exact McNemar, topic-clustered exact sign test, and Cochran's Q.
    • Budgets: Diagnostic-9 paired 40 rounds/120 requests; stress test 500 rounds/unlimited requests; ladder samples 35 (DeepSeek)/40 (GLM) rounds; prompt comparison 40 rounds/unlimited requests, temperature 0.0, sampling seed 7; ranges_v2 40 rounds (also reporting ≤22-round projections).
    • Sample sizes: main-set API runs 430 episodes, ranges_v2 375 episodes, 805 analysis episodes in total (Table 10); LoRA 60 trajectories per arm, 120 steps, 3 training seeds.
    • Repetitions: the four-gate rerun is 3 times each; LoRA 3 seeds; the rest are single runs.

    Main Results

    270-episode three-model prompt comparison (10 compiled clusters × P2/P3/P4, 40 rounds, temperature 0.0; according to Table 7):

    Swipe horizontally to read the table

    Modelcontract_onlygeneric_hintfull_guide
    glm-5.35/30 (16.7%)6/30 (20.0%)28/30 (93.3%)
    grok-4.75/30 (16.7%)5/30 (16.7%)23/30 (76.7%)
    deepseek-v4-pro2/30 (6.7%)3/30 (10.0%)25/30 (83.3%)
    Combined (N=90)12/90 (13.3%)14/90 (15.6%)76/90 (84.4%)
    • generic_hint has no net gain: cluster sign test vs contract_only p = 1.00 (the three models 3:2, 1:1, 2:1 respectively); the authors state that it shifts solving toward P2 (12/30 vs 5/30) but does not improve P3/P4.
    • full_guide improves: cluster sign test vs contract_only p ≤ 0.00391 (10:0, 9:0, 9:0), and the average number of requests drops from 50.7–64.4 to 20.9–26.6; the authors interpret this as scaffold solvability under a known protocol, not primitive isolation difficulty.
    • 14/90 remain unsolved under full_guide: 7 are grok-4.7's 0-request client early stops, and 7 are multi-request failures on two-hop tasks (CFB-64-VRF, CFB-55-ITK).

    Diagnostic-9 Pairing and 500-Round Stress Test

    • Paired comparison (RQ1/RQ2): deepseek-v4-pro on 9 same-family instances solves 7/9 with BASE and 0/9 with L6 (7 same-direction discordant pairs, exact McNemar and sign test p = 0.0156); the authors explicitly state this is a diagnostic observation on 9 paired samples, not a population estimate.
    • 500-round stress: grok-4.7 and glm-5.3 each solve 1/9 on the same 9 L6 instances with unlimited requests; the 16 failed episodes send 5,755 requests in total (mean 359.7), 16/16 end with model_finished, and none hits the 500-round cap.
    • Failure labels: 15/16 carry server error labels (category 8 accounts for 8, category 7 for 5, category 1 for 2), and 1 is a grok-4.7 safety refusal (stopped after 15 requests), recorded separately and not counted in the server-error denominator.

    Four-Topic Six-Level Ladder Samples (RQ3)

    • What was tested: deepseek-v4-pro and glm-5.3 on 4 topics × 6 levels, 48 episodes in total (Table 5, Figure 2).
    • Results: on BASE and P1, glm-5.3 solves 4/4 and deepseek-v4-pro solves 3/4; from P2 onward this drops to 0/4 and 1/4, and P3/P4/L7 are all 0/4; adjacent-level paired differences are not significant (P1→P2 exact McNemar p = 0.25).
    • Authors' interpretation: among the 33 ladder failures, category 7 accounts for 15, category 1 for 6, and categories 5 and 8 for 5 each; the authors emphasize that this is a label distribution rather than a proven cause, and that the 4 topics are diagnostic samples, not a representative sample of the 81 topics.

    ranges_v2 Non-TopicServer Suite and Disclosure Intervention (RQ5)

    • 126-episode evaluation: the three models combined solve 88/126 (69.8%) on 6 targets (glm-5.3 28/42, grok-4.7 31/42, deepseek-v4-pro 29/42; Table 8); v2_03 dual-network pivot 18/18, v2_02 Redis worker chain 0/18; none of the 126 episodes abandons at 0 requests.
    • Disclosure intervention: after disclosing the list of installed worker modules in v2_02's GET /sched/directory, the 18-episode matched intervention raises solving from 0/18 to 18/18 (100%, 6:0 per model, exact McNemar p = 0.03125), completed within 4–6 rounds; the authors state that this isolates an undisclosed namespace barrier rather than a queue/gate failure.
    • BASE vs L6 retention: on 18 paired instances, the three models' BASE/L6 solving is 13/18 vs 12/18, 15/18 vs 11/18, and 10/18 vs 13/18 respectively, with cluster sign tests all non-significant (p = 1.00, 0.50, 0.25).

    Three-Seed LoRA Recipe Comparison (Section 6.3)

    • Setup: Qwen2.5-Coder-7B-Instruct, Arm A pure Golden SFT (60 error-free trajectories) vs Arm B Header-Recovery SFT on a 60:40 mix of raw trajectories; 37 lineage-disjoint training families, s ∈ {42, 1024, 2025}, 33 held-out instances, 22 rounds, 231 episodes (Table 9).
    • Results: Base 0/33; Arm A across three seeds 0/33, 0/33, 5/33 (mean 1.67 ± 2.89/33); Arm B 6/33, 6/33, 2/33 (mean 4.67 ± 2.31/33, 2/3 seeds higher than Arm A, seed 2025 reverses); the two arms differ by only +1.77% in active assistant tokens.
    • Authors' interpretation: all 19 solves are concentrated in v2_03_dmz_dualnet_pivot, and the other four held-out targets are 0/27; the authors state that this is an exploratory recipe comparison on a single lineage-held-out HTTP pivot stack, not multi-target OOD transfer.

    Other Ablations and Analyses

    • RQ4 Part 2 historical intervention: after adding a shared recovery guide, deepseek-v4-pro rises from 0/12 to 12/12 and glm-5.3 from 1/12 to 12/12, but DeepSeek's round cap is simultaneously changed from 35 to 40 (Table 6).
    • 36-episode pilot: glm-5.3 on 12 ladder instances gets contract_only 0/12, generic_hint 1/12, full_guide 12/12.
    • Scheduler smoke test: glm-5.3 on 2 execution models solves 14/16 across 16 episodes, including degradation, seed rotation, and guide activation; the authors state that this does not constitute a no-guidance curriculum result or a learning gain.
    • Directory discoverability: the directories of 40/40 compiled Track-A topics expose draft endpoints and invariant fields within 3 requests.
    • Trajectory export: 2,012 historical exports, 1,797 after filtering; the authors state that no experiment trains on this single-turn summary library.
  5. What are some points that can be further explored?

    The authors list limitations including post hoc auditing, gate coverage boundaries, and sample and cohort coverage; the experiments cover 10 compilation clusters plus 6 auxiliary targets.

    Limitations and future directions noted by the authors

    • Post-hoc audit: The code-level mechanism audit was performed after the historical runs of 2026-11-08 to 2026-11-10 and was not pre-registered; the historical 116/79/88/1,597 registry remains unchanged (Section 6.4).
    • Gate coverage boundaries: G1–G4 only verify the given benign, reference-exploit, negative, and patch-switch paths, and do not prove complete remediation or unguided contract discoverability; G4 only blocks the given reference path (Sections 3.4, 6.1).
    • Interpretation boundaries of guided success: full_guide measures scaffold solvability under a known protocol, not primitive isolation difficulty; the 7 zero-request stops and 7 multi-request failures still carry workflow/fallback or unknown labels (Section 6.1).
    • Sample size and cohort coverage: The controlled multi-condition evaluation covers 10 compiled TopicServer clusters plus 6 auxiliary ranges_v2 targets, rather than all 81 topics, 39 Track-B handwritten families, or US-lab frontier APIs such as Claude/GPT (Section 6.4).
    • Nature of failure labels: The category labels report observed refusal coverage, not proven causal root causes; model-side safety refusals and zero-request stops are counted separately from server errors (Section 6.4).
    • Containment and fidelity boundaries: sitecustomize.py is a Python runtime hook rather than a kernel firewall, and the authors claim only Python-level loopback containment; the 5 degraded models execute stdlib/loopback rather than multi-host isolation; route visibility does not prove that every required state transition is discoverable from the contract (Sections 2.2, 6.4, A.7).

    Experimental coverage

    • Models tested: 3 frontier models (glm-5.3, grok-4.7, deepseek-v4-pro) plus Qwen2.5-Coder-7B-Instruct (Base and 6 LoRA adapters), for 4 model families in total.
    • Main benchmark coverage: Controlled experiments use Diagnostic-9 (9 families), the 4-topic ladder sample, and 30 P2/P3/P4 instances from 10 compiled clusters; the 42:39 split was not used for cross-split training/testing.
    • Auxiliary suite: 6 targets in ranges_v2 (2 independent workflows + 4 Docker/Linux stacks), 14 configurations × 3 instance seeds.
    • Statistical design: Exact paired tests are used for RQ1/RQ2 (N = 9), RQ4 (N = 10 clusters/30 instances), and RQ5 (N = 6 targets/18 pairs); for the ladder sample with N = 4, adjacent-level comparisons have p ≥ 0.25.
    • Not reported in the paper: The model-turn limit and request limit of the scheduler smoke test (marked as unspecified); the complete list of excluded historical instances for Diagnostic-9; the configuration cardinality of the surface randomization (the implementation does not record the breakdown).
  6. Summarize the main content of the paper

    VAST compiles vulnerability scenarios into verifiable environments with 6-level workflow hardening; experiments show that frontier models mostly get stuck at the workflow layer without guidance.

    VAST is a declarative range compiler and verifiable evaluation framework for multi-step exploit evaluation of LLM red-team agents; its core design is to separate vulnerability primitives from the compiler-introduced workflow hardening layer.

    • Problem addressed: the authors argue that existing CTF-style benchmarks have four risks — training data exposure, exaggerated execution fidelity, reskinning-style diversity inflation, and overly coarse Pass@1 scoring — and lack a mechanism for adjusting multi-step preconditions on fixed vulnerability primitives.
    • Method highlights: three-layer provenance accounting (37,990 records → 109 seed edges → 81 topics → 1,387 instances), a 6-level operator ladder (BASE to L7, with P4 and L7 mounting the same capabilities), a 42:39 topic split, four deterministic verification gates (covering only a given reference path), and two-stage failure labels; post hoc code audit withdrew 7 topics and downgraded 5.
    • Key result 1: on Diagnostic-9, deepseek-v4-pro drops from 7/9 at BASE to 0/9 at L6 (p = 0.0156); under 500 rounds with unlimited requests, grok-4.7 and glm-5.3 each solve only 1/9.
    • Key result 2: in the 270-episode prompt comparison, contract_only 12/90, generic_hint 14/90 (cluster sign p = 1.00), and full_guide rises to 76/90 (p ≤ 0.00391); the authors state that the unguided condition tests contract exploration plus state machine composition, while full_guide tests scaffold exploitation under a known protocol.
    • Key result 3: on the six non-TopicServer targets in ranges_v2, the three models solve 88/126 (69.8%); after disclosing v2_02's worker module list, it rises from 0/18 to 18/18 (p = 0.03125).
    • Key result 4: in the three-seed LoRA recipe comparison, Header-Recovery SFT averages 4.67 ± 2.31/33, higher than Golden SFT's 1.67 ± 2.89/33, but all solves are concentrated on the single target v2_03 (the remaining 0/27).
    • Authors' conclusion: VAST provides an auditable foundation for diagnosing and training multi-step security agents; the authors also emphasize that the four gates only verify the tested path, failure labels are descriptive coverage, the controlled evaluation does not cover all 81 topics, and the LoRA results are only an exploratory recipe comparison.
Read originalarxiv.org