Skip to content

P2Patch: Exploit-Guided Vulnerability Repair

P2Patch: Exploit-Guided Vulnerability Repair

arXiv· arXiv:2610.20245· Mohsen Iranmanesh, Sina Moradi Sabet, Sina Marefat, Ali Javidi Ghasr, Maxime Newman Nereyabagabo, Kousha Amouzesh, Jaivir Malhi, Vinci Velasco, Alex Chung, Mohammad A. TayebiSimon Fraser University; Amirkabir University of Technology; K. N. Toosi University of Technology; Ferdowsi University of MashhadIndexed by this site · 3 days ago· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose P2PATCH: after a patch is generated, an exploit agent continues attacking the same vulnerability; on 101 CVEs, 41% of accepted patches were bypassed, and held-out coverage rose to 0.847/0.625.

Threat modelTargets vulnerabilities in C/C++ and Java projects that have been reported and pinned to a version; the attacker controls inputs related to the finding (archive entries, files, buffers, request parameters, serialized objects, command-line arguments, network messages); the agent is not given CVE numbers, upstream patches, advisory exploit text or held-out PoVs, and network retrieval is disabled; the host and benchmark repositories are treated as trusted infrastructure.

Questions
Automated vulnerability repair often uses the same evidence (alert, crash input, PoC) both to guide patching and to determine success, so the patch may only specialize to that evidence while the vulnerability remains; the authors call this the oracle-reuse problem.
Methods
P2PATCH has the exploit agent first turn a finding into a reproducible PoV; after the patch agent repairs, the exploit agent attacks the patched program again to find new bypasses of the same vulnerability; each execution-confirmed bypass becomes a permanent repair requirement, and the final patch is scored with a held-out PoV suite isolated from construction.
Experiments and results
On 101 CVEs, 41% of runs still found execution-confirmed new bypasses after the patch blocked the original PoV; compared with single-pass repair, held-out Cfix rose from 0.781 to 0.847 and Cres from 0.421 to 0.625; re-scoring on shared subjects of three published systems was also higher.
Limitations and future work
The authors state the protocol is bounded testing rather than a completeness proof; the held-out suite consists of manually written paths rather than the entire input space; pretraining memory cannot be ruled out; executable PoVs for 18 residual suites are withheld from release; there is only one run per subject, and the paper reports no repetition counts.
Experimental setupClaude Haiku 4.5、GLM 5.2 et al. · VulnRepairBench、VulnLoc+ · Cfix、Cres et al.
Evaluated models
Claude Haiku 4.5GLM 5.2DeepSeek V4 FlashGPT-5.6-luna
Benchmarks
VulnRepairBenchVulnLoc+
Metrics
CfixCresmacro coveragebypass ratecost per subject
AI summaryResearchers propose a generative adversarial repair system, P2Patch, which treats each patch as a new attack target. An exploitation agent first constructs a proof of vulnerability (PoV), a repair agent applies a patch, and then the patched program is attacked with a new exploit of the same vulnerability; a successful exploit becomes a permanent repair requirement, and the final patch must block all accumulated PoVs, with construction-phase PoVs strictly separated from a held-out evaluation oracle. On 101 C/C++ and Java CVE samples, in 41% of runs the patch blocked the construction-phase PoV but was still broken by a new exploit of the same vulnerability. Compared with single-pass repair starting from the same finding using Claude Code, P2Patch raises held-out fix-closed coverage from 78.1% to 84.7% and beyond-upstream coverage from 42.1% to 62.5%, and on overlapping samples it certifies more blocked attack paths than recent state-of-the-art agent repair systems. The authors state that adversarial testing and an independent security oracle can expose repair gaps missed by single-pass validation.

Researchers propose a generative adversarial repair system, P2Patch, which treats each patch as a new attack target. An exploitation agent first constructs a proof of vulnerability (PoV), a repair agent applies a patch, and then the patched program is attacked with a new exploit of the same vulnerability; a successful exploit becomes a permanent repair requirement, and the final patch must block all accumulated PoVs, with construction-phase PoVs strictly separated from a held-out evaluation oracle. On 101 C/C++ and Java CVE samples, in 41% of runs the patch blocked the construction-phase PoV but was still broken by a new exploit of the same vulnerability. Compared with single-pass repair starting from the same finding using Claude Code, P2Patch raises held-out fix-closed coverage from 78.1% to 84.7% and beyond-upstream coverage from 42.1% to 62.5%, and on overlapping samples it certifies more blocked attack paths than recent state-of-the-art agent repair systems. The authors state that adversarial testing and an independent security oracle can expose repair gaps missed by single-pass validation.

Related events and news

1 events · 1 news mentions

News that mentions this paper

研究称AI编写补丁常修不完整,反成攻击路线图

  • CSO Onlinenames the paper
    AI 编写的补丁正在成为黑客的攻击地图
    More recently, in a benchmark test called P2Patch, researchers pitted two AI agents against each other: one wrote exploits for 101 real vulnerabilities, and the other patched the code until the

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Repair systems use the same evidence to construct and validate patches, which the authors say overestimates repair completeness; they propose exploit-guided adversarial repair.

    Core problem: automated vulnerability repair has an oracle-reuse problem — the same evidence (static alert, sanitizer trace, PoC, test) is used both to construct the patch and to determine success, so the patch may block only that one piece of evidence while the vulnerability can still be triggered by variants.

    • Scenario and motivation: the authors state that static alerts point to suspicious flows but do not prove exploitability, a PoC demonstrates only one failing execution, and test suites rarely define the full security boundary; in the plexus-utils zip slip case, the 2016 official fix used startsWith for a containment check, which could be bypassed by sibling paths and was only closed by a second fix more than nine years later (Figure 1).
    • Shortcomings of existing methods: the authors state that existing repair systems fix their repair target before patching, and each repair attempt tries to satisfy the same alert, crash, trace, or PoV, so the target does not adapt as the program is patched; recent evaluations also show that validation with PoCs plus functional tests overestimates repair correctness.
    • Core observation: the authors argue that once a patch changes the program, the key question is not only whether the original PoV is blocked but whether a different exploit can be found to falsify the current patch's security claim; this resembles counterexample-guided synthesis, but the verifier is a budgeted exploiter rather than a complete decision procedure.
    • Paper proposal: P2PATCH, an execution-judged exploit–patch adversarial repair system, plus VulnRepairBench, a benchmark of 101 project–CVE subjects, which strictly separates construction-time evidence from evaluation-time evidence using independently certified held-out PoV suites.
    • Threat model (Section 2.4):
      • Target: repair of reported vulnerabilities, for pinned C/C++ and Java project versions, studying repair rather than triage.
      • Attacker capability: control of inputs related to the finding, such as archive entries, files, buffers, request parameters, serialized objects, command-line arguments, or network messages.
      • Knowledge and isolation: agents do not receive CVE numbers, upstream fixes, advisory exploit text, or held-out PoVs, and network retrieval is disabled; the authors state this cannot eliminate knowledge acquired through pretraining.
      • Trust assumptions: the evaluation host and benchmark repository are treated as trusted infrastructure, and edits, commands, and claims produced by agents are treated as untrusted input; the prototype does not claim to defend against malicious repositories.
  2. What related research exists?

    The paper divides related work into five categories: automated vulnerability repair, what the repair loop optimizes, PoV generation, patch validation, and benchmarks, and notes that the difference lies in whether the repair target is movable.

    Automated vulnerability repair systems

    • Learning-based methods (SequenceR, CURE, etc.) — translate vulnerable code into fixed code under assumed edit locations; analytical systems (ExtractFix, VulnFix, SemFix, Angelix) — derive repair constraints from concrete failures or vulnerability witnesses.
    • Agentic systems (APPATCH, PatchAgent, San2Patch, CodeCureAgent, OSS-Fuzz repair agents, LoopRepair) — use LLMs to iteratively repair with vulnerable code, alerts, traces, or reproduction artifacts; the paper notes their repair target is usually fixed before patching.

    What the repair loop optimizes (Table 11)

    • San2Patch loops over reasoning paths (ToT), PatchAgent loops over agent actions, and LoopRepair loops over candidate patches, all with a fixed success target; the paper notes that P2PATCH changes the repair target itself: once a patch is accepted, the patched program becomes the next attack target.

    Exploitation and PoV generation

    • SemFuzz generates kernel exploits from vulnerability text, FaultLine generates language-agnostic PoV tests, PoC-Gym studies static trace guidance for Java PoVs, and CVE-Genie reconstructs environments and verifiable exploits from public CVE materials; the paper notes these works focus on making vulnerabilities executable, whereas P2PATCH treats that capability as a repair primitive.

    Patch validation and benchmarks

    • Test-suite overfitting is a long-standing problem in program repair (Smith et al., Le et al.); PVBench shows that PoVs plus functional tests overestimate repair success rates, and VulnRepairEval and PatchEval move toward executable containerized security tests; the paper notes P2PATCH uses executable validation but separates construction-time PoVs from the held-out evaluation suite.
    • Many repair evaluations target the official fix as the target behavior, which the authors say makes the official fix a ceiling: paths not closed by the official fix cannot be measured.

    Baselines and benchmark: comparison baselines are SINGLE PASS (single repair in the same Claude Code framework), San2Patch, PatchAgent, and LoopRepair; the benchmark is the paper's own VulnRepairBench (101 CVEs, with the C/C++ portion derived from VulnLoc+). Author positioning: this paper is the first counterexample-guided exploit–patch protocol that turns each execution-confirmed bypass into a permanent repair requirement and evaluates with an independent held-out safety oracle (the authors state this as their contribution, phrased in the original as a contribution list rather than a "first" claim).

  3. How does the paper address this problem?

    P2PATCH uses an adversarial loop with an execution judge: PoV-guided repair, the patch is attacked again, confirmed bypasses are appended as permanent repair requirements, and the held-out suite is scored independently.

    Overall approach: P2PATCH treats each patch as a new attack target, strictly separates construction-time evidence from evaluation-time evidence, and agents only propose PoVs and patches, with success determined by execution in the orchestrator.

    Problem setting and formalization

    • Given a pinned vulnerable version P0, a normalized finding f, and a project execution contract C, produce a source patch π or a structured failure; a patch is construction-valid if and only if P0 ⊕ π blocks all accumulated live PoVs and has no differential regression.
    • The execution judge is three-valued: EXEC(P, e) ∈ {REPRO, BLOCKED, INCON}, denoting reproduction, blocking, and inability to decide due to timeout/build failure etc.; INCON scores for neither side, avoiding counting bad infrastructure as a successful repair.
    • The requirement set only grows: whenever execution confirms a new bypass, Er+1 = Er ∪ {er+1}, and b = |E| − 1 denotes the number of bypasses appended after bootstrap.

    Bootstrap: making findings executable

    • An exploiter agent generates candidate PoV e0 from the normalized finding; it is accepted only if EXEC(P0, e0) = REPRO; timeouts, build failures, missing harnesses, or unobserved vulnerable behavior all count as INCON or BLOCKED and do not enter the requirement set.
    • If no reproducible PoV is found within budget, a structured failure is returned, and the patch agent is not allowed to repair an unexecuted claim; e0 is construction evidence, not an evaluation oracle.

    Patch convergence and adversarial hardening

    • The patcher agent edits product code with a safety-first gate: regression checks are considered only after all protected PoVs are judged BLOCKED; each round the gate checks the latest bypass plus e0, and early bypasses are replayed uniformly after the hardening loop ends; a failed replay rejects the patch outright with no further chance to fix it.
    • Adversarial hardening: after a patch passes the current gate, the exploiter receives the patched program and its diff and looks for new bypasses of the same vulnerability and same security boundary but with different payloads, encodings, entry points, or sibling paths; the main configuration fixes R=4 rounds, and stable exit only means no new bypass was found within budget, not a proof that the vulnerability does not exist.

    Execution judge and artifact integrity

    • Regression checks are differential: a test that fails after patching counts as a regression only when comparison with the original version supports attribution; live gating and final replay use the same execution semantics, and the held-out suite is mounted only after repair ends.
    • Role permissions are asymmetric: the exploiter can create attack artifacts but cannot weaken product code, and the patcher can change code but cannot delete or rewrite protected PoVs; the orchestrator snapshots before calls and restores unauthorized changes.
    • The prototype is a Python orchestrator with network retrieval and inherited settings disabled; evaluation uses κatt=3 attack attempts, κgate=3 gate evaluations, and R=4 rounds (Appendix C).

    Independent safety oracle and success determination

    • Fix-closed PoV: reproduces on the vulnerable version and is blocked by the official fix, measuring held-out attack path coverage Cfix; Residual PoV: reproduces on both the vulnerable version and the official fixed version, measuring hardening beyond upstream Cres.
    • Coverage is computed only over conclusive executions, with INCON removed from the denominator; the final patch and upstream diff are not compared syntactically but behaviorally: whether they block the independent certified paths that the official fix blocks.
    • Leakage control: agents do not receive CVE numbers, upstream patches, advisory exploit text, or held-out files, and network retrieval is disabled; the authors state this cannot eliminate pretrained knowledge and claim only that the repair loop cannot inspect the evaluation suite.
  4. What experiments did the paper conduct?

    On 101 CVEs, 41% of accepted patches were bypassed by new attacks; held-out Cfix 0.781→0.847, Cres 0.421→0.625, with gains mainly from the adversarial loop, reproduced across four model families.

    Experimental setup

    • Models tested: the main experiments (6.1–6.3) fix Claude Haiku 4.5; the model sweep (6.4) additionally tests GLM 5.2, DeepSeek V4 Flash and GPT-5.6-luna, with the same model serving as both exploiter and patcher in each arm.
    • Benchmark: VulnRepairBench, 101 CVEs (60 Java, 41 C/C++), 61 repositories, 20 weakness classes; the fix-closed suite has 324 PoVs (1–14 per subject), and 40 subjects additionally have a residual suite with 66 PoVs in total (Table 1); 101 reproducible subjects were selected from 138 candidate findings (Appendix D).
    • Baselines: SINGLE PASS (single fix in the same Claude Code v2.1.241 framework, no live PoV); the published systems San2Patch, PatchAgent and LoopRepair were re-scored on shared subjects with the same held-out suite, all using Claude Haiku 4.5.
    • Metrics: macro coverage Cfix/Cres (the proportion of conclusive PoVs blocked within each subject, then averaged), intention-to-treat: subjects that produced no accepted patch count as zero; micro coverage, bypass rate and cost are also reported.
    • Statistical tests: exact Wilcoxon signed-rank test, Holm correction; language breakdown, stratification, re-scoring of published systems and the model sweep are exploratory analyses.
    • Review method: after automated gating passes, two reviewers independently manually review patch diffs under a fixed rubric, blinded to model and configuration, with disagreements jointly adjudicated; the held-out oracle does not participate in acceptance. One run per subject; the paper does not report the number of repetitions.

    Main results

    According to Table 2 (intention-to-treat macro coverage) and Table 4 (model sweep):

    Swipe horizontally to read the table

    Configuration/modelnCfixCresAccepted patchesbypass rateCost $/subj
    SINGLE PASS (Haiku 4.5)101/400.7810.421——0.45
    P2PATCH (Haiku 4.5)101/400.8470.62598/10141%2.09
    P2PATCH (GLM 5.2)101/400.9210.795100/10147%2.43
    P2PATCH (DeepSeek V4 Flash)101/400.8210.60894/10135%0.54
    P2PATCH (GPT-5.6-luna)101/400.5940.40066/10131%0.16
    • The authors say both main tests pass Holm correction (Cfix p=0.0286, Cres p=0.0254); micro coverage Cfix 0.710→0.781, Cres 0.409→0.712.
    • The authors say GPT-5.6-luna reaches Cfix 0.909 within the 66 accepted patches, but 35 subjects left unfixed lower the intention-to-treat score, 26 of which are patches rejected by manual review.
    • The authors say the comparison is an end-to-end coverage-cost trade-off, not an efficiency ranking.

    RQ1: Bypasses of accepted patches

    • What is measured: after a patch blocks the PoV it was built on, whether the exploiter can find a new bypass of the same vulnerability. Results: 41 of 101 runs (41%) found a different bypass with execution confirmation, 68 in total; the per-round conditional probabilities are 41%, 37%, 53%, 50% (rounds 1–4, samples 99/41/15/8, Figure 4); the authors say the later rounds have small samples and should not be interpreted precisely, but the bypass rate does not disappear with depth.
    • Authors' interpretation: the orchestrator only enforces confirmation and does not machine-check semantic novelty, so 41% is an upper bound on semantically different bypasses; 4 runs still confirmed new bypasses in the fourth round, so the search was truncated rather than exhaustive.

    RQ3: Attribution of gains

    • According to Table 3: on Cfix, live PoV contributes +0.027 (p=0.2944) and the adversarial loop contributes a further +0.038 (p=0.0020); on Cres, PoV contributes +0.050 (p=0.5000) and the loop contributes +0.154 (p=0.0156). The authors say the detectable gains come from the loop rather than the PoV.
    • The loop itself has no regressions: on Cfix, 10 improved, 91 unchanged, 0 worsened; on Cres, 7 improved, 33 unchanged, 0 worsened; all 3 negative pairs in Table 2 occur before the loop (2 unfixed count as zero, and 1 is CVE-2018-17297, where PoV anchoring narrowed the fix surface; the authors say the exploiter found no bypass on that subject, so the loop did not run).
    • Round-budget sweep (Table 9, 98/39 subjects): at R=1, Cres rises from 0.457 to 0.556, then increments diminish, with +0.000 from R=3 to R=4; the authors say most residual gains arrive in the first two rounds.

    RQ5: Re-scoring of published systems

    • According to Table 5: P2PATCH versus San2Patch (n=33) Cfix 0.922 vs 0.608 (+0.314); versus LoopRepair (n=39) 0.930 vs 0.325 (+0.605); versus PatchAgent (n=16) 0.929 vs 0.836 (+0.093); residual gains +0.454 to +0.500.
    • The authors say this does not contradict the original evaluations: the original validation criteria did not measure coverage of the same held-out attack paths; the comparison reflects overall differences in inputs, protocols, budgets, implementation maturity and validation strategy, not a global ranking.
    • Cost (Table 6): P2PATCH averages $2.09/run, SINGLE PASS $0.45, LoopRepair $0.38, PatchAgent $0.32 (the paper notes this is a lower bound).

    Other ablations and analyses

    • Language breakdown (Table 8, exploratory): Java Cfix +0.071, C/C++ Cfix +0.057; C/C++ Cres +0.333 (n=12).
    • Upstream history (Appendix G, Figure 6): of 23 decidable residual gaps, 14 were later closed upstream (median about two years later, maximum nine and a half years; one plexus-utils case was assigned CVE-2025-67030), and 9 remain open in current upstream code and have been disclosed to 8 projects.
    • Negative results: 3 subjects produced no accepted patch under P2PATCH (2 PoVs did not reproduce, 1 patch was rejected by manual review); on CVE-2018-17297, Cfix dropped from 0.80 to 0.20, which the authors attribute to bootstrap PoV anchoring.
  5. What are some points that can be further explored?

    The authors state that the protocol is bounded testing rather than proof, that the held-out suite has coverage bias, that pretraining memorization cannot be ruled out, and that confirmation of some residual executions is left for follow-up work.

    Limitations and future directions noted by the authors

    • Bounded hardening rather than proof (Section 7): the exploiter has a budget, the live PoV set is finite, and the held-out suite consists of manually written paths rather than the entire input space; a stable run only means the configured exploiter found no further bypass, which remains a testing claim rather than a formal proof.
    • Oracle coverage bias (Section 7): the suite contains manually written paths rather than all relevant paths; residual PoVs are certified by inverse rules, and the official fix itself does not certify them; 14 residual gaps were corroborated by later upstream fixes, of which 8 execution-confirmed PoVs were blocked in later trees, and the authors state that extending execution confirmation to every line is future work.
    • Scope of comparison with published systems (Section 7): this is an end-to-end oracle comparison and does not claim that prior systems fail under their original criteria; differences reflect input modality, repair protocol, budget, implementation maturity, and validation strategy, not only the adversarial loop.
    • Pretraining memory (Section 7): pretraining exposure of models to public CVEs, advisories, fixes, or exploit examples cannot be ruled out and may inflate absolute coverage across all configurations; the authors state this is a limitation of evaluating LLM repair on public vulnerabilities.
    • Deployment implications of residual PoVs (Appendices A, G): a residual PoV only means some behavior remains reproducible under the benchmark contract and does not prove exploitability in every deployment; "still open upstream" does not equal exploitable, and reachability from attacker input must be established case by case.

    Experimental coverage

    • 4 models were tested: Claude Haiku 4.5 (main experiments), GLM 5.2, DeepSeek V4 Flash, GPT-5.6-luna; in each arm the same model serves as both exploiter and patcher (Section 6.4).
    • The benchmark is 101 CVE subjects (60 Java, 41 C/C++), 20 weakness classes; C/C++ is derived from VulnLoc+, and Java comes from GitHub Security Advisories and NVD (Section 5).
    • Comparison with published systems covers San2Patch (33 shared subjects), LoopRepair (39), and PatchAgent (16), all C/C++ shared subsets (Table 5, Appendix E).
    • One run per subject per configuration; the paper reports neither repetition counts nor inter-rater agreement values for manual review.
    • The main configuration budget is κatt=3, κgate=3, R=4 (Appendix C); executable PoVs for 18 residual suites (30/66 PoVs) and 4 fix-closed suites (16/324 PoVs) are withheld from release (Appendix B).
  6. Summarize the main content of the paper

    P2PATCH treats each patch as a new attack target for adversarial repair; 41% of accepted patches were newly bypassed, and held-out coverage exceeds single-pass repair and three published systems.

    P2PATCH is an execution-judged, exploit-guided vulnerability repair system targeting the oracle-reuse problem in automated repair, where "the same evidence is used to construct and validate the patch."

    • Problem: the authors state that a patch may be accepted if it only blocks the PoC or alert used to construct it, while variant payloads, encodings, entry points, or sibling paths of the same vulnerability can still trigger it; the plexus-utils zip slip fix being repaired a second time nine years later is an example (Figure 1).
    • Method: the exploiter first turns a normalized finding into a reproducible PoV; after the patcher repairs, the exploiter attacks the patched program again; each execution-confirmed bypass of the same vulnerability is appended as a permanent repair requirement (only grows, never shrinks), and a three-valued execution judge (REPRO/BLOCKED/INCON) prevents counting infrastructure failures as successful repairs; the final patch is scored with held-out fix-closed and residual PoV suites strictly isolated from construction.
    • Bypass rate: on 101 CVEs, 41% of runs still found execution-confirmed new bypasses after the patch blocked the original PoV, 68 in total; the bypass rate does not disappear with hardening rounds (Figure 4).
    • Coverage: compared with SINGLE PASS, held-out Cfix rose from 0.781 to 0.847 (n=101) and Cres from 0.421 to 0.625 (n=40), both tests passing Holm correction; attribution analysis shows the detectable gain comes from the adversarial loop (Cfix +0.038, Cres +0.154) rather than bootstrap PoVs (Table 3).
    • Models and baselines: bypass discovery replicates across four model families (31%–47%); GLM 5.2 reaches Cfix 0.921 and Cres 0.795; on re-scoring shared subjects of San2Patch, LoopRepair, and PatchAgent, P2PATCH has higher Cfix/Cres (Table 5), and the authors state this is an end-to-end comparison rather than a global ranking.
    • Conclusion and cost: the authors argue that adaptive counterexample growth plus an independent safety oracle can expose repair gaps missed by one-shot validation, but it remains a bounded testing method rather than a completeness proof; average cost is $2.09/subject, more than four times single-pass repair ($0.45); 14 residual gaps were later closed upstream, and 9 remain open and have been disclosed.
Read originalarxiv.org