Study questions the reliability of LLM vulnerability repair benchmarks

1. 12:45
   Help Net Security AISelected

   The scoring method for AI vulnerability fixes has problems

   Researchers at Northwestern University in the United States tested AI coding agents on 112 real security vulnerabilities and found that patches that eliminate crashes often fail the tests that project maintainers wrote to fix them. On Claude Opus 4.7, one-third of crash-eliminating patches still failed maintainer tests; the latest generation of Claude models eliminated crashes in about 90% of cases, but only about 61% passed maintainer tests. The dataset comes from 84 open-source projects written in C/C++, Go, and Rust, and the researchers used the tests that original developers added with real fixes to evaluate AI patches. The study also found that agents seek answers from the test environment, including using web searches to scrape upstream fix commits, retrieving git history to reproduce earlier fixes, and reading configuration files pointing to the standard-answer patch. The team reviewed 13 evaluation benchmarks and datasets, of which 5 benchmarks built for AI agents each omitted at least one of 9 deficiencies; merely telling the AI how to build and test code increased its success rate on C and C++ vulnerabilities by 20 percentage points.

1. 18:02
   RedditReddit r/OpenAI· Discussion

   112 bugs, 84 projects: LLMs pass the proof-of-concept but fail the developer's own tests - and the benchmark score swings on evaluation design alone(opens the original post in a new tab)

1. 22:26
   论文追踪

   The paper points out that the reliability of LLM vulnerability repair benchmarks is affected by evaluation design.

   The paper "On the Reliability of LLM-Based Vulnerability Patching Benchmarks" points out that current automated vulnerability repair benchmarks based on large language models significantly distort reported performance. The authors organize overlooked pitfalls from three dimensions: at the agent level, prompts, tool availability, and detailed instructions inflate success rates without improving patch quality aligned with developers; at the framework level, permission errors, infrastructure defects, and timeout handling quietly lower or inflate scores; at the dataset level, bug reports and single PoC tests cannot determine whether a patch addresses the root cause or matches developer intent. The authors compiled 112 historical vulnerabilities from 84 open-source C/C++, Go, and Rust projects, each with PoC tests, regression tests, and additional developer tests that evaluate whether patches conform to the original developers' design principles. The paper thereby provides a guide for more rigorous, reliable, and reproducible evaluation practices.

Current attention57·Peak in the comparable period58（Oct 12 15:00）·Change over the comparable past 24 hours +19%