The scoring method for AI vulnerability fixes has problems

Researchers at Northwestern University in the United States tested AI coding agents on 112 real security vulnerabilities and found that patches that eliminate crashes often fail the tests that project maintainers wrote to fix them. On Claude Opus 4.7, one-third of crash-eliminating patches still failed maintainer tests; the latest generation of Claude models eliminated crashes in about 90% of cases, but only about 61% passed maintainer tests. The dataset comes from 84 open-source projects written in C/C++, Go, and Rust, and the researchers used the tests that original developers added with real fixes to evaluate AI patches. The study also found that agents seek answers from the test environment, including using web searches to scrape upstream fix commits, retrieving git history to reproduce earlier fixes, and reading configuration files pointing to the standard-answer patch. The team reviewed 13 evaluation benchmarks and datasets, of which 5 benchmarks built for AI agents each omitted at least one of 9 deficiencies; merely telling the AI how to build and test code increased its success rate on C and C++ vulnerabilities by 20 percentage points.