Trail of Bits says 1Password's AI patch benchmark is misleading

Trail of Bits questions 1Password's FLAWED vulnerability patch evaluation, arguing that sample selection, deliberately guided incorrect fixes, and the prohibition of compilation testing affected the headline conclusions. Their recalculation under a different set of experimental conditions yielded a higher proportion of blocking the given exploit, but they explicitly state that blocking this test does not equal a complete fix, and the two proportions cannot be directly compared. The article also discusses limitations of automated scoring; Trail of Bits collaborates with OpenAI on patch projects, creating a relevant conflict of interest. Here, the rebuttal's arguments are preserved, not treated as an independent ruling on the original report.