CWE-bench: shifting the focus of agent security from exploitation to defense

The researchers introduce a private safety evaluation benchmark, CWE-bench, which uses 120 held-out repository audit and repair tasks to measure agents' vulnerability defense capabilities, covering 73 CWE categories, eight languages, and all categories of OWASP Top 10 2025. Agents receive only the repository to be audited and a threat model description, and are not told the weakness type, location, or count; patches must be confirmed by a programmatic verifier to both block vulnerability exploitation and preserve original functionality, and the verifier was validated during development by a set of agent judges and manually reviewed for disagreements. The best agent achieves 68% pass@1 and 81% pass@4; analysis shows that agents with the same pass@1 differ markedly in retry gains and cost, no single agent leads across all languages or vulnerability categories, and task difficulty varies more with vulnerability category than with language. The authors state that as agents' attack capabilities improve, this benchmark can be used to measure whether their defense capabilities keep pace.