CyberChainBench: Can AI agents protect smart contracts from real on-chain vulnerabilities?

Researchers propose the CyberChainBench benchmark to evaluate LLM agents on three tasks in smart contract security: vulnerability detection, exploitation, and patch synthesis. The benchmark consists of 541 real exploit incidents from DeFiHackLabs, covering 9 EVM chains, with each case anchored to a specific block, and agents read code, trace transactions, and verify exploits on a mainnet fork in an isolated environment orchestrated by Harbor. Exploits are scored by their economic impact on the historical fork, and patches are validated by replaying historical attacks and normal transactions. Evaluation shows a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, and only 23.4% on patching; the best-performing Codex with GPT-5.5 obtained a total of $57.4M in exploit proceeds on the 200-case exploit set, at a cost of $2.39 per case.