AISI evaluation: Can AI coding agents rebuild biological AI tools from papers?
Can AI coding agents rebuild biological AI tools from a paper?
The UK AISI chemical-biology team, in collaboration with SecureBio, built the ReproBAIT evaluation to test whether AI coding agents, given only public information such as papers and training data, can reproduce existing biological AI tools (BAITs). The evaluation was conducted in a sandbox with a single GPU and no network, covering the open-source ProteinMPNN and RFdiffusion3 and the closed-source molecular design tool CMD, and tested four coding agents: Claude Opus 5, GPT-5.6 Sol, Kimi K3, and Qwen3.8-2.4T. Results showed that the simplest tool could be successfully reproduced within 4 hours and with a compute budget of less than $20, and reproduction costs for the two simpler tools were typically between $4 and $20; Claude Opus 5 reached 96.5% fidelity on ProteinMPNN when given the full paper, but no agent reproduced the most complex tool, RFdiffusion3, within the time budget, with each attempt costing $102 to $301.
AISI's ReproBAIT evaluation provides reproduction cost and fidelity figures, showing that not disclosing code alone does not equal a safe release, and that controlling underlying biological data may be a more effective direction.