ReproBAIT: Can coding agents build biological AI tools on their own?

The researchers propose the ReproBAIT evaluation to measure coding agents' ability to design, implement, and train published biological AI tools under restricted information conditions. The tested agents run in a secure sandbox without internet access, receive training data, fixed time and compute budgets, and limited information about the target tool, and the reproduced tools are scored on held-out data using tool-specific metrics. The evaluation covers the inverse protein folding tool ProteinMPNN, the all-atom generative diffusion tool RFdiffusion3, and a closed-source molecular design tool; across multiple leading closed-source and open-source coding agents, some selected tools were reproduced to a substantial degree, while reproduction of highly complex tools was limited. Because open-source tools may already be in model pretraining corpora, the authors designed a memorization test method and found moderate evidence consistent with memorization, with differences across coding agents. The authors therefore caution that tool developers should be more careful when publicly releasing tool information, as reconstruction based on such information could undermine the commercial value of tools or render dual-use controlled-access policies ineffective.