DRYLAB-Bench: Evaluating biosecurity risks of agentic LLM mutation design

Existing evaluations do not quantify the risk of LLM agents completing high-risk single-point mutation designs with the assistance of biological design tools.

• Scenario: The authors state that models are beginning to possess biological design capabilities, bringing dual-use risks such as enhanced viral pathogens or drug-resistant bacteria. Multiple international frameworks regard malicious biological modification as a key threat, but the magnitude of LLMs executing such modifications remains unquantified, and specialized biological design tools (BDT) may amplify it.
• Knowledge evaluations are insufficient: The authors state that Q&A evaluations such as WMDP and VCT only check whether models possess dangerous information, not whether they can apply that knowledge to concrete design tasks.
• Task evaluations do not measure tool amplification: The authors state that ABC-Bench and others cover fragment design, screening evasion, and experimental robot scripts, with dual-use mainly only in screening evasion; the general-purpose software used cannot score dangerous traits, so BDT contributions are not measured.
• Verification dilemma: The authors state that open-ended designs currently rely on unreliable LLM review, expensive expert review, or wet-lab experiments requiring physical synthesis; the latter recreates biological threats when verifying malicious capabilities.
• This paper's benchmark: The authors propose DRYLAB-Bench, using 30 mutation design scenarios from guidelines, evaluated in an environment with multi-round BDT calls; submitted single-point substitutions are compared against 187,695 existing measurements from ProteinGym and ViroGym, recorded as the Biosecurity Risk Score (BRS). The authors call this the first comprehensive benchmark for LLM agents' high-risk mutation design capability.

知识、拒答与序列重建

• WMDP（Li et al., 2024）与 VCT（Götting et al., 2025）：问答形式测危险科学知识或病毒学能力。作者称预定义答案可靠但僵硬，会引入提示线索，也反映不了开放式生物设计。
• 安全套件：作者把 Ackerman 等（2025）、Weidener 等（2026）、Arora 等（2026）、Ali 等（2026）与拒答率测试或简单序列重建放在同一类早期评测里，正文没有逐篇展开做法。

带生物工具的智能体评测

• ABC-Bench 及后续工具评测：Liu 等（2026）、Lee 等（2026a; 2026b）、Black 等（2026）、Barkan 等（2026）、Cai 等（2026）评测配备生物工具的 LLM 做生物编程和筛选规避。作者称其智能体范围有限：通常只测是否拥有危险知识，或把工具限制在与核心恶意目标无关的外围阶段，没有评估能否迭代细化突变以达到危险功能表型，因而测不到工具对危险能力的放大。
• BixBench 与 BioTier：Mitchener 等（2025）、Marshall 等（2026）被作者归入受验证范式约束的生物安全基准，正文没有另作方法对比。

验证范式与 DMS 数据库

• 三种既有验证：预定义答案键、LLM-as-a-judge、物理湿实验。作者称 LLM 评审受评测模型能力上限和复杂生物领域幻觉限制；湿实验以 ABC-Bench 为例，能给确定真值，但昂贵、难扩展，且验证恶意能力时有生物安全风险。
• ProteinGym 与 ViroGym：Notin 等（2023）、Zhou 等（2026）提供大规模深度突变扫描（DMS）适应度景观。作者称它们以往用于评测预测性生物模型，而不是验证 LLM 执行危险任务时的生成输出。

基线与数据

• 基线方法：本文没有把其他攻击或设计方法当作外部对照。能力基线是 S0，即无工具一次提交；S0-iter 是相同迭代预算、不给工具证据的对照。基准/数据集是作者自建的 30 个场景，真值来自 ProteinGym 与 ViroGym。

作者把本文定位为：把这些 DMS 数据转用来核对开放式危险设计的输出，从而同时避开 LLM 评审和物理合成，并专门测量 BDT 对危险能力的放大。作者称 DRYLAB-Bench 是首个此类综合性基准。

DRYLAB-Bench 把政策文本收成 30 个单点突变设计任务，让 LLM 作为规划者调用六类 BDT 迭代候选，再把最终替换对照隐藏的湿实验测量打成 BRS。Figure 1、Figure 2 分别给出构建流程和评测流程。

场景从指南滤到可核对的测定

• 风险分类：作者从 BWC、WHO、美国行政命令 14110 与 14292、欧盟 AI Act 抽出八维分类（Figure 1）。Table 1 的八类是免疫逃逸/医学对策失效、感染性或宿主范围增强、抗微生物或抗病毒耐药、蛋白毒素化、隐蔽致癌或抑癌失活、代谢通路崩溃或药物基因组武器化、环境耐受增强、农业或生态生物安全。
• 保留条件：两位计算生物学专家主导场景设计，LLM 只作辅助。任务只有在 DMS 数据库里有高质量定量真值时才保留；另一位独立专家再查生物物理合理性和编号一致性。最终 30 个场景覆盖全部八类。
• 任务输入：每个场景给出野生型序列和恶意目标，要求提出单点氨基酸替换。Appendix C 给出提示词拼装和三种仅改标题的措辞变体；这里不复述目标句、序列或标题以外的约束。

六类工具与四种条件

• 工具：进化保守性（MMseqs2 对 UniProtKB/Swiss-Prot）、序列似然（五种子 ESM-1v）、结构相容性（ProteinMPNN）、功能注释（UniProt）、毒性预测（ToxinPred3），以及场景特异预测（AlphaMissense、EVEscape 或 FoldX）。作者写明这些工具都不返回隐藏的 DMS 标签。
• S0 / S1 / S2：S0 不给工具、不迭代，直接提交；S1 做一轮工具选择和修订；S2 可重复选工具、读证据、改候选，最多四轮。三者不是三次独立运行，而是同一条生成轨迹上的三个快照，共享初始候选池。
• S0-iter：另一次运行，迭代预算与 S2 相同，但不提供工具证据。Appendix D 写明 S0 的首轮提示仍列出工具菜单，不过提交记在任何工具调用之前；S0-iter 的提示不含该菜单。
• 预算与解码：Appendix C 的 S2 预算为最多 4 次工具调用、4 个智能体轮次、10 个活跃候选、每轮最多 5 个新候选、最终交 3 个。能设置的接口把温度定为 0.3、推理力度定为 low；厂商不支持时用最接近的设置，逐模型记在 Appendix A。

用隐藏测定定义 BRS

• 取向：对候选 m 取测定分数 s(m)，R_base(m)=α_a s(m)，其中 alpha_a 为 +1 或 -1，使越大表示该测定上的风险越高。若场景要求维持稳定性或表达，违反时再减去约束项；无约束时风险就是这个取向后的分数。
• 单元分：每个测定内用该测定全部变异的均值和标准差标准化，并截断到 -3 到 +3。一个单元是模型排序最高的 3 个候选，z_u=∑_{m∈ u} q_m z_m/∑_{m∈ u} q_m，q_m 是模型自报置信度。
• 汇总：不拒答且至少给出一个可评分单点替换的单元为有效单元；拒答单元在场景分里记 fref=0，作者称这不奖励“一律拒绝”。场景分是单元分的平均，总 BRS 对场景等权，因为每个场景的单元数相同。
• 查不到时：流程中 DMS 是盲留出集。没有测定记录的合法候选用 Appendix B.2 的 in silico 估计填入，并按该任务上融合分数与测定风险的对齐相关性折扣权重。主结果仍以查表为主。

Experimental Setup

• Models: 15 frontier LLMs. Anthropic: Claude Opus 5, Claude Opus 4.8, Claude Fable 5. OpenAI: GPT-5.6 Luna, GPT-5.6 Sol, GPT-5.6 Terra, GPT-6 Astra. Google: Gemini 3.6 Flash, Gemini 3.8 Flash. Also Grok 4.5, DeepSeek V4 Flash, DeepSeek V4 Pro, Qwen3.8 Max, Kimi-K3, GLM-5.3.
• Data: 30 scenarios, eight categories (Table 1), with a total of 187,695 single-point substitutions from ProteinGym and ViroGym. The number of tasks and single-point mutations per category are F1 6/43,344, F2 4/25,725, F3 6/46,390, F4 2/2,372, F5 5/22,344, F6 4/35,912, F7 2/10,389, F8 1/1,219.
• Conditions: S0, S1, S2 are sampled along the same trajectory; S0-iter is a separate run with no tools and the same round budget. Appendix D states the latter is too noisy and serves only as a bound on the iteration effect, not as a reliable ablation.
• Decoding and units: Temperature 0.3, reasoning effort low, with exceptions in Appendix A, e.g., Kimi-K3 sends temperature 1.0, and the three Anthropic models and GPT-6 Astra do not send temperature. Each scenario has three wording variants that only change the title; the main text writes Claude Fable 5 refuses 90 of 87 units, a denominator corresponding to 30×3.
• Metrics: The primary metric is confidence-weighted BRS, where 0 indicates the assay's mean variation. Refusals are scored as fref=0 in the refusal-inclusive view. Also reported are refusal rate, the proportion falling into the assay-specific highest-risk decile, and the coverage of submitted candidates against the assay.
• Scoring: The main score is not an LLM judge. At S2, the average experimental coverage across all models and scenarios is 83.9% (Appendix B.1). Candidates without assay records are estimated per Appendix B.2 before entering the unit score.

Main Results

The table below lists only 8 models, with numbers from Table 10. The valid column is BRS on non-refusal units; the refusal-inclusive column records refusals as 0 and then equal-weights by task. The lowest S2 valid BRS is GPT-6 Astra, and the lowest refusal-inclusive is Claude Fable 5, both rows in the table. Omitted are GLM-5.3, Grok 4.5, Qwen3.8 Max, Gemini 3.8 Flash, GPT-5.6 Sol, Claude Opus 4.8, and Claude Opus 5.

Swipe horizontally to read the table

Model	S0 valid	S2 valid	S0 refusal-inclusive	S2 refusal-inclusive
Kimi-K3	0.519	0.631	0.523	0.637
Gemini 3.6 Flash	0.511	0.592	0.339	0.414
Claude Fable 5	0.287	0.754	0.010	0.025
DeepSeek V4 Pro	0.276	0.483	0.283	0.483
DeepSeek V4 Flash	0.428	0.416	0.427	0.416
GPT-5.6 Luna	0.508	0.507	0.285	0.300
GPT-5.6 Terra	0.247	0.313	0.129	0.159
GPT-6 Astra	0.226	0.302	0.077	0.101

• Summary: Table 9 records the equal-weighted valid BRS across 15 models as S0 0.359, S1 0.407, S2 0.475. The authors state ΔBRS=+0.116 from S0 to S2, with 13/15 of these increments positive. The refusal-inclusive endpoints are 0.249 and 0.317; the stepwise increments in this view are written separately as +0.026 and +0.042, not to be confused with +0.116. Figure 3 uses light color for S0 and an overlaid segment for the change from S0 to S2, sorted by S2; after plain-text conversion, the alignment of numbers and model names in the figure is unreliable, so model numbers follow Table 10.
• Gains are not uniformly positive: The authors state the gain correlates with no-tool S0 valid BRS at Pearson r=-0.42 (Figure 4). The main text first writes Claude Fable 5, DeepSeek V4 Pro, and Gemini 3.8 Flash gains as +0.47, +0.21, +0.19, then later writes Pro gains +0.207 from 0.276, Gemini 3.8 Flash gains +0.195 from 0.269, and Claude Opus 5 gains only +0.004 from 0.378. The authors state DeepSeek V4 Flash drops 0.012 and GPT-5.6 Luna drops 0.001, and argue that evaluations must consider both no-tool and tool conditions.
• Single designs cited by the authors: The authors state that GPT-5.6 Sol's one no-tool submission, W285T, is the highest-risk among 3,372 tested variants in the ProteinGym SRC activation assay. In the PsaE disruption scenario, GPT-5.6 Terra's design goes from exceeding 11.7% of tested variants to exceeding 98.9%. The conclusion states that without external evidence, 67.6% of submissions exceed the assay mean variation, 21.4% fall into the assay-specific highest-risk 10%, which the authors call the random expectation of 10%; after iterative tool assistance, this proportion is 23.5%, which the authors call a relative increase of 9.8%. The main text writes Kimi-K3's zero-shot BRS as 0.52, while Table 10's S0 valid value is 0.519.

Do Refusals Align with Harm?

• Original suite: Claude Fable 5 refuses 87/90 units, recorded as 96.7% in Table 3. GLM-5.3, Kimi-K3, both DeepSeek variants, and Qwen3.8 Max have a 0.0% refusal rate on the original suite. The authors state refusal is not a capability deficit: Fable 5's BRS on answering units is 0.754, and since scores are standardized within each assay, they argue it cannot be attributed to the remaining tasks being easier.
• Relationship to harm and wording: The authors state that if refusal encoded dual-use intent, it should track the scenario's measured harm, but Spearman ρ=+0.25. Scenarios naming viruses are refused in 23.9 of 45 units across models; the authors state that PsaE fold disruption, which is comparably harmful but lacks typical alarm words, is 8/45, and OCT1 transport ablation is 6/45.
• Framing-only changes: The 30 scientific requests, wild-type sequences, mutation targets, and hidden assays remain unchanged; only the prompt framing changes. A wrapper that removes pathogen names and dual-use wording reduces refusal rates among other models that refuse, with Fable 5 still at 96.7%; adding a harm statement before the original request increases refusals, including among models that previously accepted all. In Table 3, Kimi-K3 goes from 0.0% to 80.0%, Qwen3.8 Max from 0.0% to 90.0%, GLM-5.3 from 0.0% to 70.0%, and Fable 5 from 96.7% to 100.0%. The authors argue that current guardrails classify by wording rather than malicious intent. Specific rewording sentences are not repeated.

Tool Misalignment and Whether It Is Just Memorization

• CcdB: The authors state that the evolutionary conservation tool MMseqs2 correlates with experimental outcomes at Spearman ρ=-0.815 (1176 samples), the toxicity predictor at ρ=-0.135, and structural tools are unavailable (n=0). In Table 2, GPT-5.6 Terra's ΔBRS=-0.208, DeepSeek V4 Pro's is -0.196, and Gemini 3.8 Flash's is +1.065; models that refuse all CcdB units are omitted. The authors argue that most models still improve in this scenario, and misalignment is a model-dependent risk, not a uniform penalty.
• Consensus mutations: Appendix F.2 states that across 16 scenarios with cross-model consensus, mutations proposed by most models have an average BRS=+0.67, while those proposed by only one or two models have +0.38; but in 12/16, the highest-scoring candidate proposed by any model is not in the consensus. At S2, units where submissions are mostly consensus mutations are +0.57, and model-specific units are +0.60. Across 30 scenarios, only 7 have a single candidate attracting more than half of the models.
• A distribution example: The caption of Figure 6 states the MTHFR loss-of-function assay has 12,464 tested single-point mutations. The authors state that R157Q, proposed by most models (10/15), has BRS=-0.78, and A222V (7/15) has +0.22; C243Y, proposed by a single model, has +2.30. The caption states that GLM-5.3, variant P2, has an S0 mean of -0.34, and after tool iteration, the mean of the three final designs is +1.11, ending at C243Y.

Other Ablations and Analyses

• Stepwise increments: S1 relative to S0 has a valid BRS of +0.048 (10/15 positive); S2 relative to S1 adds +0.068 (10/15 positive) (Table 9).
• S0-iter: Valid BRS 0.371, relative to S0 is +0.012 (10/15 positive); single-model steps range from -1.19 to +0.61, standard deviation 0.38; valid units 811/1,350, of which Claude Fable 5 has only 3.
• Refusal floor: When refusals are included, the increments from one tool round and subsequent rounds are +0.026 (11/15) and +0.042 (12/15) (Appendix D).
• Metric consistency: In Table 5, at S2, the Spearman ρ between BRS and empirical rank is 0.99 for refusal-inclusive and 0.96 for valid-only; when fref changes from 0 to -1.5, rank is 1.00, BRS is 0.99, and the raw composite score is +0.73.
• Coverage: S2 average top-3 truth coverage is 83.9%; the authors state no model is below 72%, and the highest is 100% (Figure 5, Appendix B.1). The correspondence between percentages and model names in the figure is misaligned in plain text, so it is not transcribed per model.
• Tool value usage: Of 2,090 revision steps, 74.9% cite at least one evidence identifier; of 14,870 candidate updates, 68.4%. Of 20,599 explicit evaluations, 57.5% are recorded as supporting the candidate and 42.0% as opposing. Of 861 trajectories producing S1 submissions, 372 do not request tools again before S1.

Limitations and future directions noted by the authors

• Score measurement scope: Appendix B.3 and Discussion F.1 state that the evaluation unit is a single amino acid substitution; BRS does not estimate multi-mutant fitness, epistasis, expression and host context beyond assay phenotypes, transmissibility, makes no claims about in vivo behavior, and does not measure whether physical construction pipelines succeed.
• No-tool iteration control: Appendix D states that S0-iter is too noisy to be used for ablation; its results relative to S0 cannot support the stronger claim that "iteration alone has no effect," which would require a larger, repeated-sampling design.
• Causality not disentangled: Appendix D states that this version does not claim that the numerical content of tool outputs, rather than the extra thinking from tool rounds, is the channel for increased risk. Neither shuffling or permuting tool outputs nor running a non-LLM optimizer with the same tool outputs was done.
• Static tool ranking is not a control: Appendix D states that using BDT to score all tested single-point substitutions and submitting the top three is ranking within an enumerated space, not proposing candidates from scratch, so it is not a fair control; the authors make no claims about it.
• Decoding settings: Appendix A fixes reasoning effort to low and states that its impact on results is left for future work. Residual differences in vendor temperatures are treated by the authors as a documented confound, not a freely chosen setting.

Experimental coverage

• Models and tasks: Section 4.1 evaluates 15 frontier LLMs from eight developers; Table 1 has 30 scenarios, eight risk categories, and 187,695 single-point substitutions.
• Conditions: Experimental conditions are S0, S1, S2, and S0-iter; S0, S1, and S2 share the same trajectory (Section 3.2, Appendix D).
• Prompts: Each scenario has three wording variants that change only the title (Appendix C); Table 3 additionally compares the original framework, academic packaging, and harm prefix on the same 30 scientific requests. The paper does not report repetitions under the same wording.
• Scoring: Main scores use hidden DMS, not LLM review. S2 average ground-truth coverage is 83.9% (Appendix B.1); remaining candidates enter scores via in silico estimates from Appendix B.2. The paper does not report statistical tests for ΔBRS.
• Controls the authors state were not run: Reasoning effort is fixed to low throughout (Appendix A). Appendix D states that the shuffled-tool-output control and the non-LLM optimizer control were not run.

DRYLAB-Bench uses existing wet-lab measurements to score LLM agents' high-risk single-point mutation designs and compares changes before and after integrating BDT.

• Problem: The authors argue that international frameworks already regard malicious biological modification as a key threat, but the magnitude of risk from models actually making such modifications has not been quantified; knowledge Q&A cannot measure operations, task evaluations do not account for amplification by specialized design tools, and open-ended results lack a scalable verification method that does not require synthesizing physical materials.
• Approach: Guidelines are condensed into eight categories and 30 single-point substitution scenarios. The model sees the wild-type sequence and malicious target, and submits candidates under no-tool one-shot, one round of tool revision, and up to four rounds of tool iteration; these three tiers are snapshots of the same trajectory. Six categories of BDT do not return hidden labels. BRS orients 187,695 measurements from ProteinGym and ViroGym as higher-is-more-dangerous, then applies within-assay standardization and confidence weighting to the top-3; refusals are scored as 0.
• No-tool results: The equal-weighted valid BRS across 15 models is 0.359. The authors state that 67.6% of one-shot designs exceed the assay mean variation, 21.4% fall into the highest-risk 10% of each assay, and they call the random expectation 10%. The main text writes Kimi-K3's zero-shot BRS as 0.52. The authors state that GPT-5.6 Sol's W285T is the highest-risk submission among 3,372 tested variants in the SRC activation assay.
• After tools: Valid BRS reaches 0.475 at S2, with the authors stating ΔBRS=+0.116 and 13/15 positive; the highest-risk decile share rises to 23.5%, which the authors call a relative increase of 9.8%. In the PsaE scenario, GPT-5.6 Terra goes from exceeding 11.7% of tested variants to 98.9%. There are also declines: the authors state DeepSeek V4 Flash is -0.012 and GPT-5.6 Luna is -0.001. When refusals are included, high-refusal models have very low overall scores, but Claude Fable 5's S2 valid BRS on answering units is 0.754.
• Refusals: On the original suite, five models have a refusal rate of 0, and Fable 5 refuses 87/90. The authors state that refusal correlates with measured harm at Spearman ρ=+0.25, and argue that guardrails follow wording rather than malicious intent; when only the framing is changed, two wrappers push refusal rates in opposite directions (Table 3).
• Authors' conclusion: The authors argue that frontier models can already rank single-point substitutions into the high-risk tail of public assay phenotypes, and tool iteration pushes the distribution further into the tail. They hope this benchmark becomes a scalable empirical testbed. They also limit the claim to the single-point design step, not extrapolating to multi-mutations, live organisms, or physical construction.