When do biological reasoning models actually use their biological inputs?

Does a biological reasoning model's benchmark accuracy indicate that it uses the biological foundation model representations it receives?

• Scenario: Such models feed foundation model representations of genomes, proteins, or cells, along with text such as genes, pathways, functional annotations, or expression-ranked gene lists, into a language model for prediction. The authors argue that when annotations already support the correct answer, benchmark improvements need not equal greater use of these representations.
• Existing practice cannot separate the causes: The same biological object often has multiple representations, and the information can overlap. The authors argue that a small contribution from one pathway could mean another pathway already supports the prediction, the input has limited predictive value, or the model does not use an input that could be predicted; benchmark scores alone cannot distinguish these.
• This paper's approach: The authors use four tests—perturbation, evidence conflict, linear probes, and reasoning traces—to examine input use in six models on DNA, protein, and single-cell tasks, and compare post-training checkpoints and model scale.
• The second question to answer: When post-training or scaling up the model improves accuracy, do the performance contributions of foundation model representations or differentially expressed genes also grow; whether the generated reasoning connects to the given biological evidence, and whether auxiliary supervision can increase sensitivity to the sequence.

The paper organizes related work by biological reasoning models, multimodal evaluation, probes and generated rationales, and positions itself on "whether predictions depend on biological input content," rather than reporting task scores again.

• Biological reasoning models: ChatNT uses Nucleotide Transformer representations, BioReason combines Evo2 with gene and pathway text (de Almeida et al., 2025; Brixi et al., 2026; Fallahpour et al., 2025). Prot2Text-V2 uses ESM2 plus protein name and taxonomy text, BioReason-Pro uses ESM3 plus GO-GPT predictions and InterPro annotations (Fei et al., 2025; Hayes et al., 2025; Fallahpour et al., 2026). Cell2Sentence-Scale writes transcriptomes as expression-ranked gene sentences, CellWhisperer uses continuous transcriptome representations (Levine et al., 2024; Schaefer et al., 2025).
• Systems in the same direction and existing evaluations: Other protein models connect biological representations to language for retrieval, annotation, or generation (Queen et al., 2025; Su et al., 2026; Xu et al., 2023; Liu et al., 2024; Jararweh et al., 2025; Wang et al., 2024; Char et al., 2025). The paper states that existing evaluations often emphasize task performance, scale, and post-training (Fesser et al., 2026; Wu et al., 2025; Zhao et al., 2026; He et al., 2025; Dip et al., 2026), and instead distinguishes between target predictability and whether the model uses that input.
• Multimodal use and shortcuts: The introduction uses shortcut learning to explain that benchmark scores can come from shortcuts rather than signals (Geirhos et al., 2020; DeGrave et al., 2021). Vision-language models can exploit language priors, be insensitive to relevant visual content, or improve benchmark scores without increased cross-modal interaction (Goyal et al., 2017; Agrawal et al., 2018; Cadene et al., 2019; Hessel and Lee, 2020; Frank et al., 2021; Asadi et al., 2026; Gao et al., 2026). Counterfactual inputs and controlled perturbations are used to test whether predictions respond to a modality (Niu et al., 2021; Thrush et al., 2022). Representation alignment, composition, and decomposition are listed separately as background (Liang et al., 2024; Dufumier et al., 2025; Huh et al., 2024, etc.), without pointing out deficiencies paper by paper.
• Probes and rationales: Linear probes test whether representations can predict targets (Alain and Bengio, 2017; Hewitt and Liang, 2019). Subsequent work points out that probe performance should be evaluated separately from downstream behavior (Ravichander et al., 2021; Elazar et al., 2021). Plausible explanations need not be features that influence predictions (Jacovi and Goldberg, 2020; DeYoung et al., 2020; Wiegreffe et al., 2021; Turpin et al., 2023).

Baselines and data: Simple prediction baselines in experiments are linear probes, majority vote over prompt text, and the most common base pair in training. Data are in Table 1: KEGG disease prediction queries, ChatNT three tests, reviewed human proteins with temporal holdout, Prot2Text-V2 test split, five single-cell atlases.

The authors state that the special aspect of biological reasoning models is that multiple representations of the same protein, genome, or cell can be simultaneously available, so the question is which one contributes to downstream predictions.

The authors organize the tests into four research questions, with the core objects being the continuous representations Zrepr and biological texts Ztext received by six biological reasoning models. Predictions and optional rationales are written as (ŷ,R̂)=f_θ(q,Z), i.e., producing an answer, and sometimes a reasoning trace, given query q and biological input Z.

Problem setup

• Input: q specifies the task and answer format. Z comes from protein sequences, genomic sequences, or single-cell transcriptomes. The authors define usage as: changing the task-relevant biological content in Zi, fixing q and other inputs, then observing prediction sensitivity and task performance.
• Evidence conflict: concatenate Zi from one query with the rest of the inputs from another query, so that the two sources of evidence support different predictions, then observe which prediction the model follows.
• Underutilization: a simple baseline can predict the target from Zi, but changing its task-relevant content has little effect on the reasoning model's performance; when possible, also control for predictive information in other inputs. The authors use this definition to separate "input is uninformative" from "input is informative but contributes little."

Four tests

• Perturbation: shuffle or resample one input stream and recompute the representation. For DNA, shuffle both reference and variant sequences, preserving length and base composition but not edit positions. On the protein side, shuffle the amino acid sequence and recompute only ESM3 or ESM2; GO-GPT, InterPro, biological names, and other text remain from the unshuffled source. On the cell side, shuffle the order of gene sentences, resample from all expressed genes, or permute counts across gene identities before recomputing CellWhisperer representations.
• Evidence conflict: use inputs from two unmodified queries. For example, BioReason uses A's Evo2 representation paired with B's gene and pathway text. The authors state that ChatNT's evaluated queries have no query-specific biological text that could conflict with DNA, so this test is not performed for it.
• Linear probe: when the performance contribution is small, fit linear probes on the base model outputs and on the representations after the reasoning model's projector, and compare with a text-only predictor.
• Trace and auxiliary supervision: check whether the trace or prompted rationale agrees with the input, changes with perturbation, and is consistent with the final prediction. Models that do not produce traces are prompted to point out biological evidence supporting the prediction. RQ4 replaces BioReason's long traces with short targets indicating the reference and variant bases at the edited position, while disease answers are retained.

Checkpoints and DEG controls

• Post-training: RQ2 compares BioReason, BioReason-Pro's SFT and RL, and 42 BioReason checkpoints under the same perturbation; C2S-Scale compares 2B and 27B.
• DEG: use Wilcoxon rank-sum test per atlas. A gene is recorded as a DEG of that type when log2 fold change is at least 1.0 and Benjamini–Hochberg corrected p-value is below 0.05. Removal proportions include the strongest DEGs, weakest DEGs, random genes, and non-DEGs with the closest expression values.

Scoring and probes

• BioReason: generate disease mappings to 37 labels; mismatches are counted as errors. RL checkpoints have 1.5% mapping failures on wild-type queries.
• ChatNT: compare the log-softmax of the first token of " Yes" and " No" at the final prompt position.
• BioReason-Pro: use cafaeval according to its CAFA configuration, reporting information accumulation weighted Fmax. Conflicts are judged by whether the GO sets after ontology propagation satisfy required or forbidden entries, determining which side is followed.
• Prot2Text-V2: compute BioBERT BERTScore F1 with UniProt functional text, using model dmis-lab/biobert-large-cased-v1.1, layer 24. A side is considered followed only if the score difference exceeds 0.005.
• Cell models: C2S-Scale's cell sentences are truncated to 1,000 genes by descending expression, with options being all cell types in that atlas. CellWhisperer scores by per-token log-likelihood of candidate answers; the cell type metric is 1 minus the perplexity quantile.
• Probes: logistic regression, L2, feature standardization. Genomic-dependent queries are split five-fold by genome; proteins are split five-fold by InterPro family, with AUROC averaged equally within families.

实验设置

• 被测系统：BioReason（SFT、RL及42个后训练检查点）、ChatNT、BioReason-Pro（SFT、RL）、Prot2Text-V2、C2S-Scale的2B与27B、CellWhisperer。除第4.4节辅助监督外，作者称使用已发布检查点且不改权重。
• 规模：BioReason为1,449条KEGG查询、708个基因组、37个疾病标签，其中165条是基因组依赖查询。ChatNT为4,578条（供体2,000、受体2,000、TATA启动子578）。BioReason-Pro为14,102个reviewed human proteins，另有91蛋白时间留出。Prot2Text-V2扰动实验为3,917个蛋白。单细胞为五个atlas、4,846个细胞；理由分析2,000个细胞。
• 指标与不确定性：疾病、剪接/启动子和细胞类型用准确率；蛋白功能用IA加权Fmax；Prot2Text-V2用BioBERT BERTScore F1；CellWhisperer用1减perplexity分位数。图中误差线为95% CI。辅助监督报告三次训练种子的标准误。
• 探针子集：165条查询的提示文本相同，但基因组和疾病标签不同。蛋白探针为540个蛋白、140个InterPro家族，家族注释本身不能决定酶活性、细胞器定位或DNA结合。
• 判定：疾病和细胞类型按标签映射或选项匹配，不是单独的LLM裁判。Prot2Text-V2的描述用BERTScore对照UniProt功能文本。

主结果

Swipe horizontally to read the table

模型	扰动	完整输入	扰动后	出处
BioReason RL	DNA打乱，文本保留	准确率0.847	0.847	Table S2
ChatNT	序列打乱，splice donor	0.988	0.502	Fig. 2b
BioReason-Pro RL	ESM3打乱，GO-GPT与InterPro保留	Fmax 0.855	0.856	正文、Fig. 6c
Prot2Text-V2	去掉蛋白名与分类单元	未报告	BERTScore F1下降0.177	Fig. 2d
C2S-Scale 27B	基因句顺序打乱	准确率0.446	0.351	Fig. 2e
CellWhisperer	immune1内置换基因计数	1−分位数0.98	0.53	Fig. 2f

• 准确率不变时答案仍可改变：Table S2把BioReason RL两端都记为0.847，净变化为+0.001，同时有101/1,449个答案改变（7.0%）。BioReason-Pro有46.8%的蛋白GO集合改变，平均Jaccard为0.889（Table S3）。作者称去掉基因名时BioReason准确率下降最大（Fig. 2a），正文未给出该条件的数字。
• 另外两个ChatNT任务与CellWhisperer的lung：splice acceptor从0.987到0.502，TATA启动子从0.971到0.577。作者称打乱序列的正确标签定义不清，但该结果支持回答相对原序列发生变化。lung上1−分位数从0.84到0.25；C2S-Scale 27B重采样后为0.406（Fig. 2e、f）。
• 证据冲突与探针：BioReason有97.9%与文本来源B一致（Fig. 3a），BioReason-Pro有99.7%与GO-GPT一致（Fig. 3b）。Prot2Text-V2有97.0%偏向序列（Fig. 3c），C2S-Scale 27B有93.7%跟随基因句（Fig. 3d），CellWhisperer有96.1%给表示支持的答案更低perplexity（Fig. 3e）。165条查询上，Evo2输出与投影后表示的探针为0.736和0.735，文本预测器为0.708（Fig. 4a）。三种蛋白性质上，ESM3探针为0.766、0.898、0.830，投影后为0.713、0.921、0.898，BioReason-Pro为0.729、0.890、0.736（Fig. 4b）。作者称这些表示含任务信息，但总体性能贡献有限。

后训练与规模

• SFT对照RL：全部生物文本在场时，BioReason SFT为0.823对打乱后0.814，RL为0.847对0.847（Fig. 6a）。BioReason-Pro的Fmax从SFT的0.829到RL的0.855；换成打乱序列的表示后，SFT为0.829到0.833，RL为0.855到0.856（Fig. 6c）。作者称RL提高了任务表现，但没有增大Evo2或ESM3的性能贡献。
• 42个检查点：模型规模、SFT与RL的LoRA秩和RL时长变化时，准确率变化，原ZEvo2相对打乱序列表示的优势没有稳定变大（Fig. 6b）。作者称后训练可以提高疾病预测准确率，包括基因组能区分标签的查询，而不增加ZEvo2的性能贡献。
• C2S-Scale从2B到27B：未改动细胞句的平均准确率从0.366到0.446。100%去掉DEG时，2B在最强DEG与表达相近非DEG上为0.140和0.363，差值为0.223；27B为0.127和0.446，差值为0.319（Fig. 5）。作者称27B在五个atlas的全量DEG去除上，DEG特异效应都更大。

轨迹、理由与辅助监督

• BioReason写出的碱基：111条单碱基替换留出查询上，RL与SFT写出正确碱基的比例为7.2%和3.6%（Fig. 7a、b；图中为8/111与4/111）。RL在全部145条回答中都写成替换且为T>C，序列被打乱时也如此。作者称SFT对序列有部分响应，不总是写出同一个突变。
• 功能被轨迹提到但未进入预测：53个BioReason-Pro轨迹中，23个在最终预测前用非InterPro条目名的措辞说出被测性质，但全部53个最终响应都省略该性质对应的GO条目（Fig. 7c为例）。C2S-Scale 27B理由引用63.4–67.8个基因，62.6–84.7%为核糖体蛋白基因；全部引用的平均DEG精度为10.63%，低于随机抽取参照，前五个引用则有富集（Fig. S3）。452条点名atlas类型的错误注释中，对预测类型DEG的精度为24.2%，括号中的随机抽取参照为9.9%（Table S6）。加入理由请求后，汇总准确率从0.45到0.41（Table S7）。同一模板下2B没有生成含Rationale的回答。
• 辅助监督很少转到疾病预测：257-bp窗口下，编辑碱基对准确率在完整DNA上为0.671，打乱后为0.122；2,048-bp为0.531和0.099，无辅助监督时为0.080（Fig. 8b）。Table S9记录的下降分别为0.549和0.432，疾病预测下降为0.021和0.028。218条训练中见过的基因组上为0.691和0.606；72条未见基因组为0.148和0.106，始终预测最常见训练碱基对为0.111（Table S10）。作者称改进集中在训练中出现过的基因组。

其他消融与分析

• 时间留出91蛋白：正文并列给出Fmax 0.685与0.682，该句未写明哪一个对应完整输入；作者称只解释留出集内部对比。
• PRNP占165条中的98条，探针准确率0.855，文本预测器0.708。
• GO NOT：106个冲突中98.1%与GO-GPT和InterPro一致（Fig. S1）。
• 七种理由问法、200个细胞：DEG精度10.4%–13.3%，原问法11.2%，细胞句中DEG占16.2%；one-shot准确率从0.40到0.18（Table S5）。
• 基因符号到疾病的多数票，测试集准确率0.932；91个基因都出现在训练中（附录B.5）。
• 附录B.5称，训练提示是否总含可预测文本与输入使用的跨模型对照只是观察；模型的架构、任务、数据和规模不同，不能隔离提示内容的作用。

Limitations and future directions noted by the authors

• The evaluation covers six biological reasoning models, specific tasks, and biological inputs; the authors believe the findings may not generalize to other architectures or biological modalities (§5).
• Some analyses shuffle sequences, which may move representations out of the natural sequence distribution; the authors state that evidence conflicts composed of unmodified inputs support the same conclusion (§5).
• Linear probes show that the target can be predicted from the representations given to the reasoning model, but do not indicate how a language model could use that representation (§5).
• The authors propose future evaluation of more models and, while fixing other training settings, examining training prompt variants within the same model (§5).
• Another proposed direction: reinforcement learning objectives that, in addition to final accuracy, reward correct responses to changes in biological input (§5). The abstract also states that future work should test whether post-training objectives that reward correct use of biological input can increase its contribution.
• Appendix B.5 states that the comparison across the six models regarding whether a predictable answer text is always present during training cannot isolate the role of training prompt content; the authors did not perform controlled interventions.

Experimental coverage

• The evaluated systems are BioReason, ChatNT, BioReason-Pro, Prot2Text-V2, Cell2Sentence-Scale, and CellWhisperer, covering genomic, protein, and single-cell modalities (Table 1).
• BioReason uses 1,449 KEGG queries, 708 genomes, 37 disease labels, and compares SFT, RL, and 42 post-training checkpoints (Table 1, Fig. 6).
• ChatNT has 4,578 three-item binary classification test queries. The authors state that these queries do not provide query-specific biological text that could conflict with DNA, so evidence conflict is not performed (§4.1).
• Protein experiments include 14,102 reviewed human proteins, 91 protein temporal holdouts, family controls for 540 proteins, and 3,917 proteins for Prot2Text-V2 (Table 1, §4.1).
• Single-cell covers five atlases, 4,846 cells, with rationale analysis on 2,000 cells. Figures show 95% CI, and auxiliary supervision gives standard errors across three seeds (Fig. 2–8, Table S9). The paper does not report hypothesis test p-values for these differences, nor agreement between automatic and manual judgments.

作者检验生物推理模型的答案是否依赖它收到的生物输入，而不是把基准准确率直接当成使用证据。

• 问题：模型同时收到基础模型表示，以及基因、通路、注释或基因列表等文本。作者认为这些表示可能重叠，所以高分不能说明某一路被用上。
• 做法：固定查询和其他输入后扰动一路生物内容；用两个未改动实体做证据冲突；贡献小时用线性探针看目标能否从该表示预测；再核对推理轨迹。对BioReason还加上编辑碱基的辅助监督。
• 谁在用输入：作者报告ChatNT、Prot2Text-V2、CellWhisperer的生物输入，以及C2S-Scale基因句里的差异表达基因，会改变任务表现。BioReason打乱DNA后，1,449条疾病预测的准确率仍为0.847（Table S2），证据冲突中97.9%跟随文本（Fig. 3a）。BioReason-Pro在GO-GPT和InterPro保留时Fmax为0.855对0.856，冲突中99.7%跟随GO-GPT（Fig. 3b、Fig. 6c）。
• 有信息但不贡献：165条基因组依赖查询上，Evo2输出的线性探针准确率为0.736，文本预测器为0.708（Fig. 4a）。作者称表示里有任务信息，但给这两个推理模型的总体提升有限。
• 后训练、轨迹和辅助目标：SFT到RL提高了BioReason和BioReason-Pro的分数，作者称Evo2、ESM3的性能贡献没有一起变大。C2S-Scale从2B到27B时，全量去掉DEG相对去掉表达相近非DEG的准确率差，从0.223到0.319（Fig. 5）。BioReason的RL轨迹在145条留出查询上都写成T>C。257-bp辅助监督下，打乱使编辑碱基对准确率下降0.549，疾病预测只下降0.021（Table S9），碱基预测的改进集中在训练见过的基因组。
• 作者的结论：高基准分可以在预测性生物输入贡献有限时出现。作者认为，拿到可预测表示，或SFT、RL之后分数更高，都不保证该表示有性能贡献；设计和评测应把正确使用与任务相关的生物输入，和预测准确率一起作为目标。