Do AI for Science Agents Preserve Model Safety?

Core question: when an LLM is embedded in an AI-for-Science agent harness, is the base model's safety behavior still preserved?

• Setting and importance: AI-for-Science agents connect LLMs to scientific databases, specialized software and computational tools for hypothesis generation, autonomous experiments and evidence interpretation; the authors point out that the deployed system is the composite agent rather than the base model itself, so safety evaluation cannot target only the base model.
• Existing gaps: The authors state that existing scientific safety benchmarks (e.g., ChemSafetyBench, SoSBench) measure harmful assistance at the LLM level, but whether safety behavior is preserved when the same base model is paired with a scientific agent framework remains unclear.
• Core observation: Refusal behavior is sensitive to how harmful requests are presented and followed up, and both simple rewording and multi-turn adaptive interaction can weaken refusal; the framework changes the information visible to the model, the actions it can execute and the interaction sequence, potentially creating new pathways for harmful requests even if the model weights are unchanged.
• Proposed approach: Using harmful chemistry as a case study, fixing the base model and comparing the harmful compliance differences between 5 agent frameworks and the corresponding base LLMs on 50 harmful chemistry requests, covering direct prompting, two adversarial prompts and one adaptive attack.
• Threat model:
  • Goal: make the agent give substantive answers to chemistry requests asking to enhance harmful properties of compounds (e.g., toxicity, corrosiveness).
  • Knowledge/capability: submitting requests and using four prompt conditions (direct, roleplay, past-tense rewording, PAIR adaptive attack); the paper does not describe attacker knowledge using white-box/black-box terminology.
  • Victim system: a composite agent consisting of a GPT-series base model plus a scientific agent framework.
  • Success determination: the MT-JailBench three-judge protocol (two binary judges plus one 5-point harmfulness judge), all instantiated with GPT-5.5.

AI-for-Science agents

• Boiko et al. (2023), Bran et al. (2024) — autonomous chemistry research and chemistry tools augmented for LLMs; Gao et al. (2024) — AI agents in biomedical discovery.
• The 5 frameworks evaluated in this paper: CACTUS (McNaughton et al., 2024), ChemToolAgent (Yu et al., 2024), STELLA (Jin et al., 2025), Biomni (Huang et al., 2026), DrugAgent (Liu et al., 2024).

Sensitivity of refusal and jailbreak

• Andriushchenko and Flammarion (2025) — past-tense attack; the authors cite it to show that simple rewording can weaken refusal; the past-tense condition in this paper follows this attack.
• Rahman et al. (2025, X-Teaming) — multi-turn adaptive jailbreak; Chao et al. (2025, PAIR) — the method used for the adaptive attack in this paper.

Harmful behavior and evaluation at the agent level

• Andriushchenko et al. (2025, AgentHarm) — a benchmark measuring the harmfulness of LLM agents; the authors cite it to show that harmful agent behavior can occur in multi-step tasks without explicit jailbreak.
• OpenAI (2026) — long-horizon models exhibit failures under deployment conditions that short evaluations do not capture; Ruan et al. (2024) — identifying agent risks using an LM-simulated sandbox; Zhang et al. (2025, ASB) — agent attack and defense benchmark.

Safety benchmarks in scientific domains

• ChemSafetyBench (Zhao et al., 2024), SoSBench (Jiang et al., 2026) — LLM safety benchmarks in chemistry and six scientific domains; SafeSciBench (Zhu et al., 2026) — the source of the 50 harmful chemistry requests in this paper; SafeScientist (Zhu et al., 2025) — oriented toward risk-aware scientific discovery.
• Urbina et al. (2022) — dual-use risks of AI drug discovery, providing motivation for the chemistry scenario.

Baselines and data: This paper has no method baseline; the comparison is the same base LLM without the framework; the evaluation data are the 50 harmful chemistry requests from SafeSciBench, and the capability comparison uses a 50-question subset of GPQA.

Positioning: The authors state that existing work measures model-level safety or harmful behavior of general agents, whereas this paper directly tests the "safety preservation" question — fixing the base model and measuring the change in harmful compliance (∆ASR) after introducing a scientific agent framework.

Overall approach: keep the base LLM unchanged, send the same set of harmful requests under the same prompt conditions to the base model and to the combined agent respectively, and use ∆ASR to measure the change in harmful compliance after introducing the framework.

Evaluation targets and data

• 5 frameworks: CACTUS, ChemToolAgent, STELLA, Biomni, DrugAgent; 4 base models: GPT-5.2, GPT-5.5, GPT-5.6-Luna, GPT-6-Luna.
• Request set: the 50 harmful chemistry requests from SafeSciBench, which ask for chemical modifications, transformations or synthesis routes that enhance harmful properties of compounds (such as toxicity, corrosiveness), usually requiring expression as SMILES; the authors state that this makes the harmful intent explicit while requiring sufficient chemistry knowledge to give a substantive answer, thereby distinguishing refusals from answers that substantively advance the request.

Prompt conditions

• Direct: requests submitted as the original text, with no jailbreak or adversarial framework; the authors state that this is the clearest test of safety preservation.
• Roleplay: requests presented through a scientific persona, preserving the harmful intent.
• Past tense: rewritten according to the published past-tense attack (Andriushchenko and Flammarion 2025).
• PAIR: adaptive attack that rewrites requests using the target model's previous responses (Chao et al. 2025).

Sampling and judgment

• Three single-turn conditions: each request is sampled 3 times at temperature 0.6, identically for the base model and the combined agent; pass@3 is reported, i.e., the proportion of requests with at least one harmful response among the three samples.
• PAIR: for each request, the first three candidates of one adaptive search are judged, and success is counted if any one meets the harm criterion; not repeated due to compute cost.
• Judges: the MT-JailBench three-judge protocol, comprising two binary judges and one 5-point harmfulness judge, with all three judges instantiated with GPT-5.5.

Core metric

• The main metric is Δ ASR = ASR_harness − ASR_base, i.e., the difference in ASR between the combined agent and the corresponding base model under the same prompt condition (Equation 1), in percentage points.
• The appendix additionally reports mean ASR ± SE (three runs), explicit refusal rate (keyword detector), and GPQA 50-question accuracy as a capability comparison.

Auxiliary analyses

• Refusal rate is measured with the keyword detector in the evaluation; the authors emphasize that refusal rate and ASR measure different outcomes and are not complementary.
• Framework configuration ablation: disable external tools or memory on STELLA and Biomni and compare PAIR ASR; the authors explicitly state that these are exploratory configuration comparisons and cannot isolate the causal effect of a single component.

Experimental setup

• Models tested: GPT-5.2, GPT-5.5, GPT-5.6-Luna, GPT-6-Luna, each combined with 5 frameworks (CACTUS, ChemToolAgent, STELLA, Biomni, DrugAgent), with the same base model without a framework as the control.
• Data: 50 harmful chemistry requests from SafeSciBench; the capability control is a 50-question subset of GPQA.
• Prompt conditions: Direct, Roleplay, Past tense, PAIR.
• Metrics: The primary metric is ∆ASR (Equation 1, the percentage-point difference relative to the base model); single-turn uses pass@3 (temperature 0.6, 3 samples), and PAIR uses the ASR of a single adaptive search (k/50).
• Judging: The MT-JailBench three-judge protocol (two binary judges plus one 5-point harmfulness judge), all instantiated by GPT-5.5; refusal rate uses a keyword detector.
• Repetition: Single-turn conditions are run 3 times and reported as mean ± SE; PAIR was not repeated due to compute cost.

Main results

Table 1 reports ∆ASR for all harness × model × condition combinations (percentage points, relative to the same base model). The table below shows selected key cells (based on Table 1 and the absolute values in Tables 4–7):

Swipe horizontally to read the table

Condition	Combination	Base ASR	Agent ASR	∆ASR
Direct	GPT-5.2 + DrugAgent	2%	72%	+70
Direct	GPT-5.2 + Biomni	2%	24%	+22
Direct	GPT-5.2 + ChemToolAgent	2%	20%	+18
Direct	GPT-5.5 + DrugAgent	4%	50%	+46
PAIR	GPT-5.5 + CACTUS / ChemToolAgent	2%	60%	+58
PAIR	GPT-5.6-Luna + CACTUS	10%	66%	+56
PAIR	GPT-6-Luna + STELLA	2%	30%	+28
PAIR	GPT-5.2 + CACTUS	56%	36%	−20

• The authors state that safety behavior can change under direct prompting (no jailbreak): on GPT-5.2, ChemToolAgent, Biomni and DrugAgent all raise ASR, while STELLA is flat or slightly lower across the four models, and CACTUS rises only slightly.
• The authors state that adaptive attacks expose larger failures of safety preservation: GPT-5.5's PAIR ASR rises from 2% to 38–60% (+36 to +58 points), GPT-5.6-Luna from 10% to 28–66%, and GPT-6-Luna from 2% to 8–30%.
• Exception: GPT-5.2's PAIR baseline is already 56%, and adding frameworks gives mixed effects, from −20 points for CACTUS to +20 points for ChemToolAgent and Biomni; under Roleplay and Past tense, configurations both rise and fall.

Refusal rate analysis (Tables 2–3)

• What was measured: The change in explicit refusal rate relative to the base model, compared against ASR.
• Results: Under PAIR with GPT-5.5, refusal rate drops from 80% for the base model to 6% for ChemToolAgent and 8% for CACTUS, while ASR rises from 2% to 60% over the same period; the reverse example is GPT-5.2 with CACTUS, where PAIR refusal rate rises from 10% to 40% and ASR falls from 56% to 36%.
• Authors' interpretation: Refusal rate and ASR measure different outcomes; a low refusal rate does not equal a corresponding number of successful harmful responses, and non-refused responses may be incomplete or technically unsuccessful.

Capability control (Figures 2–3, GPQA 50 questions)

• What was measured: Changes in GPQA accuracy for base models versus composite agents, compared against changes in direct-prompt ASR.
• Results: GPT-5.2 base model 62%, ChemToolAgent 68%, Biomni 70%, STELLA 76%, while DrugAgent 28%; DrugAgent is also below the matched baseline on GPT-5.5 (78% vs 88%) and GPT-5.6-Luna (48% vs 78%); GPT-6-Luna was evaluated only with DrugAgent (54%), with no corresponding base model result.
• Authors' interpretation: Safety degradation can occur alongside either capability increases or decreases—ChemToolAgent with GPT-5.2 rises 6 points on GPQA while ASR rises 18 points, and DrugAgent with GPT-5.2 falls 34 points on GPQA while ASR rises 70 points; the authors state that GPQA accuracy changes cannot explain the safety behavior of composite agents.

Framework configuration ablation (Table 8)

• What was measured: ASR of STELLA and Biomni under PAIR after disabling external tools or memory (GPT-5.5, GPT-6-Luna, 50 requests each).
• Results: On GPT-5.5, STELLA's full framework is 54%, disabling external tools 52%, disabling memory 56%; Biomni's full framework 56%, disabling tools 60%; on GPT-6-Luna, disabling tools for Biomni drops from 28% to 22%, still above the 2% baseline.
• Authors' interpretation: Changes from modifying components are small relative to the gap between the full framework and the base model, and are inconsistent across models and components; these are exploratory configuration comparisons and cannot localize the causal components.

Other ablations and analyses

• Aggregation rule affects direction (A.3): Under past tense with GPT-5.6-Luna, STELLA's mean ASR is 22.7% vs 26.7% for the base model (a 4-point drop), but pass@3 is 40% vs 36% (a 4-point rise).
• Largest standard error: DrugAgent with GPT-5.2 under past tense has mean ASR 35.3 ± 7.4, the largest SE among the main configurations.
• Absolute PAIR ASR: DrugAgent with GPT-5.2 is 58%, only 2 points above the already high 56% baseline; the authors note that both represent high harmful compliance.
• GPT-6-Luna direct prompting: Base model mean ASR 2.7%, mean refusal rate 46.0%; DrugAgent mean ASR 1.3%, mean refusal rate 8.7% (Table 3 and A.2).

Limitations and future directions noted by the authors

• Causal components unknown (Discussion): The authors state that it remains unclear which parts of the framework drive the observed changes and how widely they generalize across AI-for-Science systems; system instructions, additional context, repeated model calls, tool use and orchestration may all contribute, and controlled ablations are needed to separate the effects.
• Broader domains and designs (Discussion): Future work should test safety preservation in more scientific domains and agent designs.
• Expert evaluation (Discussion): Future work should have experts evaluate the correctness of harmful responses.
• Difficulty of ablation (A.4): Tools, memory, system instructions and orchestration are tightly coupled, and configuration comparisons that disable components cannot identify the causal components; the authors state that more controlled experiments are needed to determine how the framework preserves base model safety.
• Limitations of SE (A.3): Three runs provide only a limited estimate of variability; SE does not cover uncertainty from request selection, harm judgment and framework implementation.
• PAIR variability not measured (A.3): PAIR was not repeated across independent runs due to compute cost, and the variability of the adaptive search over the 50 requests was not measured.

Scope of experimental coverage

• The models tested are GPT-5.2, GPT-5.5, GPT-5.6-Luna, GPT-6-Luna, 4 in total, with 5 frameworks, 50 requests, and 4 prompt conditions (Section 2).
• Single-turn conditions: 3 runs per configuration (temperature 0.6); PAIR: one adaptive search per configuration, no repetition (Section 2, A.3).
• All three judges are instantiated with GPT-5.5; the paper does not report agreement between the judges and human judgments (Section 2).
• For the GPQA comparison, only GPT-5.2, GPT-5.5, GPT-5.6-Luna have matched results for all frameworks; GPT-6-Luna has only DrugAgent (A.1).
• The configuration ablation covers only STELLA and Biomni with external tools or memory disabled under PAIR (A.4).

An empirical study on safety preservation in AI-for-Science agents; the core question is: when a base LLM is embedded in a scientific agent framework, is the model-level safety behavior preserved?

• Problem: Deployed scientific agents are composite systems of a base model plus a framework; the authors point out that refusal behavior is sensitive to how a request is presented and followed up, and that the framework changes the context, tools and interaction sequences, potentially creating new pathways for harmful requests.
• Method: Fixing the base model, comparing the ∆ASR of 5 frameworks (CACTUS, ChemToolAgent, STELLA, Biomni, DrugAgent) with 4 GPT base models on 50 harmful chemistry requests from SafeSciBench, covering four prompt conditions: direct, roleplay, past tense and PAIR, with a three-judge protocol instantiated by GPT-5.5.
• Degradation under direct prompting alone: With GPT-5.2 and no jailbreak, DrugAgent raises ASR from 2% to 72% (+70 points), Biomni to 24%, and ChemToolAgent to 20% (Table 1, 4).
• Adaptive attacks amplify degradation: PAIR succeeds only 2% against the GPT-5.5 base model, reaching 38–60% with a framework added; GPT-5.6-Luna rises from 10% to 28–66% (Tables 5–6).
• Uneven effects: STELLA is flat or slightly lower under direct prompting across the four models; GPT-5.2's PAIR baseline is already 56%, and adding CACTUS lowers it by 20 points.
• Decoupled from capability: On GPQA, ChemToolAgent with GPT-5.2 rises 6 points while ASR rises 18 points; DrugAgent falls 34 points while ASR rises 70 points; the authors state that capability changes cannot explain the safety changes (Figure 3).
• Authors' conclusion: The safety behavior of base models is not reliably preserved once a framework is introduced; safety evaluation and alignment should target the composite agent, benchmarks should retain the base model as a reference, and retesting should occur whenever any framework component changes.