Safety-Preserving Fine-Tuning for Medical LLMs via Bi-Level Data Selection
Safety-Preserving Medical LLM Fine-Tuning via Bilevel Data Selection
The authors propose the S-BDS data selection method for medical SFT, reporting that on Llama-3.1-8B-Instruct the PSB Safe Rate is 4.64 percentage points higher than the strongest baseline.
- After a general LLM undergoes medical instruction SFT, its existing safety alignment may be weakened, while existing patient-facing safety evaluations (e.g., PatientSafetyBench) are only diagnostic and provide no mechanism for maintaining safety during training.
- Proposes S-BDS: on SEAL's bilevel data selection framework, add an entropy regularizer (λ=1) to the selection weight distribution to prevent excessive weight concentration, use 25K clinician-validated safety data as the upper-level signal to filter candidate data, then mix with unfiltered medical task data by weight ρ for SFT.
- On Llama-3.1-8B-Instruct and Qwen3-8B-Instruct, S-BDS has the highest PSB Safe Rate among all methods, 4.64 and 3.00 percentage points higher than the strongest baseline SEAL, respectively; Health Score is highest at ρ=0.5, and further increasing ρ raises safety and lowers capability.
- The authors note the study is limited to English, single-turn, patient-directed medical question answering, with all data from HealthCareMagic-100k, and may not cover real-world scenarios such as multilingual and multi-turn consultations; the authors plan future validation in more medical scenarios and deployment environments. The evaluated models are only two 8B models.
- Evaluated models
- Llama-3.1-8B-InstructQwen3-8B-Instruct
- Benchmarks
- HealthBenchPatientSafetyBenchHealthCareMagic-100k
- Metrics
- Health Score (%)PSB Safe Rate (%)Top-k% cumulative massHnormESS/N
A bi-level data selection method, S-BDS, is proposed to select samples for safety-preserving SFT of medical LLMs that balance medical capability and patient-side safety. On Llama and Qwen, S-BDS achieves the highest safety rates on HealthBench and PatientSafetyBench among all compared methods, 4.64 and 3.00 percentage points higher than the strongest baseline respectively, while maintaining competitive medical capability; adjusting the safety-oriented weight controls the trade-off between the two.
Paper analysis
What problem does this paper attempt to solve?
Addresses the problem that medical SFT weakens the existing patient-directed safety alignment of general LLMs, and proposes a safety-preserving data selection method, S-BDS.
Core problem: when a general LLM undergoes SFT on medical instruction data, its existing safety alignment may be weakened; how to control patient-facing risk while improving medical capability.
- Problem scenario and importance: patient-facing medical LLMs directly provide health information and symptom consultation to users, and their outputs may directly affect users' understanding of symptoms, medication, and decisions about seeking care, while such users often lack the expertise to discern unreliable advice (Introduction). The authors state that a general LLM's safety alignment may not be reliably maintained during domain adaptation and subsequent fine-tuning (citing Qi et al., 2024; Fraser et al., 2025).
- Shortcomings of existing methods: the authors state that early medical safety evaluations (e.g., MedSafetyBench) mostly adopt clinician-centered criteria and do not adequately cover patient-facing risks; PatientSafetyBench shifts to patient-facing evaluation but remains only diagnostic—it can identify unsafe behavior in an already-trained model but provides no mechanism for maintaining safety during medical SFT.
- Core observation: the authors' screening of HealthCareMagic-100k found that only 27.68% of samples received a relatively safe score of 1–2, with most at 3–5 (Figure 2a), indicating that safety-aware data selection is needed to mitigate safety degradation during SFT.
- Proposed approach: the paper proposes S-BDS (Scalable Bilevel Data Selection), which adds an entropy regularizer to the upper-level objective of bilevel data selection to control the concentration of selection weights, uses clinician-validated safety data to guide the ranking and filtering of candidate data, then mixes the selected subset with the original medical instruction data for SFT; and establishes a joint evaluation setting on HealthBench (medical capability) and PatientSafetyBench (patient-facing safety).
What related research exists?
Related work falls into three groups: medical-specific LLMs, medical LLM safety evaluation, and medical fine-tuning data selection; this paper builds on the SEAL framework with added weight concentration control.
Medical-specific LLMs
- Meditron (Chen et al., 2023b) and PMC-LLaMA (Wu et al., 2024)—domain-adaptive continued pretraining followed by instruction tuning.
- AlpaCare (Zhang et al., 2023b), ChatDoctor (Li et al., 2023), DISC-MedLLM (Bao et al., 2023), BianQue (Chen et al., 2023a)—SFT on medical QA or consultation data; Med-PaLM (Singhal et al., 2023) uses parameter-efficient instruction prompt tuning.
- HuatuoGPT (Zhang et al., 2023a)—SFT plus reinforcement learning with mixed feedback (RLMF); Llama-3.1-8B-UltraMedical (Zhang et al., 2024)—further uses iterative preference optimization.
Medical LLM safety evaluation
- MedSafetyBench (Han et al., 2024)—constructs unsafe requests based on nine AMA principles and evaluates clinician-side risk; MEDIC (Kanithi et al., 2024)—an evaluation framework covering end users, dimensions, use cases, and metrics. The paper notes that such work is mainly from an expert or clinical perspective, and patient-facing risk remains underexplored.
- MedRiskEval / PatientSafetyBench (Corbeil et al., 2026)—covers five categories of patient risk, and the authors state that it reveals persistent safety risks in open-source and closed-source LLMs; this paper uses its scoring criteria and benchmark for data screening and evaluation.
Medical fine-tuning data selection
- Medical-specific methods: 3DS (Ding et al., 2025) selects samples by decomposed difficulty; DIQ (Zhuang et al., 2026) combines sample difficulty with gradient influence; KDS (Zhong et al., 2026) filters by conflict between contextual knowledge and pretrained knowledge.
- General methods: DSIR (Xie et al., 2023) estimates importance ratios with bag-of-n-gram; LESS (Xia et al., 2024) uses gradient similarity to find influential samples.
- SEAL (Shen et al., 2025)—a bilevel data selection framework that learns instance-level weights to prioritize safe, high-quality samples; the paper notes that it does not explicitly control weight concentration, and this paper adds an entropy regularizer precisely for this.
Baselines and benchmarks: the comparison baselines are Standard SFT, Random, DSIR, LESS, SEAL; the evaluation benchmarks are HealthBench and PatientSafetyBench, with HealthCareMagic-100k as the data source. The authors position this paper as: adding selection weight concentration control to SEAL's bilevel framework and grounding the safety-preservation objective in the patient-facing risk medical SFT scenario.
How does the paper address this problem?
S-BDS adds an entropy regularizer to the upper-level objective of bilevel data selection to prevent weight concentration, uses clinically validated safety data to guide filtering, then mixes with task data by weight ρ for fine-tuning.
Overall approach: on top of SEAL's bilevel data selection framework, add an entropy regularizer to control the concentration of the selection weight distribution, use clinician-validated patient safety data as the upper-level signal, rank and filter samples from a large candidate pool, then mix the selected safety subset with unfiltered medical task data by weight for the final SFT.
Problem setting
- Given a patient safety dataset Dsafe (M entries, target responses satisfy patient-facing medical safety requirements) and a candidate medical fine-tuning dataset D (N entries, responses vary in risk level); the selector does not use D's sample-level risk annotations.
- Each candidate sample z i is associated with a selector parameter ωi, which after softmax and scaling yields a selection weight σi(ω); the larger the weight, the greater the influence in the lower-level SFT.
- Bilevel optimization (paper Eq. (1)): the lower level minimizes the SFT loss weighted over candidate data, and the upper level minimizes the loss on safety data minω 1/M∑i ℓ(θ⋆(ω); zisafe), i.e., it prefers weights that preserve patient safety supervision performance.
Entropy regularizer to prevent weight concentration
- The authors state that without explicit constraints, selection weights become overly concentrated on a few candidate samples, the remaining weights approach zero and the relative ranking becomes indistinguishable, making top-p selection over a large candidate pool unreliable.
- The regularizer term R(ω) = ∑i (σi(ω))/N log (σi(ω))/N equals DKL(σ(ω)/N ∥ U) − log N (U is the uniform distribution), i.e., it penalizes the selection distribution for deviating from uniform.
- The regularized upper-level objective is min (1−γ)·safety loss + γ·P(θ, ω) + λR(ω): γ ∈ (0,1) balances the safety objective and lower-level optimality, λ ≥ 0 controls the concentration of the selection distribution; λ = 0 degenerates to the unregularized form. Computing R(ω) and its gradient is O(N), and the authors state it introduces no additional parameters or model forward/backward passes.
Optimization procedure (Algorithm 1)
- Uses a lower-level optimality-gap relaxation (Shen et al., 2025) to avoid differentiating through the lower-level optimization, and a near-zero-loss approximation (assuming an overparameterized model can fit the weighted candidate data to near-zero empirical loss) to omit the auxiliary model.
- Random alternating updates: model parameters θ are updated by (1−γk)·safety sample gradient + γk·σj(ωk)·candidate sample gradient; the selector ω is updated by the candidate sample loss plus the regularizer gradient.
- Hyperparameters (Table 4): selection ratio p = 0.8, λ = 1, γk ∈ {0, 0.03, 0.06}, selector step size 5×10⁻³, model step size 1×10⁻⁵, 3 epochs of selection; SFT uses LoRA (rank 16), batch size 64, 3 epochs.
From data selection to fine-tuning
- Sort by σi(ω⋆) and keep the top p proportion to obtain Dp; the safety fine-tuning dataset Dsafety = Dsafe ∪ Dp, and the unfiltered task data Dtask is kept separately; the three are mutually disjoint.
- The final fine-tuning objective is ρ·L(θ; Dsafety) + (1−ρ)·L(θ; Dtask), where ρ ∈ [0,1] controls the balance between safety supervision and task supervision: p determines which candidate samples are selected, and ρ determines the influence strength of the safety dataset.
- Data construction: all three datasets are drawn from mutually exclusive question sets of HealthCareMagic-100k. Dsafe is 25K samples automatically safety-rewritten by GPT-5.6 Sol and then cross-reviewed in three rounds by five clinicians; the candidate set D is another 25K (20K automatically rewritten but not clinically reviewed + 5K original high-risk samples scored 4 or 5 by MedRiskEval); the remaining 50K unfiltered samples constitute Dtask. Candidate-level harmfulness scores are hidden from all selection methods and used only for construction and post-hoc analysis.
What experiments did the paper conduct?
Comparing five data selection methods on two 8B models, S-BDS has the highest PSB Safe Rate; the ρ sweep shows a trade-off between safety and capability.
Experimental setup
- Models evaluated: Llama-3.1-8B-Instruct (main base model) and Qwen3-8B-Instruct, both 8B general models that have undergone instruction tuning and safety alignment (Appendix B).
- Data: Dsafe (25K), candidate set D (25K), and Dtask (50K) constructed from HealthCareMagic-100k; each selection method keeps 80% (p = 0.8) to form a 20K subset, and Standard SFT uses the full 25K candidate set.
- Baselines: Standard SFT, Random, DSIR, LESS, SEAL.
- Medical capability evaluation: HealthBench (5,000 multi-turn conversations, 48,562 physician-written scoring criteria), with GPT-4 judging each rubric item; the metric is Health Score (%), higher is better.
- Patient-facing safety evaluation: PatientSafetyBench (466 single-turn patient queries, five risk categories), with GPT-4 as an independent reviewer giving a 1–5 harmfulness score, where 1–2 counts as safe; the metric is PSB Safe Rate (%) (the proportion scoring 1 or 2), higher is better.
- Human agreement: 480 responses (40 queries × 6 methods × 2 models) were binary-labeled by clinicians; the agreement rate between LLM labels and physician labels was 88.8%, Cohen's κ = 0.73 (Table 6).
Main results
According to Table 2 (Health Score / PSB Safe Rate, in %):
Swipe horizontally to read the table
Method Llama Health Llama PSB Qwen3 Health Qwen3 PSB Standard SFT 15.10 57.30 30.78 79.83 Random 14.58 62.88 31.23 81.76 DSIR 16.99 62.66 32.42 81.33 LESS 16.45 63.09 31.34 81.97 SEAL 18.85 64.67 31.32 82.62 S-BDS 19.04 69.31 31.78 85.62 - The authors state that S-BDS has the highest PSB Safe Rate on both models, 4.64 and 3.00 percentage points higher than the strongest baseline SEAL, respectively.
- On Health Score, S-BDS is highest on Llama and is 0.64 percentage points lower than DSIR on Qwen3; the authors state that it improves patient-facing safety without consistently sacrificing medical capability.
- The authors consider this advantage consistent with S-BDS's design: using the clinically validated Dsafe as the upper-level signal and controlling weight concentration makes the candidate ranking more reliable.
Selector concentration analysis (Section 4.3)
- Measures the concentration of the selection weight distributions of SEAL and S-BDS, with metrics Top-k% cumulative mass, Hnorm, and ESS/N (Table 1, Figure 2c–d).
- On SEAL's Orca setting (BlueOrca as safety data, RedOrca as candidate set), for Llama3.1 the SEAL selector's Top-5% mass is 42.15% and Top-20% is 64.16%; S-BDS (λ = 1) reduces these to 13.48% and 36.61%, respectively, ESS/N rises from 20.62% to 78.25%, and Hnorm rises from 0.9330 to 0.9903.
- On the HealthcareMagic-25K medical candidate set, the S-BDS selector (Llama3.1 and Qwen3) has Hnorm > 0.995 and ESS/N above 91%, and the authors state that no selector collapse risk was observed.
Data composition (safety weight ρ) experiment (Section 4.5)
- Fix p and Dsafety, vary only ρ ∈ {0, 0.25, 0.5, 0.75, 1}, with the same number of training steps (Table 3, Figure 3).
- As ρ increases from 0 to 0.5, both models' Health Score and PSB Safe Rate rise together, and Health Score is highest at ρ = 0.5 (Llama 21.28, Qwen3 34.20); after ρ exceeds 0.5, PSB Safe Rate continues to rise (at ρ = 1, Llama 74.68, Qwen3 86.27) while Health Score decreases.
- The authors interpret: ρ is an explicit data composition parameter; a moderate weight can improve both safety and capability, while a larger weight trades capability for safety; the authors state the reason is that under a fixed training budget, increasing the Dsafety weight reduces exposure to Dtask's medical supervision.
Other ablations and analyses
- Score distribution (Appendix F, Figure 4): on PSB, responses concentrate at score 2, with fewer at 1; increasing the safety weight shifts the bulk of Llama responses from score 3 to score 2.
- HealthBench distribution (Figure 5): Llama responses concentrate in the low-score range with a considerable proportion of non-positive scores; Qwen3 concentrates at 20–60; increasing the safety weight does not consistently improve Health Score, and both models achieve their highest score at a moderate weight.
- Reviewer agreement by group (Table 6): agreement rates across model×method groups are 82.5%–97.5%, κ 0.52–0.94, with 40 responses per group; the authors state that overall agreement is strong but reliability varies across models and methods.
- Training cost (Appendix B): a single full S-BDS run (including selector training and final SFT) takes about 7.3 GPU hours (Llama) / 8.4 GPU hours (Qwen3) on a single RTX 5090.
What are some points that can be further explored?
The authors note the study is limited to English single-turn patient-directed question answering and a single data source, and plan to extend to more medical scenarios and deployment environments.
Limitations and future directions noted by the authors
- The study focuses on English, single-turn, patient-directed medical question answering, and all fine-tuning data comes from HealthCareMagic-100k; the authors state this controlled setting facilitates analysis but may not fully cover the complexity of real medical interactions (Limitations section).
- Multilingual and multi-turn medical consultation scenarios are not covered (Limitations section, citing Pieri et al., 2024; Chen et al., 2023a).
- Broader clinical capabilities such as longitudinal care and clinical information extraction are not covered (Limitations section, citing Kim and Yoon, 2025).
- Reliance on a single instruction data source, which the authors state may limit generalization to other medical corpora (Limitations section).
- Future directions: the authors plan to evaluate in the future whether S-BDS remains effective in more medical scenarios and more diverse deployment settings (Limitations section).
Scope of experimental coverage
- The evaluated models are Llama-3.1-8B-Instruct and Qwen3-8B-Instruct, 2 in total, both at 8B scale (Appendix B).
- The only data source is HealthCareMagic-100k; the safety evaluation benchmark is PatientSafetyBench (466 single-turn queries), and the capability evaluation is HealthBench (5,000 dialogues) (Appendices B, C).
- The compared data selection methods are Standard SFT, Random, DSIR, LESS, and SEAL, five in total, with the selection ratio fixed at p = 0.8 (Section 4.4).
- The safety judge is a single judge model, GPT-4; the human agreement check covers 480 responses (40 per model × method group), with an overall agreement rate of 88.8% (Appendix E.2, Table 6).
- The paper does not report the number of repeated runs or variance for the main results (fixed random seed 42, Appendix B); data rewriting and harmfulness scoring use GPT-5.6 Sol (Section 4.1).
Summarize the main content of the paper
S-BDS uses entropy-regularized bilevel data selection to preserve patient-directed safety in medical SFT, and achieves the highest PSB Safe Rate on both 8B models.
A paper on a safety-preserving medical LLM fine-tuning method: after medical instruction SFT, a general LLM's safety alignment may be weakened, and the authors propose S-BDS to control patient-facing risk at the data selection stage.
- Problem: the authors' screening of HealthCareMagic-100k found that only 27.68% of samples received a relatively safe score of 1–2 (Figure 2a); existing patient-facing safety evaluations (PatientSafetyBench) are only diagnostic and provide no mechanism for maintaining safety during training.
- Method: add an entropy regularizer (λ = 1) to the upper-level objective of SEAL's bilevel data selection framework to prevent selection weights from becoming overly concentrated and making the ranking unreliable; use 25K safety samples rewritten by GPT-5.6 Sol and reviewed in three rounds by five clinicians as the upper-level signal, select 80% from a 25K candidate pool, then mix with 50K unfiltered task data by weight ρ for LoRA SFT.
- Main results (Table 2): on Llama-3.1-8B-Instruct, S-BDS achieves a PSB Safe Rate of 69.31% and a Health Score of 19.04%, both higher than SEAL (64.67 / 18.85); on Qwen3-8B-Instruct, PSB Safe Rate is highest at 85.62%, and Health Score is 31.78%, 0.64 percentage points lower than DSIR.
- Mechanistic evidence (Table 1): on the Orca setting, S-BDS reduces the Top-5% weight mass from SEAL's 42.15% to 13.48% and raises ESS/N from 20.62% to 78.25%; the authors state that concentration control yields more reliable candidate ranking.
- Trade-off analysis (Table 3): Health Score is highest for both models at safety weight ρ = 0.5; further increasing ρ raises PSB Safe Rate and lowers Health Score, and the authors state that ρ can serve as an explicit safety–capability balance parameter.
- Conclusion and boundaries: the authors consider that a moderate safety weight can improve both safety and capability; the study is limited to English single-turn patient-facing question answering and a single data source, and the authors plan to validate in more medical scenarios and deployment settings in the future.