When Medical Safety Alignment Fails: The MedHarm Benchmark for Evaluating Large Language Models on High-Risk Medical Queries

Researchers release MedHarm, a high-risk medical safety benchmark, evaluating large language models with 1100 medically based queries, and find that well-aligned models may still give unsafe or actionable answers. The benchmark covers 10 safety-critical categories including toxicology, pharmacology, covert poisoning, anesthesia, and fetal harm, focusing on real-world clinical, educational, and technical prompts that require refusal, caution, or safe referral rather than direct help. The evaluation targets include 15 large language models (general, medical-specific, closed-source, and downstream SFT models) and 4 representative guardrail models; results show that medical fine-tuning may amplify the specificity of harmful answers, and that external guardrails, while reducing some failures, come with brittle interception and weaker safety helpfulness. The authors accordingly note that medical safety cannot be inferred from general alignment or medical capability, and that domain-specific stress testing is needed before deploying safety-critical medical applications; code and data will be released upon paper acceptance, with sensitive data available only to qualified researchers.