HealthBench Professional: evaluating large language models with real clinician conversations

The researchers introduce an open benchmark, HealthBench Professional, evaluating large language models on real tasks that clinicians posed to ChatGPT in their work, covering three scenarios: clinical consultation, writing and documentation, and medical research. Each example is a doctor-written conversation with ChatGPT for Clinicians, and the scoring rubric was written by three or more doctors in three stages with iterative adjudication; examples were selected from a candidate pool of 15,079 based on quality, representativeness, and difficulty, expanded by about 3.5 times with difficult examples for recent OpenAI models, and about one third of examples involve doctors deliberately adversarially testing the model. The benchmark also collected human doctor responses as a baseline, under conditions of unlimited time, matched specialty, and internet access; the highest-scoring system is GPT-5.4 in ChatGPT for Clinicians, surpassing the base version of GPT-5.4, all other models, and human doctors.