InclusionMed: An AI benchmark for inclusive healthcare
InclusionMed: Benchmarking AI for Inclusive Healthcare
InclusionMed organized medical AI benchmarks with Atlas across 3 roles × 3 capabilities, selecting 5 benchmarks from 388 candidates to evaluate 22 models; the Index total score ranged from 17.5 to 49.1.
- Medical AI evaluation needs to cover diverse medical roles, tasks, and care settings, but existing benchmarks have uneven coverage, and a single total score is hard-pressed to reflect the needs of different users.
- The authors propose InclusionMed: using a nine-grid Atlas of 3 medical roles × 3 AI capabilities to organize benchmarks, selecting 4 public benchmarks from 388 candidates through audit against six criteria and independent reproduction, adding a self-built private diagnostic benchmark InclusionMed-DxFiles, and aggregating them with equal weights into an Index.
- The Index total scores of the 22 models range from 17.5 to 49.1; rankings vary by role and capability; the top ten models all score higher on patient communication than on clinical decision-making; models with similar scores can differ in reasoning cost by about 24 times.
- The authors note that the current Atlas covers only 3 of the 9 cells, has no benchmarks aimed at patients and the public, and its tasks are mainly U.S.-based, English, and plain text; cost estimates exclude failed retries and review costs; future work will include weight sensitivity analysis and periodic reproduction.
- Evaluated models
- Gemini 4 ArgonClaude Opus 5.5Claude Fable 5.1GPT-6 AstraGPT-6.1 SolDeepSeek V4.1 FlashGrok 4.7Kimi K3.1 ProQwen3.8 MaxKimi K3Gemini 3.8 FlashGPT-5.6 LunaGLM-5.3DeepSeek V4 ProGPT-6 LunaMiniMax M3.1 FlashStep 5 PreviewMiMo V2.6 ProGLM-5.3 FlashQwen3.8 27BGrok 4.3Step 3.7 Flash
- Benchmarks
- HealthBench ProfessionalPhysicianBenchALE-CLI v1.0: Health and MedicineCHI-BenchInclusionMed-DxFiles
- Metrics
- InclusionMed Indexlength-adjusted rubric scorepass@1task scoreperfect-grade rateestimated inference cost per task run
InclusionMed is an open evaluation program for inclusive healthcare, releasing Atlas and Index successively. Atlas organizes benchmarks by healthcare role (patients and the public, healthcare professionals, healthcare management professionals) and AI capability (knowledge, reasoning, agentic capability), revealing coverage and gaps. Index aggregates model results across benchmarks; the first version takes an equal-weight average of scores on five benchmarks, broken down by role and capability, and rankings and scores differ.
Paper analysis
What problem does this paper attempt to solve?
Medical AI benchmark coverage is uneven; the authors propose the InclusionMed Atlas and Index, which organize benchmarks by role × capability.
Core problem: medical AI evaluation benchmarks cover medical roles, tasks and care settings unevenly, and a single overall score is hard-pressed to reflect the needs of different users.
- Setting and motivation: medical resources are unevenly distributed, and medical AI is entering settings such as personal health management, clinical practice and care organization; the authors argue that evaluation needs benchmarks reflecting diverse medical roles, tasks and settings (§1).
- Existing gaps: the authors point out that existing benchmarks each have their own focus—MedQA/MedMCQA are exam questions, HealthBench is open-ended health dialogue, MedAgentBench/CHI-Bench are simulated workflows; although HELM, MedHELM and the Artificial Analysis medical index aggregate, the meaning of the overall score depends on what the included benchmarks evaluate and whose needs they represent (§1).
- Core idea: organize benchmarks along two axes—medical role (patients and the public, medical professionals, medical administrative professionals) and AI capability (knowledge, reasoning, agency)—forming a nine-cell Atlas that makes coverage gaps explicitly visible (§2).
- Paper outputs: an open evaluation plan, InclusionMed, containing the Atlas framework, 5 benchmark combinations selected through auditing and independent reproduction, the InclusionMed Index for 22 models, and breakdown analyses by role, capability and task attribute plus a reasoning cost analysis (§1, §5).
What related research exists?
Related work falls into four categories: medical task benchmarks, agent workflow benchmarks, aggregate evaluation frameworks, and evaluation methodology and documentation standards.
Medical knowledge and dialogue benchmarks
- MedQA, MedMCQA—medical exam question-answering datasets (§1).
- HealthBench—open-ended health dialogue evaluation (§1); excluded in this reproduction due to insufficient discrimination (§3.3).
- XLingHealth, NEJMQA-Exams, PubHealthBench—medical knowledge benchmarks, excluded in this reproduction due to score saturation (§3.3).
Medical agent and workflow benchmarks
- MedAgentBench—benchmark for medical LLM agents in a virtual EHR environment (§1).
- CHI-Bench—end-to-end U.S. medical operations workflow benchmark, included in the Atlas this time (§3.4).
- PhysicianBench, ALE-CLI v1.0—EHR workflow and computational research task benchmarks, included this time (§3.4).
Aggregate evaluations and indices
- HELM, MedHELM—overall evaluation frameworks across tasks and datasets (§1).
- Artificial Analysis Healthcare & Medical Index—aggregates selected benchmark results into a total score (§1).
Evaluation methodology
- Jacobs & Wallach's work on measurement and fairness, and Raji et al.'s benchmark critique—the authors use these to emphasize that the meaning of a total score depends on whose needs the benchmarks represent (§1).
- Datasheets for Datasets—a reference for benchmark card documentation (§3.1).
Baselines and comparisons: this paper does not propose a new method to compare against baselines, but evaluates 22 models; the 5 included benchmarks are HealthBench Professional, PhysicianBench, ALE-CLI v1.0: Health and Medicine, CHI-Bench, and the self-built InclusionMed-DxFiles (§4).
The authors position the paper as combining benchmark audit, independent reproduction, and a role- and capability-organized Atlas to make coverage gaps explicit, supplemented by task-attribute breakdowns and cost analysis, distinguishing it from aggregate indices that give only a total score (§1, §6).
How does the paper address this problem?
The Atlas nine-cell grid organizes the benchmarks, six criteria are audited plus independent reproduction screening, and five benchmarks are equally weighted and aggregated into the Index.
Overall approach: first use the InclusionMed Atlas (a 3-role × 3-capability nine-grid) to position benchmarks, then filter benchmarks through three steps—collection, audit, and independent reproduction—and finally aggregate the scores of each benchmark with equal weights into the InclusionMed Index, along with task-attribute breakdowns and inference cost analysis.
Atlas framework
- Role axis: patients and the public, medical professionals, medical administrative professionals (§2.1).
- Capability axis: knowledge (providing or applying medical knowledge), reasoning (interpreting evidence to form judgments), agency (executing actions or producing deliverables) (§2.2).
- Mapping rules: for cross-domain benchmarks, first take the medical-related subset, then assign it to the corresponding cell according to the dominant role–capability combination in the task (§2.3).
Benchmark screening process
- Collection: candidates published from May 2023 to September 2026 were collected from literature, public platforms, and community contributions, and LLM-assisted organization into benchmark cards yielded 388 in total (§3.1, Appendix A.1).
- Audit: six criteria—medical relevance, credibility, usability and licensing, complementary coverage, timeliness and data independence, and discriminative ability; an agent skill assisted the review, and the research team reviewed each exclusion decision (§3.2, Appendix A.1).
- Independent reproduction: the 8 shortlisted benchmarks were reproduced with the same set of 8 models (GPT-6 Astra, Claude Opus 5.5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Qwen3.8 Max, Qwen3.8 27B, GLM-5.3, Grok 4.3); exclusion criteria were score saturation (all top three ≥85 points) or insufficient discrimination (sample standard deviation of the 8 models' scores ≤4) (Appendix A.1).
The five included benchmarks
- HealthBench Professional: 525 clinical dialogue cases, scored by physician-authored case-level rubrics, reporting length-adjusted rubric scores (0–100), assigned to medical professionals × reasoning (§3.4).
- PhysicianBench: 100 EHR workflow cases, 670 checkpoints, pass@1 requires all checkpoints to pass, assigned to medical professionals × agency (§3.4).
- ALE-CLI v1.0: Health and Medicine: 22 computational research tasks, task-specific graders give partial credit, assigned to medical professionals × agency (§3.4).
- CHI-Bench: 75 U.S. medical operations cases, each case run 3 times independently and averaged, pass@1 requires passing both the workflow check and the rubric evaluation, assigned to medical administrative professionals × agency (§3.4).
- InclusionMed-DxFiles (private): 312 diagnostic reasoning cases adapted from Massachusetts General Hospital case records in the New England Journal of Medicine, with the final diagnosis hidden; scored against reference diagnoses using GPT-6.1 Sol, perfect is recorded only when the primary diagnosis is correct and all additional diagnoses are supported, and the perfect proportion is reported (§3.5, Appendix C).
Index aggregation and cost
- Aggregation formula: Ii = 1/n∑j=1n xij, i.e., model i's Index is the equal-weight average of its n=5 benchmark scores (each 0–100), with each benchmark weighted 20% (§4.3).
- Role/capability scores: the mean of benchmark scores within that category; the current Atlas covers only 3 of the 9 cells, and the remaining categories have no scores (§4.3, Table 2).
- Inference cost: per-task cost estimated from recorded input/output token usage and prices published on November 13, 2026, including cached token rates; averaged across benchmarks by Index weight, and the Pareto frontier identified (§4.4, Appendix A.2).
- Task attribute labels: 8 attributes grouped into 4 dimensions (medical context, medical content, capability and evaluation, information characteristics), used for within-benchmark breakdown analyses (§3.3, Figure S1).
What experiments did the paper conduct?
22 models were evaluated on 5 benchmarks; Index total scores range from 17.5 to 49.1, rankings change with role/capability, and cost differences are about 24-fold.
Experimental setup
- Models evaluated: 22 released versions across 11 families—Claude (Opus 5.5, Fable 5.1), DeepSeek (V4.1 Flash 1017, V4 Pro 0813), Gemini (4 Argon, 3.8 Flash), GLM (5.3, 5.3 Flash), GPT (6 Astra, 6.1 Sol, 6 Luna, 5.6 Luna), Grok (4.3, 4.7), Kimi (K3, K3.1 Pro), MiniMax M3.1 Flash, MiMo V2.6 Pro, Qwen (3.8 Max 0902, 3.8 27B), Step (5 Preview, 3.7 Flash) (Table S1).
- Benchmarks and scale: HealthBench Professional 525 cases (8 scored runs per task), PhysicianBench 100 cases (1), ALE-CLI 22 cases (1), CHI-Bench 75 cases (3), InclusionMed-DxFiles 312 cases (1) (Table 1).
- Metrics: length-adjusted rubric score, pass@1, task score (partial credit), perfect proportion, all on a unified 0–100 scale; the Index is the equal-weight average of the five benchmarks (§4.2–4.3).
- Grading method: HealthBench Professional uses GPT-5.4 as grader (reasoning effort set to low); InclusionMed-DxFiles uses GPT-6.1 Sol to score against the reference diagnosis (Appendix A.2, C.3).
- Execution setup: HealthBench Professional and DxFiles use direct API calls; PhysicianBench uses an EHR agent in a FHIR environment; CHI-Bench and ALE-CLI use CLI agents (Codex for GPT models, Gemini CLI for Gemini models, Claude Code for the other 16); the two Qwen versions use reasoning-effort xhigh, the other 20 use high (Appendix A.2).
- Failure handling: runs still invalid after retries and timed-out runs are counted in the denominator as failures (§4.2, Appendix A.2).
Main results
The InclusionMed Index total scores of the 22 models range from 17.5 to 49.1 (Figure 4); scores by role and capability are in Table 2 (0–100; dashes indicate the current Atlas does not cover them):
Swipe horizontally to read the table
Model Medical professionals Medical administrative professionals Reasoning Agency Gemini 4 Argon 55.0 25.3 54.2 45.7 Claude Opus 5.5 55.6 22.2 55.4 44.7 Claude Fable 5.1 54.8 25.3 53.3 46.1 GPT-6 Astra 52.4 16.0 58.7 36.1 GPT-6.1 Sol 52.1 14.2 58.3 35.4 Grok 4.3 24.1 5.8 37.8 8.9 Step 3.7 Flash 18.9 12.0 23.7 13.4 (According to Table 2; only the top five by total score and the last two are listed, the other 15 models are omitted; the "patients and the public" and "Knowledge" columns currently have no benchmark coverage in the Atlas and are all dashes.)
- Total score ranking: Gemini 4 Argon is highest (49.1), followed by Claude Opus 5.5 (48.9), Claude Fable 5.1 (48.9), GPT-6 Astra (45.1), GPT-6.1 Sol (44.5) (§5.1).
- Ranking varies by dimension: the authors report that Claude Opus 5.5 leads on the medical professionals dimension (55.6), GPT-6 Astra leads on reasoning (58.7), and Claude Fable 5.1 leads on agency (46.1); GLM-5.3 ranks 14th on medical professionals and 4th on medical administrative professionals; GPT-6 Luna ranks 8th on reasoning and 20th on agency (§5.2).
- Authors' interpretation: the Atlas lets users go beyond the overall ranking and compare models by the roles and capabilities relevant to their own work (§5.2).
Task attribute breakdown (HealthBench Professional, top ten models by Index)
- Medical expertise dimension: patient communication has the highest median (69.1), followed by medical understanding (60.5); medical calculation (47.0) and safety and ethics (48.8) have the lowest medians, and safety and ethics has the widest cross-model range (21.0–62.5) (Table 3).
- Clinical specialty dimension: obstetrics and gynecology has the highest median (60.6), psychiatry the lowest (49.4); all ten models score lower on surgical specialties than on internal medicine specialties (median 51.5 vs. 58.2) (Table 3).
- Workflow dimension: all ten models score lower on treatment planning and implementation than on clinical assessment and diagnosis (median 53.0 vs. 59.1); in PhysicianBench, eight of the ten show the same pattern (Table 4, Table S5).
PhysicianBench failure checkpoint analysis
- What was measured: the 912 failure checkpoints of the top ten models were counted by category (Table S6).
- Results: clinical reasoning accounts for 469 (51.4%) and is also the most-tested category; the highest median within-category failure rate is action execution (17.4%), followed by documentation (14.6%) (§5.3, Table S6).
- Authors' interpretation: there is a need to evaluate both clinical reasoning and the ability to reliably execute clinical actions and document care (§5.3).
Performance–cost trade-off
- Similar scores, large cost differences: Claude Fable 5.1 and Claude Opus 5.5 both score about 48.9 overall, but Fable's estimated cost per task is about 2.5 times Opus's ($4.12 vs. $1.68); between Kimi K3 (37.9 points, $1.51) and GPT-5.6 Luna (36.9 points, $0.064), the latter can run about 24 times as many tasks under a fixed budget (§5.4).
- Pareto frontier: five models lie on the observed Pareto frontier—GPT-6 Luna, GPT-5.6 Luna, DeepSeek V4.1 Flash, GPT-6.1 Sol, Gemini 4 Argon (Figure 5).
Other ablations and analyses
- Benchmark reproduction screening: XLingHealth, NEJMQA-Exams and PubHealthBench met the saturation criterion (top three ≥85), and HealthBench met the insufficient-discrimination criterion with a standard deviation of 2.67 ≤4 across 8 models; all were excluded; the five retained benchmarks have standard deviations of 6.04–18.62 (Figure S3).
- Physiological system dimension: systemic/multisystem tasks have the lowest median (42.2), and no model reaches 60 (Table S4).
- Rubric evaluation targets: information gathering and clarification has the lowest median weighted pass rate (66.1%), and seven of the ten models rank last on this item; contextual and resource adaptation is highest (79.7%) (Table S4).
- Task distribution: of 1,034 tasks, 318 tasks with a determinable geographic setting are associated with the United States; 964 (93.23%) are in English; 1,032 (99.81%) are plain text (Appendix B.2).
What are some points that can be further explored?
The authors list four extension directions: breadth of coverage, contextual interpretation, resource constraints, and temporal comparability; the experiments cover 22 models and 5 benchmarks.
Limitations and future directions noted by the authors
- Coverage gaps: the current Atlas has no benchmarks primarily aimed at patients and the public, and tasks with a determinable geographic setting are mostly U.S.-based; the authors state that improvements on the current combination cannot establish improvements for these populations and scenarios (§6.1).
- Total-score sensitivity: the first version weights the 5 benchmarks equally and covers only 3 of the 9 cells; the authors plan future sensitivity analyses that change benchmark weights and omit benchmarks one at a time to test which model comparisons remain stable (§6.2).
- Comparability across categories: differences in benchmark content and scoring methods limit direct comparisons across roles and capability categories (§6.2).
- Scope of cost estimates: estimates are based on token usage and published prices, exclude failed runs and retries on the four public benchmarks, and exclude review, search, tool, sandbox, and platform internal costs, and do not represent total deployment expenditure; the authors state that future work should cover the full workflow (§6.3, Appendix A.2).
- Timeliness and reproduction: tasks may approach score ceilings as models improve and require periodic re-evaluation; of the 388 candidates, only 104 provided directly reproducible materials, and the authors call for clearer access conditions and reproducible environments (§6.4).
- Clinical applicability: the authors state that after narrowing the candidate models, applicability to a specific clinical deployment still needs validation in the target setting, and clinical benefit needs separate evaluation (§6.2, §6.3).
Scope of experiments
- The models tested are 22 released versions across 11 families (Table S1).
- The evaluation benchmarks are 5: 4 public benchmarks plus the private InclusionMed-DxFiles, covering 3 of the Atlas's 9 cells (§4).
- The task-attribute breakdown analyses target only the top ten models by Index and are conducted mainly within the HealthBench Professional and PhysicianBench benchmarks (§5.3).
- The screening in the independent reproduction stage used 8 models scored on 8 shortlisted benchmarks (Appendix A.1).
- The paper does not report an experiment on the agreement between the scoring of the reviewer models (GPT-5.4, GPT-6.1 Sol) and human judgment.
Summarize the main content of the paper
InclusionMed organizes medical AI benchmarks with a role × capability nine-grid; the 22 models' Index total scores range from 17.5 to 49.1, and rankings change with the dimension.
InclusionMed is an open medical AI evaluation program that, addressing the uneven coverage of medical roles, tasks, and scenarios by existing benchmarks, proposes organizing benchmarks with a nine-grid Atlas and aggregating them into scores.
- Framework: the Atlas positions benchmarks by 3 medical roles (patients and the public, medical professionals, medical administrative professionals) × 3 AI capabilities (knowledge, reasoning, agency), making coverage gaps explicitly visible.
- Screening: starting from 388 candidate benchmarks, through audit against six criteria and independent reproduction of 8 shortlisted ones, 3 saturated knowledge benchmarks and the insufficiently discriminative HealthBench were excluded, retaining 4 public benchmarks plus a self-built 312-case private diagnostic reasoning benchmark, InclusionMed-DxFiles.
- Main results: the 22 models' Index total scores range from 17.5 to 49.1, with Gemini 4 Argon highest (49.1); the leaders on the role and capability dimensions differ from the overall ranking, e.g., GPT-6 Astra leads on reasoning (58.7) but scores only 36.1 on agency.
- Breakdown findings: among the top ten models by Index on HealthBench Professional, patient communication scores (median 69.1) are all higher than clinical decision-making (55.5), and treatment planning and implementation (53.0) are all lower than clinical assessment and diagnosis (59.1); among PhysicianBench failed checkpoints, 51.4% are clinical reasoning.
- Cost findings: models with similar scores differ greatly in estimated inference cost, with Kimi K3 and GPT-5.6 Luna differing by about 24-fold in per-task cost.
- Authors' conclusions: the authors argue for evaluations covering more diverse medical needs and for selecting models by scenario, and plan to expand the Atlas through community contributions, conduct weight sensitivity analysis, and perform periodic reproduction.