Skip to content

Correct diagnosis, unsafe path: auditing clinical AI decision trajectories and the effectiveness and potential harms of corrective oversight

Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm

arXiv: 其他 AI 安全论文· arXiv:2611.22612· Guanqiao Chen, Wei Jiang, Shu Zhang, Qian Lin, Guanxu Yu, Siran Li, Chunhui Wang, Tianyu ShiNew York University; Xi’an Jiaotong University; The University of Hong Kong; Eberhard Karls Universität Tübingen; Effyic Technology Co., Ltd; McGill UniversityIndexed by this site · 2 days ago· Originally publishedSelected Evaluation· Its place in the scan record →SelectedAttention39

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors propose the ClinTraj audit framework, which step by step evaluates the potential harm of clinical AI decision trajectories; the authors state that 55.2% of hospital trajectories with an acceptable final diagnosis passed through moderate-to-severe potential harm decisions.

Questions
Clinical AI may reach a correct final diagnosis through unsafe intermediate decisions, whereas existing stepwise evaluations only score diagnostic correctness, information gathering or cost, and do not assess whether the decisions patients pass through along the way are safe.
Methods
Proposes ClinTraj: replays real inpatient records as checkpoint trajectories; an LLM reviewer scores each decision for clinical validity (V) and potential harm (H) against the final diagnosis and a physician rubric, calibrated by triple physician annotation; a supervision audit branches at the first error and compares four corrective feedbacks.
Experiments and results
On 300 cases from three sources, among trajectories with an acceptable final diagnosis, 55.2% (hospital), 76.2% (MIMIC-IV), and 39.6% (ASCENT) passed through moderate-to-severe potential harm decisions; structured correction resolved 73.6% of first errors, but 30.2% of post-correction trajectories introduced new potentially unsafe elements.
Limitations and future work
The authors state that the audit is demonstrated on one Chinese cardiology department, MIMIC-IV cardiology and ASCENT teaching vignettes; agreement among annotators on potential harm judgments is low (pilot Fleiss' κ 0.16); the authors plan to map physician disagreement across specialties and seniority and to trace the source of new risk after correction.
Experimental setupGPT-5.6 Sol、DeepSeek-V4-Pro et al. · hospital cohort(96 例心内科住院)、MIMIC-IV(96 例) et al. · cutr、citr et al.
Evaluated models
GPT-5.6 SolDeepSeek-V4-ProGemma-3-27B-itMedGemma-27B-text-itHuatuoGPT-o1-7BClinDiag-style multi-agent workflow (GPT-5.6 Sol backbone)ClinDiag-style multi-agent workflow (Gemma-3-27B backbone)
Benchmarks
hospital cohort(96 例心内科住院)MIMIC-IV(96 例)ASCENT(108 例)
Metrics
cutrcitrP(E=1)resolutiondefensible recoverynew harmCohen's κKrippendorff's α
AI summaryand selection rationaleThe researchers propose a clinical AI audit framework, ClinTraj, and audit 7 AI systems including GPT-5.6 Sol, DeepSeek-V4-Pro and Gemma-3-27B, finding that a correct final diagnosis does not mean the decision trajectory is safe. The framework replays inpatient records as decision trajectories and evaluates each decision for potential harm and clinical validity, covering 300 cases from a teaching hospital in China, MIMIC-IV and ASCENT. Results calibrated against physician reference standards show that in the hospital cohort, 55.2% of trajectories with a qualified final diagnosis passed through moderate-to-severe potentially harmful decisions, versus 76.2% and 39.6% on MIMIC-IV and ASCENT, respectively; the most common failure was omission of time-critical actions. On 150 physician-rated first-error events, structured correction resolved 73.6% of errors versus only 24.8% for refusal alone, but 30.2% of corrections introduced new potentially unsafe decisions within the following two checkpoints, higher than 19.7% after refusal.

The researchers propose a clinical AI audit framework, ClinTraj, and audit 7 AI systems including GPT-5.6 Sol, DeepSeek-V4-Pro and Gemma-3-27B, finding that a correct final diagnosis does not mean the decision trajectory is safe. The framework replays inpatient records as decision trajectories and evaluates each decision for potential harm and clinical validity, covering 300 cases from a teaching hospital in China, MIMIC-IV and ASCENT. Results calibrated against physician reference standards show that in the hospital cohort, 55.2% of trajectories with a qualified final diagnosis passed through moderate-to-severe potentially harmful decisions, versus 76.2% and 39.6% on MIMIC-IV and ASCENT, respectively; the most common failure was omission of time-critical actions. On 150 physician-rated first-error events, structured correction resolved 73.6% of errors versus only 24.8% for refusal alone, but 30.2% of corrections introduced new potentially unsafe decisions within the following two checkpoints, higher than 19.7% after refusal.

Why it was selectedUsing physician-calibrated trajectory audits, the paper shows that a passing final diagnosis does not equal a safe decision process, adding a trajectory-level patient-safety dimension to clinical AI evaluation.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Clinical AI can reach a correct diagnosis via unsafe paths; the authors propose auditing potential harm by trajectory and testing corrective oversight.

    Core problem: clinical AI systems can reach a correct final diagnosis through unsafe intermediate decisions, and existing evaluations cannot see this.

    • Problem setting: the authors point out that stepwise evaluation releases case evidence over time but only scores whether the system reaches the correct diagnosis, collects correct information, or controls examination cost; safety evaluation only scores a single response or recommendation. Neither asks the question patients care about most—whether decisions along the way could harm the patient.
    • Shortcomings of existing methods: the authors state that a correct discharge diagnosis can coexist with decisions such as failing to use antiplatelet drugs on the first day or discharging a high-risk patient too early; a correct final diagnosis cannot prove the path was safe.
    • Oversight gap: regulation increasingly requires human oversight of high-risk AI, but the authors cite literature stating that when overseers' interventions are not designed to be effective, oversight may become a formality; clinicians themselves are also susceptible to erroneous recommendations.
    • Core observation: correcting an error and restoring a safe trajectory are two different things—the correction content may become outdated as evidence updates, and if the system carries it forward it introduces new risk.
    • Paper proposes: the ClinTraj audit framework, comprising trajectory audit (scoring each intermediate decision for potential harm H and clinical validity V, with ratios calibrated against physician reference standards) and oversight audit (branching at the first correctable error to test whether corrective feedback can restore safe decisions); run on 300 cases from three sources and 7 AI systems.
  2. What related research exists?

    Related work covers four groups: stepwise clinical evaluation, medical AI safety evaluation, accountable evaluation, and human oversight; the authors state that none evaluates the potential harm of decisions along the way.

    Stepwise clinical AI evaluation

    • Interactive benchmarks (Johri et al. 2025; Schmidgall et al. 2026) — found that accuracy drops substantially when vignettes are turned into sequential information gathering or dialogue; MediQ (Li et al. 2024) tests whether systems ask the required questions.
    • Simulated hospital workflows (AI Hospital 2024; Medchain 2024) score history taking, tests and diagnosis in stages; Hager et al. 2024 found on real inpatient records that systems perform poorly when gathering and integrating information on their own; ClinDiag (Chen et al. 2026) found that systems proficient at static tasks cannot run a complete diagnostic process.
    • Staged analyses (Rao et al. 2026; Qiu et al. 2025) found that systems' final diagnoses were accurate but differential diagnoses failed in more than 80% of attempts; ASCENT (Choi et al. 2026) incorporates information-acquisition cost into the task. The authors note that these works measure correctness or efficiency, and none evaluates the potential harm of decisions along the way.

    Medical AI safety evaluation

    • MedSafetyBench (Han et al. 2024) tests whether systems refuse harmful medical requests; HealthBench (Arora et al. 2025) scores single conversational replies with physician rubrics; Diekmann et al. 2025 tests the agreement between LLM reviewers and physicians on the safety of patient replies.
    • NOHARM (Wu et al. 2025) evaluates the potential harm of single-consultation advice and found that omission errors dominate severe harm; Agweyu et al. 2026 tracked harmful advice in deployment. The authors note that these works all score single replies when information is complete, whereas patients are exposed to sequences of decisions as evidence accumulates.

    Evaluation as accountability practice

    • FAccT literature such as Jacobs & Wallach 2021 and Raji et al. 2021/2022/2020 — treats measurement as a normative choice and argues that audits should cover the entire development lifecycle; Liao & Xiao 2023 frames evaluation as closing the gap between models and sociotechnical use contexts; Wornow et al. 2023 criticizes health-record foundation models for being evaluated on tasks far from clinical value; DECIDE-AI (Vasey et al. 2022) requires early clinical evaluations to report human-factors and usage evidence.

    Human oversight and process disparities

    • Goh et al. 2024 found that merely giving physicians access to AI does not necessarily improve diagnostic reasoning, while Everett et al. 2026 found that structured collaborative workflows can; Gaube et al. 2021 and Qazi et al. 2026 measure the harms of erroneous advice and automation bias; Agweyu et al. 2026 reported that 67 of 115 actively harmful suggestions entered the final medical record; Green 2022 notes that oversight requirements may provide false reassurance.
    • Process-level disparities: Omar et al. 2025 and Pfohl et al. 2024 found that demographic attributes change AI recommendations; Gourabathina et al. 2025 found that non-clinical perturbations change treatment allocation.

    Positioning: the authors state that this paper brings patient safety into the stepwise setting, scoring each decision in a trajectory on the evidence visible when the decision was made, and intervening within evolving trajectories to distinguish error repair from downstream new risk.

  3. How does the paper address this problem?

    ClinTraj replays inpatient records as checkpoint trajectories, uses an LLM reviewer plus physician calibration to score each decision for V/H, and branches at the first error to test four corrective feedbacks.

    Overall approach: replay real inpatient records as a sequence of decision checkpoints, where each decision is made using only the evidence released at that time, then scored one by one by an LLM reviewer that knows the final diagnosis but not the system's identity, with physician reference standards calibrating the rates; the supervision audit branches on the same prefix to test corrective feedback. The framework is named ClinTraj.

    Data and trajectory reconstruction

    • Three sources: 96 cardiology inpatient admissions at a Chinese teaching hospital (457 checkpoints), 96 MIMIC-IV cardiology emergency admissions (553), and 108 ASCENT multi-specialty vignettes (288), totaling 300 cases and 1,298 checkpoints (Table 1).
    • Reconstruction: the model segments free-text course notes into atomic events, each with a source quote and a knowable date; events with inferred or uncertain timing are released at the latest possible date, so reconstruction errors withhold information rather than leak it.
    • Hidden reference: clinicians' assessments, plans, consultation opinions, diagnoses and discharge orders are never shown to any system; the recorded final diagnosis serves as the audit reference. T0 is admission, and at Tend the system submits its final diagnosis.

    Scoring criteria

    • Atomic decomposition: each decision is decomposed into atoms such as diagnostic assertions, tests, treatments, management and factual claims, each scored for clinical validity V (2 acceptable, 1 suboptimal but defensible, 0 unacceptable) and potential harm H (0 none to 3 severe) (Table 3).
    • Aggregation: a decision takes the lowest V and the highest H; H ≥ 2 is recorded as potentially unsafe. Final-diagnosis acceptability E is assessed only once at Tend. V and H are never combined into a single error rate.
    • Metrics: citr and cutr are computed on trajectories with an acceptable final diagnosis, citr = ∑ EI / ∑ E, cutr = ∑ EU / ∑ E, i.e., the proportion of trajectories with an acceptable final diagnosis that pass through unacceptable (V=0) or potentially unsafe (H≥2) intermediate decisions; the physician-calibrated version uses sampling weights w_j = 1/π_j for design-weighted ratio estimation.

    Trajectory audit and physician calibration

    • LLM reviewer: self-hosted Qwen3.8-27B (not within the evaluated systems), with a prompt containing three parts: a fixed general review prompt, a physician disease rubric filled in once per final diagnosis, and the checkpoint input; four rules guard against hindsight bias (judge by evidence available at the time, defensible alternatives are acceptable, no penalty for not reaching a final diagnosis before evidence suffices, H≥2 requires a credible clinical consequence).
    • Physician reference standard: stratified sampling by reviewer label E×I×U eight cells per source (hospital 120, MIMIC-IV 100, ASCENT 80 trajectories); 3 of 10 annotators with clinical backgrounds independently annotate each checkpoint, taking the median; H range ≥2, V range =2, or disputes go to 3 senior physicians for adjudication; an additional random 10% is sent to senior physicians for review to test the referral rule.
    • Statistics: design-weighted 95% intervals (Taylor linearization, logit scale); citr, cutr and ΔRecovery(C2,C1) were fixed in advance before any model runs.

    Supervision audit

    • Branch design: for each trajectory, take the first correctable error t_e (V=0, H≥2, or a fabricated/unavailable claim), freeze the prefix, and rerun t_e+1 and t_e+2 on four branches: C0 no correction, C1 rejection only with a request to reconsider, C2 structured correction (problem, corrective action, rationale), C3 the same content in prose form.
    • Three outcome types: resolution (the original error no longer exists at t_e+1), defensible recovery (resolved and V≥1 and no H≥2 element), new harm (an H≥2 new problem unrelated to the original error appears at t_e+1 or t_e+2); the primary comparison is ΔRecovery(C2, C1).
    • Annotation: 50 events stratified per source, 150 total, with 3 annotators reviewing in a fixed order (original decision → four continuations A–D in random order → the correction package only at the end), maintaining branch masking; at most 35 events per system per source.
  4. What experiments did the paper conduct?

    On 300 cases from three sources and 7 systems, 39.6%–76.2% of trajectories with an acceptable final diagnosis passed through potentially unsafe decisions; structured correction resolved 73.6% of first errors but 30.2% introduced new harm.

    Experimental setup

    • Systems under test (7): GPT-5.6 Sol, DeepSeek-V4-Pro, Gemma-3-27B-it, MedGemma-27B-text-it, HuatuoGPT-o1-7B, and a ClinDiag-style multi-agent workflow on two backbones, GPT-5.6 Sol and Gemma-3-27B (denoted MAS, run on half the cases per source). GPT-5.6 Sol and DeepSeek-V4-Pro were accessed via API and run only on the hospital cohort; MIMIC-IV and ASCENT were run on the four self-hosted systems.
    • Data: 300 cases, 1,298 checkpoints, 1,289 stepwise trajectories; hospital cohort 96 cases (457 checkpoints), MIMIC-IV 96 cases (553), ASCENT 108 cases (288).
    • Review: the main reviewer was self-hosted Qwen3.8-27B; GPT-6.1 Sol and GLM-4.7-Flash repeated the review on a seeded quarter of the open-system hospital trajectories. Physician reference standard: 300 sampled trajectories, 1,372 decisions, 4,116 ratings; oversight audit of 150 first-error events.
    • Metrics: citr, cutr (physician-calibrated, design-weighted 95% intervals); oversight audit's resolution, defensible recovery, new harm; agreement via Cohen's κ (reviewer vs. physician) and Krippendorff's α (inter-annotator).
    • Decision thresholds: H ≥ 2 is potentially unsafe, V = 0 is unacceptable, and E determines at Tend whether the final diagnosis belongs to the acceptable set.

    Main results

    Physician-calibrated ratios (Figure 4 ❶, 95% intervals in brackets):

    Swipe horizontally to read the table

    SourceFinal diagnosis acceptablecutr (H≥2)citr (V=0)
    Hospital cohort79%55.2% [42.9, 66.9]24.8% [16.3, 36.5]
    MIMIC-IV46%76.2% [59.5, 87.5]51.0% [34.9, 66.9]
    ASCENT68%39.6% [25.8, 55.3]24.3% [18.5, 32.0]
    • Per system: GPT-5.6 Sol reached an acceptable final diagnosis in 91.6% of hospital-cohort trajectories, but 35.2% of these passed through potentially unsafe decisions; the authors state that endpoint diagnostic accuracy and decision safety rank systems differently (Gemma-3-27B had a higher acceptable-final-diagnosis rate than DeepSeek-V4-Pro, but its acceptable trajectories carried potential harm more frequently, Figure 4 ❷).
    • Failure types: after weighting in the hospital cohort, missing time-critical actions accounted for 42.0% and premature convergence 15.0% (Figure 5 ❷); the H≥2 proportions for admission, intermediate, and last pre-discharge decisions were 23.0%, 28.0%, and 29.0%, respectively (Figure 5 ❶).
    • Authors' interpretation: a correct final diagnosis cannot prove the path was safe; omissions are the main source of failure, and risk rises as the hospital stay progresses.

    Oversight audit: repair is not safety

    • Resolution rate: on 150 physician-rated first-error events, C1 (refusal only) resolved 24.8% [18.0, 33.1] and C2 (structured correction) resolved 73.6% [65.2, 80.6] (C2−C1 +48.8 points); C3 (prose with the same content) 72.6%, with C2−C3 only +1.0 point, and the authors state that what determines repair is content, not format.
    • Defensible recovery: C2 was 56.0% [47.0, 64.0] and C1 was 15.0% [10.0, 22.0] (ΔRecovery +41.0 points); however, only 39.6% of events after C2 reached safe recovery, and 21.4% of repaired trajectories moved toward new potential unsafety.
    • New harm: the proportion with new potentially unsafe problems within two checkpoints after C2 was 30.2% [22.8, 38.8], versus 19.7% [13.6, 27.6] after C1 (+10.5 points); among new harms after correction, 21.0% were claims unsupported by the evidence at the time (13.0% after C1). Annotators considered 72.0% of correction packages clinically valid, and only 9.0% introduced risk themselves; the authors believe this points to correction content becoming outdated after evidence updates and then being carried forward.

    Reviewer–physician disagreement

    • Agreement: Cohen's κ between reviewers and the physician reference standard was 0.54 for final-diagnosis acceptability, 0.28 for unacceptable decisions, and 0.21 for potential unsafety (Figure 7 ❶); inter-annotator Krippendorff's α on the full scale was E 0.63, V 0.28, H 0.23.
    • Direction: after weighting, reviewers judged 50.0% of decisions potentially unsafe, versus 23.5% for the reference standard; 71.0% of decisions rated severe (H=3) by the reviewer were downgraded to a lower level by the reference standard (Figure 7 ❷). The authors state that potential harm is the dimension with the largest disagreement between automated review and clinicians, and that the clinical reference standard calibrates it.

    Other ablations and analyses

    • Full-record comparison (Table 8): under full records (F), 88.7% of trajectories contained unacceptable intermediate decisions and 89.4% contained potentially unsafe decisions; with stepwise input (T) these fell to 55.3% and 80.4%, while the final diagnosis was almost unchanged (77.8% vs 77.4%); under F, 51.2% of intermediate decisions explicitly relied on evidence not yet available at the time (hrr).
    • ASCENT public test (Table 5): appending subsequent findings to the stepwise prompt raised micro-F1 from 0.480 to 0.528 (+0.048, 900 paired items) under the official ASCENT reviewer (GPT-4.1).
    • Cross-reviewer agreement (Appendix H): on the seeded quarter sample, GPT-6.1 Sol and GLM-4.7-Flash likewise judged that most acceptable trajectories contained potentially unsafe decisions (74.5% and 60.0%; main reviewer 98.0% and 96.8%).
    • Per-system reviewer labels (Table 6): hospital-cohort cutr ranged from 43.3% for GPT-5.6 Sol to 100.0% for HuatuoGPT-o1-7B; the MAS arms were run on half the sample.
    • Sensitivity (Table 10): using only the first two intermediate checkpoints (W2), hospital-cohort cutr was 61.1%; severe harm (H=3) was 35.4% (pooled across the five single-model systems).
    • Referral-rule misses (Appendix E): 23.0% of annotated decisions triggered referral, and random senior review moved 2.0% of decisions across the V=0 or H≥2 threshold.
  5. What are some points that can be further explored?

    The authors list future directions such as disagreement in harm judgments and the source of new risk after correction; the experiments cover three sources, 7 systems and 150 supervision events.

    Limitations and future directions noted by the authors

    • Disagreement in harm judgments is itself an open question (Section 5.1): the authors state that the degree of disagreement on potential harm between model reviewers and clinicians, and among clinicians themselves — which omissions count, how wide the window is, how to draw the line between moderate and severe — is a core question for clinical AI accountability; mapping these disagreements across specialties, case types and seniority, and determining which disagreements adjudication protocols should resolve, is the next step for harm-oriented auditing.
    • The source of new risk after correction remains to be located (Section 5.2): the authors plan supervision experiments with more events per system, longer post-correction windows and provenance tracking to distinguish correction content that is carried forward from new risk generated independently.
    • Demonstration scope (Section 5.3): the authors state that the audit is demonstrated on inpatient records from one Chinese cardiology department, MIMIC-IV cardiology admissions from one US center, and multi-specialty teaching vignettes from ASCENT; the pipeline consumes ordinary discharge documents, and the code path is independent of specialty and language.
    • Judged by what was knowable at the time rather than counterfactually (Section 5.3): records are replayed as they actually occurred, and decisions are judged by the information knowable at each checkpoint, not against a counterfactual care pathway.
    • Data licensing shapes what can be audited (Section 5.1): PhysioNet terms of use prevent API-accessed systems from accessing MIMIC-IV, and the authors state that the systems most likely to be procured are the hardest to examine on trusted clinical data.

    Experimental coverage

    • 7 systems were evaluated: 5 single-model systems plus 2 ClinDiag-style agent arms; GPT-5.6 Sol and DeepSeek-V4-Pro were run only on the hospital cohort, and MIMIC-IV and ASCENT each ran 4 self-hosted systems.
    • Case sources were a predominantly cardiology hospital cohort (96 cases), MIMIC-IV cardiology admissions (96 cases) and ASCENT multi-specialty vignettes (108 cases), totaling 300 cases and 1,298 checkpoints.
    • The physician reference standard covers 1,372 decisions (4,116 ratings) from 300 sampled trajectories and 150 first-error events; 3 annotators per event.
    • The supervision audit tests only the outcomes of the four feedback arms (C0–C3) within two checkpoints after the first error; at most 35 events per system per source.
    • The paper does not report the number of repeated runs for the stepwise experiments; per-system physician-calibrated rates are post-stratification estimates assuming that reference labels are independent of the system within the same reviewer stratum (Appendix E reports a test of this assumption).
  6. Summarize the main content of the paper

    ClinTraj audits the potential harm of clinical AI by trajectory: a correct final diagnosis cannot prove the path was safe, and successful error correction cannot prove safe recovery.

    A clinical AI evaluation paper that proposes the ClinTraj audit framework, making patient safety the goal of stepwise clinical evaluation.

    • Problem: stepwise benchmarks only score diagnostic correctness, and safety evaluation only scores a single response; the authors state that a correct final diagnosis can coexist with potentially harmful decisions along the way.
    • Method: 300 real inpatient records (96 hospital cardiology, 96 MIMIC-IV, 108 ASCENT) are replayed as 1,298 checkpoints; an LLM reviewer (Qwen3.8-27B) scores each decision for V/H against the final diagnosis and physician rubric, based on the evidence visible at the time; triple annotation by 10 annotators plus adjudication by 3 senior physicians forms the reference standard, and design weighting yields physician-calibrated ratios; the oversight audit branches at the first error and compares four kinds of feedback: no correction, refusal only, structured correction, and prose with the same content.
    • Result one: among trajectories with an acceptable final diagnosis, the proportion passing through moderate-to-severe potential harm decisions was 55.2% (hospital), 76.2% (MIMIC-IV), and 39.6% (ASCENT); missing time-critical actions was the main failure type (42.0% in the hospital cohort).
    • Result two: structured correction resolved 73.6% of first errors (refusal only 24.8%), and defensible recovery rose from 15.0% to 56.0%, but 30.2% of post-correction trajectories introduced new potentially unsafe elements within two checkpoints (refusal only 19.7%).
    • Result three: potential harm was the dimension with the largest disagreement between LLM reviewers and physicians (κ 0.21, versus 0.54 for final diagnosis); 71.0% of decisions rated severe by the reviewer were downgraded by physicians.
    • Authors' conclusion: a correct final diagnosis cannot prove a safe clinical decision trajectory, and successful error correction cannot prove safe recovery; evaluation should be designed for continuous clinical oversight, dynamic correction, and trajectory-level safety assurance.
Read originalarxiv.org