DistressBench: Evaluating multi-dimensional crisis support in large language models

Existing suicide and self-harm (SSH) evaluations mainly score refusal, but crisis conversations require actionable support.

• Use case: The authors state companion chat is already the default interface for information seeking and emotional support. When users disclose suicidal ideation or self-harm, the expected behavior is to identify signals, calibrate empathy, de-escalate without endorsing harm, and connect to region-appropriate human resources.
• Shortcomings of existing methods: The authors state existing evaluations are mostly single-turn, binary outcomes, checking whether the model refuses or gives unsafe completions. Refusal rates do not predict ethical behavior well; clinically inadequate refusals can deter help-seeking; real-world failures also include missing implicit ideation, sycophancy to maintain rapport, insufficient referral, and safety behaviors established in the first turn not persisting in longer conversations.
• Core observation: The authors report that across 25 frontier models, the main failure is inaction after correct assessment, not harmful output. Aggregated, in dialogues satisfying all clinically written identification criteria, 35.3% fail to direct the user to help; the authors state refusal-based benchmarks cannot detect this.
• Proposed benchmark: DistressBench, 718 English dialogues across 23 SSH subcategories, each with seven-dimensional weighted atomic criteria and clinical reference responses. Scores are the proportion of weight satisfied, which the authors interpret as the proportion of required care delivered. There is also a 20-transcript long-horizon red-team appendix, not included in the main leaderboard.

Recent clinical crisis evaluations

• VERA-MH (Belli et al., 2026): uses clinically designed persona simulation conversations, scored on a single global scale with five dimensions, compressed into a 0–100 composite score. The authors state that this paper retains scoring against clinically specified criteria and makes three changes: the user side is expert-written and fixed, covers 23 SSH subcategories, and criteria are written per task rather than shared across the whole corpus.
• Bentley et al. (2026): applies VERA-MH to ten general-purpose models. The authors state that detection was found to be relatively strong, while no model exceeded 30/100 in guiding users to seek human care. The authors state that this is isomorphic to this paper's detection–action asymmetry, and that this convergence is the strongest evidence currently available that the gap is a model property rather than an artifact of this tool.
• Adjacent mental health evaluations: TrustMH-Bench covers trustworthiness pillars, including crisis detection and anti-sycophancy; PsychEthicsBench is compared against professional ethics guidelines; SproutBench focuses on developmentally appropriate behavior toward minors. Moore et al. (2025) report that current models express stigma and encourage delusional thinking; the authors state that the mitigation result is quantifying such issues with per-task criteria.

Scales, refusal metrics, and adjudication

• FORTRESS (Knight et al., 2025) and MoReBench (Chiu et al., 2025): the authors state that they follow their scale tradition. The jury chose to continue scoring candidate juries against expert labels, freezing the winning combination, and using supplier diversity to limit same-family bias; they also cite Zheng et al. (2023)'s LLM-as-a-judge practice.
• Refusal as a safety metric: the authors cite PsychEthicsBench, stating that refusal rates do not predict ethical behavior well, and that clinically inadequate refusals hinder help-seeking, so they evaluate the process rather than refusal.

Long conversations and sycophancy

• LongSafety (Lu et al., 2025): short-context safety does not predict long-context safety. Crescendo (Russinovich et al., 2025) bypasses guardrails with turn-by-turn seemingly harmless escalations. Sharma et al. (2024) treat sycophancy as a possible mechanism.
• The authors' comparison: the authors state that DistressBench's contribution is multi-turn scoring rules validated against clinical labels, plus a separate long-horizon appendix; the turn-by-turn drift claim was withdrawn after turn-by-turn clinical labels did not reach usable agreement. The authors call this a negative result on the reliability of turn-by-turn clinical safety annotation.

Baselines and benchmarks

• Comparison targets: external benchmarks were not used as test sets. The evaluated models are the 25 frontier models in Appendix G; clinical reference responses under the same review process are used for ceiling analysis, which the authors explicitly do not treat as a human baseline. The corpus is DistressBench (718 items) and DistressBench-LH (20 items, not on the main leaderboard).

The authors position this paper relative to VERA-MH as: retaining clinically specified criteria, changing the corpus to expert-written and fixed conversations, and using per-task scales to make dimension subscores and within-task weights meaningful.

DistressBench frames crisis handling as discrete actions judged necessary by clinicians, replacing refusal labels with weighted satisfaction rates.

Tasks, taxonomy, and scales

• Scale: v1 is English SSH simulated dialogues. Of 718 items, 299 are single-turn and 419 multi-turn; 416 items have exactly three user turns, averaging 3.01 turns. It measures what is done about distress, not whether a refusal string is produced.
• Metadata: Five axes are 23 SSH subcategories, age (U18/O18), turns, harm intent (borderline or explicitly harmful), and progression stage; plus signal directness (implicit/explicit). Each subcategory has 20–41 items. Marginal distributions are in Appendix A.
• Criteria: Each item has 6–20 atomic criteria, averaging about 9.6, totaling 6,901. Each has one dimension, with integer weights from 1 to 20, indicating priority within that item only. The authors report 87% of criteria have weights above 5, with item weight sums from 25 to 291 (median 120). Not-applicable criteria are removed during editing, not kept as zero.
• Seven dimensions: Identification, Compassionate engagement, De-escalation, Actionable resources, Disclaimers, Moralization, Harmless response. Definitions are in Table 4. Dimension subscores are weighted pass rates within that dimension.

Writing, revision, and splitting

• Process: Licensed clinicians and crisis counselors write user turns and clinical notes; Gemini 3.1 Flash Lite drafts scales from notes, clinicians revise; the same model drafts reference responses from revised scales, clinicians revise again. Reference responses are not shown to models at inference.
• Revision extent: 659 items retained the pre-draft scale. The drafting model produced exactly seven criteria per item, one per dimension, with weights as a permutation of 1 to 7, summing to 28. Of 4,613 drafted criteria, only 531 were kept as-is; clinicians rewrote the remaining 88.5%, expanding these scales to 6,344 criteria, and added new ones on 69% of items, netting 1,731 additions. The authors still call the released scales clinically written.
• Menu constraint: The authors note the dimension structure comes from a fixed menu in the drafting prompt, so disclaimers appearing across the corpus is a tool artifact, not an item-by-item clinical finding. After editing, identification criteria cover 717/718 items, actionable resources 714, and both 713.
• Splitting and quality control: Public/private 359 each, stratified by subcategory, age, turns, intent, and progression stage. Heuristic checks and multi-level human review before inclusion. Multi-turn has a coherence gate: 18 items × 4 models, with a pass threshold of at least 90% for acceptable transitions; the authors state v1 meets this threshold.

Generation protocol and scoring

• Protocol: Single-turn uses a generic helpful assistant system prompt. Multi-turn fixes user turns, with assistant turns regenerated by the target model each turn; reviewers see the full dialogue and score only the last assistant response. Models do not see scales, reference responses, or clinical notes. Full runs are repeated 3 times independently, reporting the average of three.
• Decoding: Requests at temperature 0; the seven models that only accept temperature 1 are sampled at temperature 1, i.e., three GPT-5.6 variants and four Claude models. Output limits are mostly 4096 tokens; Muse Spark 1.3 and Kimi K3 are 8192, Qwen3.8 Max is 16384. Reviewers use 8192 tokens, temperature 0.
• Dialogue score: A response's score is

S(t,m,a)=∑_i w_i ℓ_i(y_t,m,a)/∑_i w_i

that is, the weight of criteria judged satisfied by majority vote, divided by the sum of weights for that item. Model score is the average over 718 items and 3 runs. Empty last turns score 0; empty attempts enter the three-run average as 0. After excluding criteria, remaining weights are renormalized.

• Two derived quantities: The identification–action gap is

G(m)=Pr[∃ i: d_i=resources,ℓ_i=0 ∣ ℓ_i=1 ∀ i: d_i=identification]

that is, the conditional proportion of dialogues where all identification criteria are satisfied but at least one actionable resource criterion fails. The choice–capability gap is the dimension's highest score minus the median score across 25 models. Task-level paired bootstrap is 10,000 times.

Jury and calibration

• Verdicts: Three LLM reviewers each give a binary verdict and one-sentence rationale; the panel label is majority vote. Appendix C gives full prompts. Partial satisfaction counts as unsatisfied; each criterion is judged independently. Parsing failures are retried; criteria that cannot be parsed are marked missing and excluded from that item's weight normalization; the authors state no criteria fell into this state in reported runs.
• Calibration: 75 items, seed 20260714, three licensed clinicians independently label the last turn, yielding 2,908 consensus labels. From all 455 three-model combinations of 15 candidate models, ranked by Cohen's κ against human consensus, claude-opus-4-8, llama4-maverick, and gemini-3.1-pro-preview were frozen.
• Agreement: This combination has κ=0.732, macro-F1=0.862; clinician inter-rater Fleiss κ=0.640. The authors state the panel can exceed human pairwise agreement because it is compared to consensus labels, so it is only diagnostic. Under Gwet AC1, the panel is 0.780 and clinicians 0.690. The nearest competing combination has Δκ=+0.0011, 95% interval [-0.008, +0.010]; after removing disclaimers, this panel drops to 14th of 455. The authors state this is a pre-committed choice, not the unique optimal panel.

Long-horizon appendix and multi-turn rules

• DistressBench-LH: 20 red-team transcripts, each with about 40 user–assistant turns, totaling 801 labeled turns. Assistant paths are all frozen to gemini-3.1-flash-lite. Labels are dialogue-level violation categories, no SSH scales, not in the main leaderboard, and not in reviewer calibration. The authors state that therefore models cannot be compared, nor can it be determined which turn the guardrail first failed.
• Scoring rules: Main results score only the last turn. Any-turn scoring is only a sensitivity analysis: harmless response requires every turn to be harmless, other criteria satisfied in any turn. The authors adopt last-turn scoring because the user receives the last response. Appendix B.9 explains that clinicians were also asked to score the last turn, so higher label agreement measures rule execution, not independent proof that the rule is the correct standard.

Experimental setup

• Models tested: The 25 models in Appendix G, each run 3 times on all 718 items. The list is GPT-5.6 Terra, Sol, Luna, GPT-OSS 120B, Claude Opus 4.8, Sonnet 5, Fable 5, Fable 5.1, Gemini 3.1 Pro, 3.5 Flash, 3.7 Flash, Muse Spark 1.2, 1.3, Muse Glimmer 30B, Llama 4 Maverick, Inkling, DeepSeek v4 Pro and Flash, Qwen3.7 Plus, Qwen3.8 Max, Kimi K2P6, K3, GLM-5, Mistral Large and Large 3.
• Corpus and metrics: 718 items, 6,901 criteria. Public/private 359 each, main leaderboard uses all 718. Primary metric is weighted criterion satisfaction rate. All met is the proportion of dialogues where every criterion is satisfied. The identification–action gap is defined only on dialogues where all identification criteria are satisfied. The choice–capability gap is the dimension's highest score minus the median score. DistressBench-LH is not in the main leaderboard.
• Reviewers: Frozen three-model panel is majority vote of claude-opus-4-8, llama4-maverick, gemini-3.1-pro-preview; the paper states these three are also tested models. Calibration set 75 items, three clinicians, 2,908 consensus labels, κ=0.732, macro-F1=0.862. The authors do not treat exceeding human Fleiss κ=0.640 as the panel being more expert.
• Decoding and intervals: Requests at temperature 0; the seven GPT-5.6 and Claude models are sampled at temperature 1. Empty last turns score 0. Task-level paired bootstrap 10,000 times, reporting 95% intervals. Rank correlations across three repeats are in Appendix B.6; the authors state no conclusion depends on a single run.
• Same-family reviewers: After removing reviewers from the tested model's own provider family, the top model is unchanged, and the main leaderboard vs. leave-one-out Spearman ρ=0.945 (Appendix B.6). The authors state no same-family advantage was found, including for top model Muse Spark 1.2.
• Drafting model: The drafting of scales and reference responses, and the frozen path for LH, is Gemini 3.1 Flash Lite, not among the 25 main leaderboard models.

Main results

Per Table 11 and Figure 2, weighted scores on 718 items, averaged over three runs, are as follows. The table lists only the extremes and some band boundaries; the omitted 18 models are in Table 11, including GPT-OSS 120B at 0.637 and Mistral Large 3 at 0.588.

| Model | Weighted score | 95% CI | All criteria met | | Muse Spark 1.2 | 0.884 | [0.875, 0.893] | 40.4% | | Muse Spark 1.3 | 0.849 | [0.839, 0.858] | 18.3% | | Muse Glimmer 30B | 0.806 | [0.796, 0.815] | 11.3% | | Inkling | 0.785 | [0.774, 0.795] | 4.6% | | Claude Opus 4.8 | 0.714 | [0.703, 0.726] | 2.3% | | Mistral Large | 0.591 | [0.576, 0.605] | 0.7% | | Llama 4 Maverick | 0.412 | [0.401, 0.424] | 0.2% |

• Bands: Paired bootstrap separates only 11 of 24 adjacent gaps, splitting the 25 models into 12 bands, with the two largest bands having 7 models each. The authors therefore report bands rather than strict rankings. The summary's median 71.4% corresponds to Table 11's 0.714.
• All criteria met: Median across 25 models is only 2.3%, second place 18.3%. The authors state most total scores come from partial scores across many dialogues; the top model is the only exception, which the authors say provides evidence that the scale can be satisfied by a deployed model, not only by clinicians holding the scale.
• Low scores and ceiling: The authors state Llama 4 Maverick's gap is concentrated in identification, with that dimension at 20.2%, while the corpus maximum is 93.3%. Clinical reference ceiling is 0.987. The authors state the top model still leaves 11.6% of weighted criteria, and the remaining error is not due to impossible criteria.

Dimension gaps: capability or choice

Table 1 compares median and highest models per dimension. Columns are rounded independently, so Δd may differ from the difference of the two columns by 0.001.

| Dimension | Median | Highest | Δd | Authors' reading | | Disclaimers | 0.083 | 0.846 | +0.763 | Choice | | De-escalation | 0.460 | 0.746 | +0.286 | Choice | | Actionable resources | 0.703 | 0.866 | +0.163 | Mixed | | Identification | 0.825 | 0.933 | +0.107 | Mixed | | Compassionate engagement | 0.852 | 0.969 | +0.117 | Mixed | | Harmless response | 0.945 | 0.970 | +0.025 | Saturated | | Moralization | 0.982 | 0.992 | +0.010 | Saturated |

• Authors' reading: Moralization and harmless response have Δd no more than 0.03, which the authors read as a common frontier. De-escalation and disclaimers spread more. The authors state that a dimension where a deployed model reaches ten times the median is unlikely to be constrained by current model capability limits. The summary writes de-escalation as 74.6% vs. median 46.0%, and disclaimers as 84.6% vs. median 8.3%.
• Disclaimer distribution: 22/25 models fall between 5.4% and 10.9%; one model reaches the table's highest value, and two others exceed 35%. On this dimension, panel vs. clinician labels κ=0.554, higher than clinicians' own 0.274. The authors also note the drafting prompt puts this criterion in every item, so the dimension's ubiquity is a tool artifact.
• Unseparated factors: The authors state one model accounts for five of the seven dimension maxima; the data cannot distinguish whether this is capability or longer responses. This does not affect the decomposition, because the decomposition only requires that some model performs the behavior.

Identification without referral

• Conditional proportion: G(m) is computed only on dialogues where all identification criteria are satisfied, with the denominator being the 713 items that have both types of criteria. Across all models, 12,824 of 36,337 identified dialogues have at least one resource criterion failure, 35.3%. The authors pool by dialogue because identification frequencies vary greatly across models.
• Extremes: Section 4.6 states the strongest model fails to refer in 15.9% of its 1,786 identified dialogues, and the weakest in 69.6% of 322. Figure 3 labels 69.6% on Llama 4 Maverick and lists the 10 models with the highest decline rates, ending with Gemini 3.1 Pro at 36.1%.
• Authors' interpretation: Harmless response and moralization can pass while de-escalation still fails, yielding responses that are unobjectionable but ineffective. Cases where empathy passes but identification fails are rarer. The authors believe the difficulty is mainly not misreading, but identification not translating into action.

How models view their own omissions

• Probe: 22/25 models, 120 items, 3,922 criterion units. Eligibility: the model is judged failed by the panel on attempt 1, while the clinical reference is judged satisfied on the same item. Graded responses were collected before the probe. Three models whose endpoints are offline were not probed (Appendix E.7).
• Decomposition: Llama 4 Maverick scores 0.656 on the wide-gap contrast and is excluded. Table 13 therefore covers 21 models, 3,687 units. Criterion disagreement accounts for 85.0% [81.9, 87.9] of abandoned weight, and discrimination failure 4.1% [3.1, 5.0]. The authors believe the remaining gap is mainly disagreement about what a response should contain, not inability to write it.
• Content of disagreement: In free text, explicit acknowledgment of omission: actionable resources 39.4%, identification 34.3%, de-escalation 33.6%, disclaimers 17.9%, empathy 5.3%. After replacing with another model's compliant response, preference for the alternative rises from 10.4% to 34.0%; same-family 27.7%, cross-family 33.9%. The authors state the design does not answer which judgment is clinically correct.

Other ablations and analyses

• Stratification: U18 0.707, O18 0.710; borderline intent 0.716, explicitly harmful 0.707; progression stage from planning 0.700 to action 0.719. The authors report only 2/25 have test-supported age differences, both favoring adults, with magnitude about 0.03 (Section 4.4).
• Multi-turn: Multi-turn 0.685, single-turn 0.742, with the authors giving a gap of 0.058. Any-turn scoring additionally marks 13.5% of criteria, and pooled κ with clinician labels drops from 0.810 to 0.612. Turn-level safety annotation Fleiss κ is about 0.03, which the authors retract and do not use for any reported metric (Section 4.5, Appendix B.4).
• Disclaimer ablation: After removing this dimension and renormalizing, models rise by 0.002 to 0.039, with rankings and bands unchanged. Muse Spark 1.2 rises only 0.002 (Appendix E.5).
• Source of reference ceiling: 0.987 vs. top model 0.884, with the authors giving headroom of 0.103. References written from scratch are 0.995, and model-drafted-then-revised are 0.986. The authors state this is contrary to the prediction of a self-identification artifact, and the ceiling only indicates the scale is satisfiable (Section 4.7).
• Empty responses: Claude Fable 5 has 162 empty last turns in 2,154 task attempts (7.5%); escalation-mode subcategory 34.2% (39/114), suicide plan and preparation 0.8% (1/120). On the 668 items where both answered, Fable 5.1 is 0.726 and Fable 5 is 0.749, with difference interval [-0.031, -0.016] (Appendix E.6).
• Human–human agreement on identification: Clinician agreement rate 75.1%, the lowest of the seven dimensions; model median 82.5%. The authors believe disagreement comes from the response-side criterion not being specified: must the pain be named, or can it be implicit in the actions (Introduction, Appendix B.5).

Limitations and future directions noted by the authors

• Language and region: v1 is English-only, with resource standards centered on the United States. The authors state they do not claim cross-regional validity, nor do they claim relevance to real-world harm outcomes (Section 5.1, Appendix F.1).
• Simulated conversations: The tasks are expert-written simulations, not transcripts of real crisis chats. The authors state they do not claim clinical generalizability (Section 5.1, Appendix F.1).
• Scale source and reference asymmetry: The scales and reference responses were drafted by a model and then clinically revised; the authors state they are not fully handwritten tools independent of language models. The reference responses were written with the scales visible, while the model responses were not; the authors use the upper bound as satisfiability, not as a human baseline (Appendix F.1).
• Disclaimers and length: Each question has a disclaimer standard because the drafting prompt provided a fixed menu of dimensions. Response length was not controlled, and longer responses have more opportunities to satisfy atomic standards (Section 5.1, Appendix F.1).
• List and long-horizon appendix: Inferences about relative capability are sensitive to the breadth of the list. DistressBench-LH covers 5 of 23 subcategories, the assistant path is fixed to one model, and labels are conversation-level; the authors state this does not support cross-model comparisons, turn-level localization, or merging with the main leaderboard weighted scores (Appendix F.1).
• Future plans: The authors plan regional extensions with local hotlines and cultural subcategories, turn-level tools with anchored examples and adjudication, and a scored long-horizon track after regenerating the assistant side per model (Appendix F.2).

Experimental coverage

• Main evaluation: 25 models in Appendix G, all 718 questions, 3 runs per model; the main leaderboard uses all 718 questions, with public and private each having 359 (Section 2.1, Section 4).
• Reviewer calibration: Three-model majority vote, selected on 75 questions, three clinical staff, and 2,908 consensus items; candidate trio 455 (Section 3.3).
• Stratification: Reported by age, harm intent, progression stage, single-turn vs. multi-turn, and seven dimension subscores and 23 subcategories (Sections 4.2–4.5, Appendix E).
• Long-horizon appendix: 20 conversations, approximately 40 turns, 801 annotated turns, assistant path is gemini-3.1-flash-lite, not in the main leaderboard and not part of reviewer calibration (Section 2.5).
• Discriminative probes: 22/25 models, 120 questions. The paper states DeepSeek V4 Pro, DeepSeek V4 Flash, and Qwen3.7 Plus were not probed because endpoints returned HTTP 404; Llama 4 Maverick was excluded from the Table 13 breakdown due to a wide gap control of 0.656 (Appendix E.7).

DistressBench uses SSH dialogues written and revised by clinical personnel, scoring general chat models on how much necessary care was delivered according to weighted atomic criteria, rather than whether they refused.

When users disclose suicidal ideation or self-harm to chat models, the authors consider the expected behavior to be identification, calibrated empathy, de-escalation, and referral to human help. Refusal-based evaluations cannot detect inaction after recognizing a crisis. The corpus is 718 English dialogues across 23 subcategories; each item has seven-dimensional weighted criteria and a reference response, totaling 6,901 criteria. Scores are the proportion of weight judged as satisfied by majority vote. The jury was frozen after comparison with three clinicians on 75 items, κ=0.732.

• Overall score: The summary reports the highest-scoring model satisfied 88.4% of weighted criteria, median 71.4%, clinical reference 98.7%. The corresponding entry in Table 11 is Muse Spark 1.2 at 0.884, median 0.714. Dialogues where all criteria were met: median only 2.3% across 25 models.
• Action gaps: Harmless response median 94.5%, avoiding moralization median 98.2%; de-escalation median 46.0%, highest 74.6%; disclaimers median 8.3%, highest 84.6%. The authors consider the latter two more like undelivered product choices rather than current inability.
• No referral after identification: Across all models, 35.3% of fully identified dialogues had at least one resource criterion unmet. Section 4.6 states the strongest model still had 15.9% in its 1,786 identified dialogues.
• Multi-turn and empty responses: Multi-turn 0.685, single-turn 0.742. Claude Fable 5 attempted empty responses in 7.5% of tasks, with empty responses more frequent in the escalation-mode subcategory rather than the suicide-plan subcategory.
• Authors' conclusion: The authors argue crisis safety should be evaluated on delivered care, and treat de-escalation and resource referral as deployment requirements. The discussion maps the gaps between 88.4% and 98.7%, and between median 71.4% and 98.7%, to omissions in de-escalation, referral, and multi-turn support; the authors also state that at least one deployed model already delivers the first two. Most multi-turn tasks have only three turns; the authors believe long-term support needs a separate evaluation track.