Open questions for chatbots and mental health

Core problem: general-purpose chatbots are widely used for emotional and mental health support, but their impact on mental health is difficult to evaluate and improve, and researchers know neither its incidence nor its causal effects.

• Setting and importance: More and more users turn to LLM-based chatbots for emotional support; some users have experienced adverse outcomes such as delusional spirals and worsening crises, including suicide (§1). These consequences can arise whether or not users actively seek mental health support.
• Shortcomings of existing methods: The authors argue that traditional benchmarks relying mainly on static datasets cannot establish chatbot safety; new simulation methods still need validation; and analysis of real products is constrained by privacy concerns (abstract, §4).
• Core observation: Better benchmark performance does not necessarily make systems safer, nor does it prove they will improve well-being (§4); evaluation, real-world evidence and data governance are interdependent and must be addressed together (abstract, §8).
• Paper's contribution: A commentary agenda paper that organizes open questions into three areas—controlled evaluation (§4), real-world evidence (§5) and policy (§6)—distilling 12 recommendations and 12 open questions, tagged by five stakeholder groups: researchers, AI developers, clinicians, users and policymakers (§1).

Review of mental health chatbots

• Hua et al. [63]—distinguish controlled tests of chatbot behavior, usability pilot studies and clinical efficacy trials; Guo et al. [59] and other reviews cover LLM applications in digital mental health and the clinical evidence gaps.
• De Choudhury et al. [40], Lawrence et al. [81]—examine how LLM applications are embedded at different levels of care, involving accessibility, equity and well-being; these reviews all emphasize unresolved privacy and ethical issues, and that studies often lack longitudinal validation [106, 165].

Behavioral standards and oversight frameworks

• Grabb et al. [58]—propose standards to limit chatbot autonomy; Yang et al. [176]—argue for clinical rigor such as randomized trials; Obradovich et al. [119]—propose oversight bodies.
• Parks et al. [135]—call for safety and evidence standards, validation of LLM judges and simulated users, and consumer participation; Lee et al. [82]—argue for transparent, standardized evaluation and discuss accountability; Zuromski et al. [181]—propose a clinical research agenda for adolescent suicide risk; Suh et al. [159]—propose interaction patterns that may amplify users' psychological vulnerability. The paper notes that how to implement and evaluate these standards in deployed products remains an open question.

Mental health evaluation

• Clinical competency tests: psychopharmacology [47], mental health Q&A [83], psychotherapy technique alignment [35, 118], crisis labeling and triage [23, 172].
• Multi-turn trajectory evaluation: single-turn evaluation with real transcripts as prefixes [73, 105], synthetic conversation histories [89], simulated users and LLM judges [15, 17, 71, 137, 139, 171]; Weilnhammer et al. [171] used clinicians to validate behavioral judgments, but the paper notes that this does not establish downstream effects on users.
• Evaluations targeting specific conditions mostly focus on crisis and suicidal ideation [72, 137, 154, 172], with some on psychosis- and delusion-related harm [105, 177]; the paper notes that these evaluations characterize behavior under the tested conditions but cannot by themselves establish mental health effects over time.

Positioning of this paper: The authors state that prior work has established evaluation standards and called for longitudinal evidence, but many open questions remain about how to generate and independently validate such evidence for general-purpose chatbot products; this paper focuses on general-purpose chatbots, even when users are not actively seeking mental health support (§2).

The paper is a commentary that conducts no new experiments, but instead proposes a research agenda by selectively synthesizing work in AI evaluation, HCI, psychology, mental health and policy along with the author team's experience (§3). The authors explicitly state that no formal literature selection or synthesis process was used, and that the selection reflects the team's research, professional experience and discussions (§3).

Controlled evaluation (§4)

• Appropriate behavior and measurement: The authors argue that appropriate behavior depends on conversational context, user intent and vulnerability, and should be defined with clinician and affected-user involvement (Recommendation 1); benchmarks should specify constructs, test scoring reliability and corroborate with related measures (Recommendation 2); evaluation metrics should be linked to validated real-world outcome measures, using randomized experiments or causal identification where feasible, and preregistering longitudinal studies (Recommendation 3).
• Interaction trajectories: Evaluation should cover multi-turn, multi-session interactions, with systems varying conversation-history features (such as when distress is disclosed) and each condition repeated (Recommendation 4); behavioral judges should assess cross-turn patterns rather than single-turn replies.
• User simulation: Simulation targets can be specific users, user categories or events; methods include scripted replay, sampling real queries, prompting, fine-tuning, pretraining and inference-time parameter interventions; the core limitation is fidelity, and the authors argue it should be justified holistically by combining target, evaluation, reference data, comparison features and similarity metrics (Recommendation 5), and testing whether simulated evaluations predict deployed behavior (Open Question 2).
• Model internals: Activation probes can detect representations of depression, suicidal ideation and the like [180], but the authors note that such work mostly uses text labels rather than long interactions, and has not been validated against independent clinical assessments; on intervention, they discuss whether activation steering can reduce traits such as sycophancy and its downstream effects (Open Questions 3, 4).

Real-world evidence (§5)

• Full-product evaluation: Chatbot products comprise multiple components such as system prompts, routing, safety classifiers, memory and personalization, retrieval tools, interface cues and human escalation (Figure 3); the authors argue for evaluating deployed products rather than only base models (Recommendation 6), and for setting standards for components and providing research APIs and component toggles (Recommendation 7).
• Human review and escalation: Analyze vendors' flagging, human review and referral practices, raising the question of which practices work and how often users receive care (Open Question 5).
• Longitudinal data collection: Discuss sampling (balancing targeted recruitment of rare high-risk cases with representative sampling, Recommendation 8), data export and interoperability (Recommendation 9), privacy and access (limitations of automated de-identification, Open Question 6; tiered review by sensitivity, Recommendation 10).
• Clinical research: Chat logs can serve as evidence of psychopathology (Recommendation 11), raising how chatbots should participate in crises (Open Question 7) and measuring the clinical impact of individual product features (Recommendation 12).

Policy (§6)

• User protection: Review legislation in multiple US states, the EU, China and the UK as of October 2026, raising the impact of enacted interventions (Open Question 8), how to protect adults whose needs differ from minors' (Open Question 9), and what transparency and evaluation can support external oversight (Open Question 10).
• Liability and standards of care: Discuss what standards should apply to general-purpose chatbots; the authors argue standards should track actual use rather than advertised use only (Open Question 11).
• Research data governance: Drawing on governance precedents from genomic and clinical databases, discuss how to organize and share chat logs while protecting privacy (Open Question 12).

Experimental Setup

• This paper is a commentary agenda paper and reports no new experiments (§3 explicitly states that no formal literature selection process was used and that the contribution is a synthesis of multidisciplinary perspectives).
• The paper cites findings from existing studies as evidence; the following is paraphrased by theme, and all numbers come from the cited works rather than from experiments in this paper.

Key Empirical Findings Cited

• UI vs. API differences: Kirgis et al. [77] manually tested and found that ChatGPT's UI and API give different responses to delusional behavior; Wang et al. [168] found that the UIs of Claude, Gemini, and ChatGPT show increased bias and sycophancy relative to their APIs; Johnson et al. [71), using mental-health simulated users, found little difference between the UI and API of Claude and ChatGPT (§5.1). The authors use this to argue for evaluating the complete product.
• Simulation fidelity: Johnson et al. [71] found that simulations conditioned on production data more closely match real user behavior; Bentley et al. [17] asked licensed clinicians to rate the realism of simulated users, and the authors note that such ratings alone cannot establish that simulations match real users' language or behavior (§4.2.1).
• Activation probes: Work by McKenzie et al. [98] shows that activation probes can generalize to medical interactions at much lower estimated compute, but on a held-out mental health dataset they are less accurate than most text-based monitors (§4.3, open question 3).

Vendor Disclosures and Demographic Figures

• Suicidal ideation indicator: OpenAI estimates that the suicide indicator appears in 0.15% of weekly active users' conversations and 0.05% of messages, but does not report human-reviewed statistics (§5.2).
• Disclosure rate: A meta-analysis estimated that only 45.9% of people with suicidal ideation disclosed the ideation [60] (§5.4).
• Sampling bias: In a survey of native English-speaking US users on Prolific, fewer than 25% of respondents provided chat logs [178]; one study estimated that under typical exclusion criteria, 79% of people with schizophrenia spectrum disorders would not meet inclusion criteria [161] (§5.3).

Other Ablations and Analyses

• The paper has no ablation experiments; §6.1 paraphrases Transluce's mental health evaluation of Anthropic and OpenAI chatbots [71]: vendors provided private production conversations and anonymized features, and the authors note that this protects user privacy but limits independent verification.

Limitations noted by the authors and future directions

• Review is not systematic: The literature review and the selection of open questions and recommendations are not systematic, but shaped by the team's composition and positions (§7).
• Team composition: The author team includes no one with lived experience of chatbot-use-related harm; the recommendation for substantive user and survivor involvement in research applies equally to the team's future work (§7).
• Not empirically validated: The authors did not empirically validate the proposed research and policy directions, nor establish which are most effective or feasible (§7).
• Focus on risk: The paper focuses mainly on risks and harms and does not establish the incidence or causal effects of harm; public reports cover only some harm types and may miss unreported harms and interactions with chatbots (§7).
• Generalization and cultural differences: The range of mental health outcomes covered is broad, and recommendations for one setting may not generalize to others; cultural and linguistic differences in mental health presentation were not systematically examined (§7).
• Product changes: Chatbot products are changing, and the analysis relies on public descriptions that may be incomplete (§7).
• Future directions: The 12 open questions (such as whether simulated evaluations predict deployed behavior, the limitations of automated de-identification, the impact of enacted interventions, and research data governance) constitute the research agenda proposed by the authors (§4–§6).

Scope of experimental coverage

• The paper conducts no new experiments; all empirical figures are cited from existing research or public vendor materials (§3, §5).
• The cited descriptions of product practices cover two vendors, OpenAI and Anthropic, as of September–October 2026 (§5.2, §6.1).
• The policy review covers legislation and proposals in multiple US states, the EU, China and the UK (§6.1).
• The paper reports no systematic search process or inclusion criteria for the cited studies (§3).

This is a commentary/agenda paper on general-purpose chatbots and mental health, written by an author team with backgrounds in AI evaluation, HCI, psychology, clinical psychiatry and technology policy (§3).

• Problem: chatbots are widely used for emotional support, but risks such as sycophancy, delusion reinforcement and inappropriate crisis response are hard to study; their incidence and causal effects are unknown, and static benchmarks cannot establish safety (§1, §4).
• Method: a selective synthesis of multidisciplinary literature and team experience, organized into three areas—controlled evaluation, real-world evidence and policy—proposing 12 recommendations and 12 open questions (§1).
• Controlled evaluation points: benchmarks should specify constructs, measure reliability and be linked to real outcomes; evaluation should cover multiple turns and multiple sessions; user simulation needs demonstrated fidelity; activation probes are less accurate than most text monitors on held-out mental health data [98] (§4).
• Real-world evidence points: complete products should be evaluated, not just base models—cited studies show UI and API behavior can differ [77, 168]; OpenAI estimates suicide indicators appear in 0.15% of weekly active users' conversations, but the manual review statistics are not public (§5).
• Policy points: as of October 2026, multiple jurisdictions have legislated, but many protections cover only minors; the authors argue standards should track actual rather than advertised use, and research data governance can draw on genomic database precedents (§6).
• Authors' conclusion: the three types of work are interdependent—evaluation needs real-world evidence for validation, evidence production depends on governed data access, and evidence should feed back into user protection policy; the authors argue that coordinated research efforts can produce substantial real-world impact (abstract, §8).