How Safe Are ChatGPT's Health Recommendations? Evidence from Real User Conversations in Four Global South Countries

Safety evaluation of LLM health advice largely relies on synthetic data and fictional personas, lacks testing on real user dialogues in the Global South, and isolated evaluation ignores potential risks arising from user history context.

• Lack of real-scenario evaluation: The authors state ChatGPT has over 700 million weekly active users globally, adoption is growing rapidly in low- and middle-income countries facing strained healthcare resources, and model advice is more likely to substitute for rather than supplement professional care, yet existing safety evaluations almost all rely on small clinician-written cases, synthetic prompts, or simulated patients.
• Limitations of isolated safety evaluation: The authors point out that the same health advice is harmless to a healthy adult but may pose risk to users with underlying conditions such as ulcers or surgical history; traditional evaluation compares a single reply against fixed criteria and cannot perceive physical conditions disclosed in earlier turns or prior conversations.
• Difficulty of obtaining real dialogue data: Health consultations are highly private data; platforms typically release only aggregated analyses without demographic attributes, and external researchers previously had difficulty obtaining real health dialogue records linked to actual age and gender.
• Research goals and core work: The authors collect real dialogues from India, Nigeria, Brazil, and Pakistan through a user-consented donation mechanism, build a health dialogue corpus with self-reported age and gender, and compare reply safety under two settings: without context and with the user's cumulative history profile.

• Large language models as health resources and clinical performance

  • Costa-Gomes et al. (2026)—analyzed over 500,000 Copilot health dialogues and found about one in five involves the user's own symptoms or conditions and one in seven involves the health of someone being cared for; the authors note this study records what users actually ask about but explicitly lists safety evaluation as future work.
  • Ramaswamy et al. (2026)—evaluated ChatGPT Health triage advice on structured clinical case tests and reported it underestimated risk in 52% of emergency cases; the authors note its clinical evidence comes from physician-written fictional cases.

• Context-dependent safety evaluation

  • Kempermann et al. (2026)—proposed that when user vulnerability is considered in evaluation, health advice scores drop from 5/7 to 3/7; the authors note this study relies entirely on synthetic user profiles and prompts.
  • In et al. (2025)—reported in a large-scale evaluation that twenty major LLMs failed to meet safety standards for specific users; the authors note this evaluation is also built on synthetic profiles and prompts.

• Demographic bias in medical LLMs

  • Omar, Soffer et al. (2025)—analyzed 1.7 million model outputs across 32 sociodemographic variants and reported that identity labels alone change triage, testing, and referral advice while clinical facts remain unchanged; the authors note this study measures bias by substituting demographic labels in fictional cases, a counterfactual construction rather than observation of real users.

• LLM health risks in the Global South

  • Chen et al. (2025)—warned that LLM development and deployment are concentrated in high-income countries and that inaccurate advice may worsen health disparities in low- and middle-income countries; the authors note this line of work is largely conceptual.
  • Si et al. (2024)—conducted simulated patient studies for common diseases in low- and middle-income countries and reported ChatGPT recommended unnecessary or harmful medication in 23 of 27 trials; the authors note this is still based on simulated tests.
  • Birk et al. (2026)—called attention to the fact that single-turn safety does not imply conversation-level safety and that digital mental health needs empirical evidence from low- and middle-income country settings.

• Benchmarks and comparison basis: This study uses the twelve-category health intent taxonomy of Costa-Gomes et al. (2026), and safety scoring rules adapted from the seven-point safety scale of Kempermann et al. (2026).

• Positioning relative to this paper: The authors state that prior work on contextual safety, medical equity, and AI risk in low- and middle-income countries is mutually independent and relies on synthetic or descriptive evidence; this paper evaluates the safety of conversational health advice using real user dialogues with known demographic information.

The authors propose an evaluation framework combining unsupervised topic discovery, LLM intent classification, persistent health feature extraction, and dual-channel safety scoring.

Data Collection and Preprocessing Pipeline

• User recruitment and dialogue donation: The authors recruited 1,252 active users in India, Nigeria, Brazil, and Pakistan through the Clickworker crowdsourcing platform (December 2025 to February 2026), collected their complete dialogue histories obtained via OpenAI's data export feature (202,590 dialogues from December 2022 to February 2026), and removed personally identifiable information via a local script before upload.
• Topic discovery and filtering: Used the gemini-embedding-001 model to extract 3072-dimensional embeddings of the first ten turns of each dialogue and applied L2 normalization; fitted MiniBatch K-means with k=500 separately in each country, then merged into about 50 macro-topics via hierarchical clustering; generated labels with BunkaTopics, and after manual review filtered dialogues on health-related topics.

Health Intent Classification and Validation

• Intent taxonomy: Uses the twelve-category health intent taxonomy of Costa-Gomes et al. (2026) (including health information and education, symptom questions, condition information and care, fitness lifestyle and guidance, mental health, etc.).
• Message-level classification and aggregation: Uses GPT-4o-mini as the classifier, labeling messages with the first five turns as context, then determining the primary dialogue-level intent by majority vote.
• Classification consistency validation: Randomly sampled 25 dialogues per country (100 total); the authors manually labeled messages under double-blind conditions and report overall agreement of 82% and Cohen's kappa of 0.78.

User History Health Profile Extraction

• Persistent health feature extraction: Uses GPT-5-mini to analyze each user dialogue, extracting only persistent physical or mental health features the user mentions (such as chronic conditions, dietary habits, medication history, etc.) according to the 12 health-related attributes in the Zaman and Garimella (2026) personal information taxonomy, filtering out one-off events and personal identifiers.
• Temporal cumulative construction: Orders all of a user's dialogues chronologically; for a target dialogue, the user profile is the deduplicated union of features extracted from all prior dialogues, simulating the background information the system would have up to that point.

Context-Blind and Context-Aware Safety Scoring

• Scoring criteria and judge model: Uses Gemini 3.1 Pro Preview as an independent review model, applying the seven-point safety scale of Kempermann et al. (2026), evaluating three dimensions—likelihood of harm (1–7), potential severity of harm (1–7), and adequacy of precautions (1–7)—and aggregating into an overall safety score from 1 to 7. The dialogue-level score is defined as the minimum safety score across all scored messages in the dialogue:

S_conv = min_{m ∈ M} S(m) One-sentence explanation of meaning: the dialogue-level safety score equals the minimum of the safety scores of all scored messages in that dialogue.

• Dual-channel evaluation setup:
  1. Context-Blind: inputs the first five turns as context and evaluates the last AI reply, without providing any user background beyond the dialogue.
  2. Context-Aware: in addition to the dialogue context, provides the user's cumulative persistent health profile up to that dialogue, requiring the judge to assess whether the reply is safe for that specific user in the current situation.
• High-risk threshold: overall scores of 3 or below are defined as the high-risk range (≤ 3), and scores of 4 and above as the low-risk range.

Experimental Setup

• Models under test: The evaluation target is ChatGPT as invoked by users in real interactions (covering GPT-3.5, GPT-4o, GPT-5, GPT-5.1, GPT-5.2 among 14,420 single-model health conversations, as well as lightweight variants GPT-4o-mini and GPT-5-mini).
• Dataset and sample size: Based on 202,590 real conversations donated by 1,252 users in India, Nigeria, Brazil, and Pakistan (December 2022 to February 2026), 15,555 health conversations were filtered (754 users contributed at least one health conversation); context-aware analysis covered 5,657 conversations from 427 users with historical profiles.
• Intent classification and auxiliary tools: A twelve-category health intent taxonomy was used, with GPT-4o-mini combining the first five turns of interaction for message-level intent labeling with majority voting; gemini-embedding-001 was used to extract embeddings; Claude Sonnet 4.6 was used for spot-check evaluation of cluster label quality; GPT-5-mini was used to extract twelve types of persistent health features to build user profiles.
• Metric definitions: A 1–7 overall safety score was used (1 = least safe, 7 = safest), with scores ≤3 classified as high-risk; three sub-dimensions include Likelihood of Harm (1–7), Severity of Potential Harm (1–7), and Safeguard Adequacy (1–7); the missed rate is defined as the proportion of conversations rated low-risk without context but turning high-risk under context-aware evaluation.
• Review method and judge model: Gemini 3.1 Pro Preview was used as an independent LLM judge, scoring messages according to the seven-point standard of Kempermann et al. (2026); the overall conversation score is the minimum score among all scored messages it contains.
• Baselines and test conditions: Two evaluation settings were compared: Context-Blind (only the first five turns of conversation provided) and Context-Aware (additionally providing the user's historical cumulative health profile); the paper did not conduct repeated sampling experiments.

Main Results

According to Table 7, the safety score comparison for conversations with cumulative health profiles in the four countries under Context-Blind (CB) and Context-Aware (CA) is as follows:

Swipe horizontally to read the table

• Safety scores systematically decline after introducing user context: The authors state that after providing users' cumulative health profiles, the mean safety score of conversations in the four countries dropped by 0.58–0.73 points (Wilcoxon signed-rank test all p < 0.001), the proportion of high-risk conversations increased by 13.4–16.6 percentage points, and the context-aware high-risk rate was 1.35–1.64 times the context-blind rate.
• Isolated evaluation has a stable and consistent missed rate: The authors point out that among conversations judged low-risk under context-blind evaluation, 25.2–26.3% were actually high-risk after combining the user's personal background, and this missed proportion remained similar across countries, intent compositions, and baseline risk differences.
• Asymmetric rating revisions and large downgrades: The authors report that after adding context, 34.3–41.3% of conversations had their scores lowered, while only 11.5–14.3% were raised; among 1,757 conversations that received a perfect score of 7 under context-blind evaluation, 11.4% fell into the high-risk range after learning the user's profile.

Intent Types and Risk Distribution Without Context

• What was tested: Analysis of the distribution characteristics of 15,555 health conversations under intent classification and the high-risk rate without context (Figure 1, Figure 8, Figure 10, Table 5).
• Results: Health information and education had the highest proportion (India 31.0%, Nigeria 48.9%, Brazil 28.3%, Pakistan 31.2%); clinical intents had the highest high-risk rates (condition information and care 39.8–47.7%, symptom questions 33.4–40.3%), while general health information was only 7.9–11.9%; at the user level, more than half of users in all four countries (51.6–61.7%, per Figure 9) encountered at least one high-risk response.
• Authors' interpretation: The authors point out that risk is concentrated in clinical consultation intents involving users' own symptoms, medications, and conditions; the association between intent type and safety rating was statistically significant in all countries (p < 10^-20, Cramér's V = 0.26–0.36), making it the strongest factor associated with risk.

Reversal of Intent Risk Ranking Under Context-Aware Evaluation

• What was tested: Analysis of the magnitude of risk changes across intent types after introducing user health profiles (Figure 2, Section 5.4).
• Results: Fitness lifestyle and guidance showed the largest increase in high-risk rate in all four countries (+26.1 to +30.0 percentage points, mean score drop of 1.05 to 1.16 points), with high-risk rates rising to 54.8–68.0%; mental health followed closely (increase of +14.3 to +23.6 percentage points), becoming the intent with the highest high-risk rate in India (from 40.7% to 61.7%); clinical intents that were already high-risk without context showed the smallest increases (+5.6 to +16.4 percentage points). Multivariate logistic regression with control variables showed odds ratios for crossing into high-risk of 3.5 for mental health, 3.2 for fitness, 2.0 for condition care, and 1.9 for symptoms (relative to general health information, all p < 0.001).
• Authors' interpretation: The authors argue that seemingly benign fitness and psychological counseling intents reveal hidden risks when combined with users' underlying conditions or trauma backgrounds, changing the risk ranking derived from isolated evaluation.

Safety Performance Across Model Version Evolution

• What was tested: In 14,420 single-model health conversations, tracked safety changes from GPT-3.5 to the GPT-5 series (Figure 17, Appendix J).
• Results: Along the flagship model release order (GPT-3.5 to GPT-5.1), the high-risk rate of single-model conversations in the four countries increased monotonically (India from 10.9% to 28.9%, Brazil from 6.2% to 34.6%, Nigeria from 4.6% to 30.4%, Pakistan from 22.2% to 37.1%, Spearman rho = -0.055 to -0.215, all p < 10^-4, Cramér's V = 0.10–0.20); lightweight variants had higher risk, with GPT-5-mini having a high-risk rate of 30.1–40.6% across the four countries (higher than GPT-5's 19.0–24.3%); the latest model GPT-5.2 showed a decline in high-risk rate in India (13.8%) and Nigeria (10.0%), but not in Brazil (29.7%) or Pakistan (23.7%).
• Authors' interpretation: The authors argue that large language model version upgrades did not bring improvements in the safety of real health conversations; possible reasons include new models reducing refusals, longer and more engaging conversation turns, and the pursuit of helpfulness competing with the willingness to stop and refer.

Other Ablations and Analyses

• Qualitative risk theme distribution: Among 128 high-risk conversations, inappropriate or harmful advice accounted for 47.7% (61 cases), lack of professional referral 43.8% (56 cases), harmful treatment or remedies 25.0% (32 cases), reinforcing harm or dependence 25.0% (32 cases), and unsafe diagnosis 25.0% (32 cases) (per Table 1).
• Sensitivity of safety dimension changes: After introducing user profiles, the likelihood of harm dimension changed most sensitively (34.8–42.0% of conversations judged to have increased likelihood of harm, mean score increase of +0.55 to +0.73 points), the severity of potential harm dimension changed least (mean score increase of +0.32 to +0.39 points), and safeguard adequacy mean scores decreased by -0.39 to -0.49 points (per Figure 16 and Appendix I.1).
• User vulnerability score regression analysis: In 5,657 conversations with profiles, regressing the overall safety score change on changes in the three dimensions yielded a coefficient of determination R^2 = 0.875 (per Appendix I.1).
• Age and gender context sensitivity gradient: After controlling for intent and context-blind scores, conversations from users over 36 years old had an odds ratio of 1.46 for crossing into the high-risk range compared to users aged 18–25 (p = 0.004); in Brazil, the new high-risk rate for women was 22.8% versus 12.0% for men (per Section 5.4).
• Gender differences in consulting on behalf of others: In Nigeria, women had an odds ratio of 10.6 for consulting on behalf of dependents compared to men (11% vs 1%), Brazil 6.5 (6% vs 1%), Pakistan 2.6 (9% vs 4%), and India 0.98 (3% vs 3%) (per Table 5).
• Same-user cross-version performance comparison: Among 240 users who asked both GPT-5 and GPT-5.1, the high-risk rate for GPT-5 conversations was 19.0%, and for GPT-5.1 conversations it was 27.8% (per Appendix J).

Limitations and future directions noted by the authors

• Sample representativeness limitation: The authors note in Limitations that participants came from a convenience sample on the Clickworker crowdsourcing platform, with samples in India and Pakistan skewing younger and predominantly male, reflecting active, digitally literate ChatGPT users, and cannot serve as national-level estimates of health question prevalence.
• Large model judge error and lack of real health outcomes: The authors note in Limitations that safety scores were assessed by a large model judge based on public standards, and although manual checks on a sample of labeled conversations support the precision of high-risk classifications, the judge may still have errors, and the study observed safety risk scores rather than actual health outcomes produced.
• Subjectivity of qualitative analysis: The authors note in Limitations that risk themes in the qualitative analysis were independently coded by a single author.
• Self-selection bias in context-aware analysis sample: The authors note in Limitations that the context-aware analysis covered only conversations with extractable profile histories (each accounting for 19–60% of health conversations), and this subset was riskier than the overall sample even under context-missing scores, so its high-risk rate cannot be directly generalized to all health conversations.
• Model version and time confounding: The authors note in Limitations and Discussion that the corpus spans three years and mixes different model versions, making it impossible to fully disentangle behavioral differences of the models themselves from the temporal context in which the service was provided.
• Privacy and surveillance risks of profile-based safety mechanisms: The authors note in Discussion that implementing personalized safety protections based on user profiles requires the system to retain more sensitive personal information, posing privacy inference and surveillance risks, and future research is needed on device-side storage, profile expiration dates, or user-controlled protection mechanisms.

Experimental coverage

• Countries and time range tested: The data covers 4 Global South countries—India, Nigeria, Brazil, and Pakistan—with conversations spanning December 2022 to February 2026.
• Model families tested: The single-model experiments covered GPT-3.5, GPT-4o, GPT-5, GPT-5.1, GPT-5.2, and the lightweight versions GPT-4o-mini and GPT-5-mini, without testing other commercial or open-source models.
• Sample size and number of repetitions: Included 202,590 donated conversations from 1,252 users, from which 15,555 health conversations were filtered for intent classification and context-missing scoring, of which 5,657 conversations received context-aware scoring; the paper does not report repeated scoring of individual conversations using different random seeds.
• Defense mechanism coverage: The experiments only evaluated the built-in protective behaviors of the ChatGPT platform at the time, without evaluating external detection guardrails, system prompt interventions, or adaptive defense mechanisms.
• Scale of manual verification: Manual verification of intent classification was based on 100 sampled conversations (25 per country); qualitative analysis of high-risk themes was based on 128 sampled conversations judged as high-risk.

• Research positioning: The authors conducted an empirical safety evaluation of large language model health advice based on real user-donated conversations in four Global South countries (India, Nigeria, Brazil, Pakistan).
• Core problem: Existing medical AI safety evaluations mainly rely on synthetic prompts and virtual profiles, detached from the actual usage of users in regions with scarce medical resources, and isolated evaluation of single conversations cannot identify personalized health risks arising from ignoring users' prior medical histories.
• Method design: Based on 15,555 real health conversations donated by 1,252 users, used unsupervised clustering and LLMs for health intent classification; extracted persistent health features from users' historical conversations to build cumulative profiles; used Gemini 3.1 Pro Preview for safety scoring and qualitative analysis under both context-blind and context-aware settings with user profiles.
• Main finding one: baseline high-risk distribution and intent characteristics: Without context, 20.6–26.3% of health conversations in the four countries fell into the high-risk range, and more than half of users in all four countries (51.6–61.7%, per Figure 9) encountered high-risk responses; risk was mainly concentrated in clinical intents such as symptom consultation (33.4–40.3% high-risk) and condition care (39.8–47.7% high-risk).
• Main finding two: context-aware evaluation reveals consistent missed rates and reversal of risk ranking: After introducing users' cumulative profiles, the proportion of high-risk conversations increased by 13.4–16.6 percentage points (per Table 7); among conversations judged low-risk under context-blind evaluation, 25.2–26.3% were actually high-risk after combining user background; seemingly benign intents such as fitness guidance and mental health showed the largest risk increases after learning medical history (fitness high-risk rate rose to 54.8–68.0%).
• Main finding three: model evolution did not reduce real-world risk: In single-model health conversations, the high-risk rate showed a monotonically increasing trend from GPT-3.5 to GPT-5.1 (India from 10.9% to 28.9%, Nigeria from 4.6% to 30.4%, per Appendix J), and lightweight variants had higher high-risk rates (GPT-5-mini at 30.1–40.6%).
• Authors' insights and recommendations: The authors argue that medical AI safety evaluation needs to incorporate real conversations and consider users' historical context; although models have personalized expression capabilities, they lack mechanisms to translate known background into safety protections and timely referrals; they call for post-deployment monitoring systems similar to pharmacovigilance, while also warning of the privacy monitoring costs that profile-based personalized safety may bring.