Safer for whom? A systematic investigation of AI safety in eating disorder interactions

Core question: When users ask about diet, exercise, or weight loss, can LLMs recognize implicit eating disorder (ED) vulnerability signals in those questions and give safe responses, and is the degree of protection consistent across users with different demographic characteristics?

• Setting and importance: People increasingly consult AI assistants about diet, nutrition, exercise, and weight loss. The authors note that ED patients may lack insight into their condition and treat disordered behaviors as steps toward a body goal, so they seek weight-loss and dieting advice in the form of ordinary questions; AI may inadvertently validate distorted beliefs, normalize dangerous behaviors, or give actionable guidance.
• Shortcomings of existing evaluations: The authors argue that existing evaluations focus on diagnostic accuracy, obviously harmful prompts, or synthetic scenarios, and provide limited understanding of whether models can recognize implicit vulnerability in natural interactions and consistently protect users; refusal-based defenses are also unsuitable in this domain, because refusing all food- and exercise-related questions would pathologize normal concerns.
• Core observation and hypothesis: ED vulnerability often manifests through implicit cues such as pro-anorexia community jargon, fear of weight gain, and guilt after eating; the authors worry that models will reproduce the stereotype that "ED mainly affects thin White women," thereby overlooking the vulnerability of male, Black, or Asian users and high-BMI users.
• What the paper proposes: An evaluation pipeline—97 real questions from online pro-anorexia communities are answered by three open-source models (GLM, Gemma, Qwen), each question paired with templated user bios varying race, gender, BMI, and explicit ED vulnerability cues, scored by an LLM judge using a clinical rubric extended from a seven-dimension ED harm taxonomy, with matched comparison isolating the effect of each user characteristic on response safety.

Safety risks in mental health human–AI interaction

• Cheng et al., Moore et al., Dohnány et al., etc.—studies of chatbots' sycophancy tendency: agreeing with users' maladaptive beliefs and, in extreme cases, amplifying delusions and psychotic symptoms; evaluations of responses to crises such as suicidal ideation and self-harm find large differences between models and repeated failures to recognize subtle risk signals.
• Yim et al. [29]—compared linguistic differences between AI-generated responses and human responses in online ED communities, and clinical review found that some responses reinforced ED behaviors and expressed implicit bias; the paper notes that this study did not systematically examine which model, question, or user attributes influence these risks.
• McCalla et al. [20]—used matched clinical vignettes to show that demographic information changes AI diagnoses of anorexia nervosa while symptoms are held constant; the paper notes that it did not examine whether such bias affects the safety of conversational responses.
• Winecoff and Klyman [28]—developed, with clinical experts, a seven-dimension taxonomy of ED harms from generative AI, on which this paper's rubric is built.

Mental health safety evaluation benchmarks

• CounselBench [19]—evaluates responses along six clinical dimensions using real patient questions, ratings from 100 mental health experts, and adversarial prompts.
• Arnaiz-Rodriguez et al. [1]—a crisis benchmark of 2,046 inputs across six categories that evaluates whether models both recognize a crisis and respond appropriately, finding weaknesses on indirect or ambiguous risk signals.
• KORA benchmark [17]—a safety benchmark for AI interactions with children (eight risk categories, three age groups); this paper's evaluation pipeline and scoring rubric are modeled on KoraBench.
• mPACT-ED-v1.0 [13]—a multi-turn ED dialogue benchmark developed with clinicians, in which humans assess whether models can recognize risk and give clinically appropriate guidance; the authors state that this paper fills a complementary gap: starting from real community questions and using controlled user bios for matched comparison.

Demographic bias in LLMs

• Poole-Dayan et al. [22]—prepended matched user bios to factual questions and found lower accuracy and truthfulness for users such as those with low English proficiency; this paper's bio design was inspired by it. Tamkin et al. [24] used a similar counterfactual design and found discriminatory patterns in 70 high-stakes decision scenarios.
• Hofmann et al. [16], Tonneau et al. [25], Dorn et al. [12]—studies of implicit identity cues: African American English triggers negative stereotypical judgments; how identity cues are expressed changes the direction and size of bias; models often misclassify LGBTQ+ reclaimed terms.
• Asad et al. [2], Warren et al. [26]—found body-size stereotypes in language models and anti-fat bias in generated images, respectively; the paper notes that evidence in conversational tasks remains limited, and in particular it is unclear whether disclosing body size changes the safety of the same request.

Positioning: The authors state that this paper differs from the above work in that it starts from real pro-anorexia community questions (rather than only clinician-written scenarios), evaluates along the Winecoff–Klyman seven dimensions, and uses matched user bios to simultaneously test implicit vulnerability recognition, response to explicit disclosure, and demographic bias.

Overall approach: following the KoraBench evaluation pipeline, a user bio generation step is added, and matched comparison (the same question paired with different bios) is used to isolate the effect of user characteristics on response safety; scoring is based on a clinical rubric extended from the Winecoff–Klyman seven-dimension ED harm taxonomy.

Question and bio construction

• Question source: Posts were collected from the ED Support Forum (an online forum for patients with ED and body dysmorphic disorder), and gpt-4o-mini was used to filter for posts asking questions or seeking advice (only the first 2,000 characters of each post were taken; see Appendix B.1 for the prompt); after randomly sampling 100 posts and removing posts with fewer than 10 words, 97 questions remained. Posts often contain community jargon, providing implicit cues to the author's vulnerability.
• Bio generation: Following the style of user profiles in the #edtwt community on X, Llama 3.3 70B was used to generate first-person bios (Appendix B.2), each 2–6 short lines and containing at least one non-experimental detail (hobbies, music, etc.).
• Attribute manipulation: Bios manipulate three attributes—race (White/Asian/Black), gender (female/male), and BMI (low/high, with both current and target BMI given); each bio discloses exactly two attributes, and pairwise crossing yields 16 variants (6 race×gender, 6 race×BMI, 4 gender×BMI; see Tables 1, 2); 70% of the generated bios contain explicit ED vulnerability cues.
• Condition scale: 97 questions × 16 bios + no-bio baseline = 1,649 conditions per target model.

Target models and prompting

• Three open-source target models: Gemma-4-31B-it (31B, dense), GLM-5.2 (753B, mixture-of-experts), Qwen3.6-27B (27B, dense).
• Prompting method: In the bio condition, the bio is placed in the user message before the question (rather than in the system prompt), simulating a user proactively sharing information in the conversation; the target model's system prompt requires answers of no more than 150 words (Appendix B.3).

Safety scoring

• Seven-dimension taxonomy: (1) generalized diet and health guidance; (2) encouraging disordered behavior; (3) supporting symptom concealment; (4) creating thinspiration; (5) reinforcing negative self-beliefs and comparisons; (6) promoting excessive focus on the body; (7) entrenching narrow or prejudiced views of ED. A single response can hit multiple categories.
• Four-level risk rubric: Each dimension defines four levels: high/medium/low/refusal (Appendix A). Three rubric principles: evaluate by the likely functional effect on a vulnerable user (not tone or intent); judge contextual appropriateness in light of implicit vulnerability cues in the question or explicit cues in the bio; disclaimers do not count as sufficient when the response still provides harmful advice.
• Scoring method: All interactions are single-turn; the judge model receives the original post, the full user message (including the bio), and the target model's response, and scores only the response. Low/medium/high risk are scored 2/1/0 respectively, and the safety score is defined as Safety Score = 1/(K · s^max) ∑_k=1^K s_k, where K=7 and s_max=2, i.e., 100% when all seven categories are low risk; refusals are not counted in the safety score and the refusal rate is reported separately.

Judging and validation

• Judge model: Claude (labeled Claude Opus 5.5 in Figure 1), which outputs a level and cited evidence for each dimension per the prompt in Appendix B.4.
• Human validation: One annotator with ED clinical expertise independently rated (blind) the three models' responses to 30 randomly sampled questions in the no-bio condition (628 category-level judgments in total), and agreement was measured with quadratic weighted Cohen's κ and the direction of disagreements.

Experimental Setup

• Models under test: Gemma-4-31B-it, GLM-5.2, Qwen3.6-27B, all open-source models.
• Data: 97 real questions from the ED Support Forum; each question paired with 16 profile variants plus a no-profile baseline, 1,649 conditions per model.
• Metrics: safety score (each of the seven dimensions scored as high=0/medium=1/low=2, then normalized; higher is safer); refusal rate reported separately.
• Evaluation method: Claude Opus 5.5 rated each dimension according to the Appendix A rubric; human validation was a blinded rating by one clinical annotator of no-bio responses for 30 questions × 3 models (628 judgments).
• Statistical tests: Wilcoxon signed-rank test for gender and BMI, Friedman test for race.
• Number of repetitions: not reported in the paper.

Main Results

According to Figure 2(a) (mean safety score, %) and Figure 2(b) (proportion of high-risk responses):

Swipe horizontally to read the table

Model	no-bio safety score	bio safety score	no-bio high-risk proportion	bio high-risk proportion
Gemma-4-31B-it	71.4	79.5	13%	6%
GLM-5.2	79.5	82.8	13%	7%
Qwen3.6-27B	61.1	67.7	20%	17%

• Differences between models: the authors state that GLM had the highest mean safety score, followed by Gemma and Qwen; adding a profile raised scores by 6–8 points, but Qwen with a profile still scored lower than GLM and Gemma without a profile.
• High-risk frequency: Figure 2(b) shows that high-risk responses were about twice as frequent without a profile as with one; the refusal rate across all responses was only 0.04%.
• Dimension differences: according to Figure 3, the Supporting Symptom Concealment dimension had the highest safety score (e.g., GLM bio 96.2), while Generalized Guidance and Encouraging Disordered Behaviors were lowest; after adding a profile, the Narrow/Stereotyped ED Views dimension score instead decreased (e.g., Gemma from 75.0 to 59.7).

Effect of Explicit ED Vulnerability Cues (RQ2)

• What was tested: profiles containing an explicit ED diagnosis or struggle statement were compared with the no-profile condition (Figure 4).
• Results: the authors report that explicit cues raised the mean safety score of the three models by 10–14 points; scores rose on all dimensions except Narrow and Stereotyped Views of EDs, which decreased (e.g., GLM from 80.7 to 68.7), and this pattern was consistent across the three models.
• Authors' interpretation: diagnosis disclosure led models to avoid giving harmful advice, but models often did not correct the narrow views expressed in the question itself; responses became more cautious without challenging misconceptions about "who gets EDs."

Matched Comparisons of Demographic Attributes and Body Size (RQ3)

• What was tested: safety score differences between matched profiles differing in only one attribute (Figure 5; positive values indicate the first group's responses were safer).
• Results: gender difference +3.2 to +4.3 points (female safer); White−Black +2.2 to +3.6, White−Asian +2.8 to +3.8; Black−Asian −0.4 to +1.3, the only comparison that was not significant (the paper's original text uses significant). BMI showed the largest difference: low BMI scored 14.9 points higher than high BMI for Gemma, 15.5 points higher for GLM, and 24.8 points higher for Qwen. All comparisons had p<.001 except Gender×GLM (p=0.002).
• Authors' interpretation: models responded more safely to female, White, and low-BMI users; the authors state this is consistent with the stereotype that "EDs mainly affect thin White women," and users who do not fit that image may receive less protective responses.

Evaluation Consistency Validation

• Consistency: the overall quadratic weighted κ between the LLM evaluator and the clinical annotator was 0.77 (Figure 6); highest was Reinforcing Negative Self-Beliefs (κ=0.82), lowest was Perpetuating Narrow Views (κ=0.54, moderate).
• Direction of disagreement: the two gave the same rating in 74% of judgments; the evaluator overestimated risk 7% of the time and underestimated risk 18% of the time (Figure 7), with underestimation ranging from 11% for Excessive Body Focus to 25% for Symptom Concealment.
• Authors' interpretation: the authors consider the consistency sufficient to support aggregate comparisons, but the evaluator's systematic tendency to underestimate risk requires further study.

Other Ablations and Analyses

• Model scale: the authors note that model size does not correspond to safety—under the bio condition, Gemma (79.5%) and Qwen (67.7%), which are of similar scale, differed by about 12 points, while GLM, about 25 times larger, scored only about 3 points higher than Gemma (82.8%) (Section 6).

Limitations and Future Directions Noted by the Authors

• Limited human validation (Section 6): only one clinical annotator labeled a subset, so systematic differences between human and evaluator judgments cannot be distinguished from variation among individual annotators; the authors plan to recruit multiple clinicians, measure inter-annotator agreement, build consensus labels, and test whether the evaluator's pattern of underestimating risk is stable across annotators.
• Single-turn evaluation (Section 6): only single-turn interactions were evaluated, because real multi-turn conversations with reliable user context were unavailable; in pilot experiments, simulated users caused conversations to drift from the original question. The authors state that the results characterize the safety of initial responses and may overestimate safety in deployment; they plan to evaluate longitudinal interactions with real participants under IRB approval.
• Model selection and scale (Section 6): the three models differ in scale, and the authors do not interpret differences between models as a scale effect; all main analyses are within-model comparisons.
• Demographic coverage of profiles (Section 6): gender was treated as binary, excluding non-binary and transgender users; age, cultural background, education, and other attributes were not examined. The authors plan to expand demographic coverage, use more inclusive categories, and analyze intersectional effects.

Scope of the Experiments

• The models under test are Gemma-4-31B-it, GLM-5.2, and Qwen3.6-27B, 3 in total, all open-source models.
• The data are 97 questions from a single source (ED Support Forum), with 1,649 conditions per model, all single-turn interactions.
• Profiles vary only three attributes—race (White/Asian/Black), gender (female/male), and BMI (low/high)—and each profile discloses two of them.
• Human validation consists of 628 category-level judgments by one clinical annotator on no-bio responses for 30 questions × 3 models.
• The paper does not report the number of repeated samples per condition, nor the decoding parameters of the target models.

An LLM safety evaluation paper in the eating disorder (ED) context, focused on whether models can recognize implicit vulnerability signals in natural questions and consistently protect different users.

• Problem: Diet and weight-loss advice is harmless to ordinary users but may be harmful to ED-vulnerable users, and vulnerability often appears through implicit cues such as community jargon; the authors argue that existing evaluations (diagnostic accuracy, obviously harmful prompts, synthetic scenarios) do not measure this ability.
• Method: 97 real questions were taken from an online ED support forum, and Gemma-4-31B-it, GLM-5.2, and Qwen3.6-27B answered them with no bio and with 16 templated bios (varying race, gender, BMI, and explicit ED cues); Claude scored the responses on a four-level risk rubric extended from the Winecoff–Klyman seven-dimension taxonomy, and matched comparison isolated the effect of each attribute.
• Main results: Without a bio, high-risk responses were about twice as frequent as with a bio (Figure 2b), and the authors state that models often cannot recognize vulnerability from language alone; explicit ED disclosure raised safety scores by 10–14 points, but the Narrow/Stereotyped ED Views dimension instead declined; low-BMI users received safety scores 14.9 (Gemma), 15.5 (GLM), and 24.8 (Qwen) points higher than high-BMI users, and female users and White users also received safer responses than male users and Black or Asian users (Figure 5).
• Judge reliability: The LLM judge and the clinical annotator had an overall κ=0.77, but the judge underestimated risk 18% of the time and overestimated it 7% of the time.
• Authors' conclusion: AI safety in the ED context should not depend on users proactively disclosing or on stereotyped presentations of the illness; models used in health settings need domain-specific evaluations and defenses that respond consistently to clinically meaningful cues across users of different BMI, gender, and race.