HelpBench evaluates the ability of 18 LLMs to provide privacy and security advice

Whether LLMs can give factually accurate and appropriately toned advice on real digital privacy, security, and safety (PSS) help-seeking, and an evaluation for open-ended help-seeking is still missing.

• Help-seeking scenario: The authors state that people already use LLMs to generate passwords, parse privacy policies, fix code vulnerabilities, and do intrusion detection, and also ask about account recovery, two-factor authentication trade-offs, whether a suspicious email is a scam, and whether an abuser can track via peripherals. The authors believe wrong advice could lead to compromised accounts or devices, or financial loss in scams.
• Existing evidence and gap: The authors cite work stating that LLMs often support or do not refute privacy and security misconceptions, generate insecure passwords, and are deficient on sensitive security topics and vulnerability identification. The authors believe guardrail evaluations target violating outputs and jailbreaks, PSS knowledge or disclosure evaluations, account tool or synthetic scam work, and multiple-choice questions are not yet consumer PSS help-seeking benchmarks.
• What this paper proposes: The authors build HelpBench: nine topics, 450 open-ended questions rewritten from Reddit, each with an expert rubric, and use an auto-rater to evaluate 18 models and 40,500 responses.
• Why the authors continue: The authors report an overall average of 82%; the abstract writes one-tenth of answers below 65%, Section 5.2 writes that proportion as 14%. The authors therefore believe the benchmark is not saturated because PSS advice requires consistently high performance.

Safety guardrails and jailbreaks

• Violating outputs: The authors classify RealToxicityPrompts, DecodingTrust, HarmBench, JailbreakBench, SafetyBench, TrustLLM, etc. (paper citations [13, 50, 30, 44, 6, 56, 19, 35]) and jailbreak research ([51, 59]) as preventing models from outputting policy-violating or biased content.
• Distinction from this paper: The authors state that this paper looks at inaccurate advice an honest user might get, such as wrong malware handling steps, rather than attackers eliciting answers that guardrails would otherwise block.

PSS and other high-risk domains

• Knowledge, privacy, and disclosure: The paper describes general safety knowledge evaluations ([17, 9]), privacy awareness ([31, 42]), and frameworks and benchmarks for contextually appropriate disclosure ([14, 23, 54, 48]), without detailing shortcomings per paper.
• Help-seeking benchmarks: The authors state that medical ([45]) and legal LegalBench ([15]) already have high-risk help-seeking benchmarks, while consumer PSS advice does not. Jing et al.'s SecBench ([20]) evaluates account and security tool proficiency; Yang et al.'s Fraud-R1 ([52]) evaluates multi-turn synthetic fraud detection. The authors state that this paper extends to nine topics and uses open-ended questions rather than multiple choice.
• Closest work: The authors state that Prakash et al. ([38]) in 2024 manually examined 900 ChatGPT responses to PSS questions. The contrast: this paper uses rubrics and an auto-rater for ongoing evaluation, covering 18 models and 40,500 responses this time.

Reddit-based benchmark

• Other tasks: The paper mentions using Reddit data to evaluate false premises ([55]), sycophancy ([10]), relationship advice ([18]), and creative writing ([40]). The authors state that HelpBench extends this practice to PSS and rewrites questions to preserve natural intent without exposing identifiable information.

Comparison and data

• Baselines: The paper has no separate external method baselines, but compares 18 model versions across seven families on HelpBench. Questions come from Thomas et al.'s ([49]) labeled Reddit PSS posts. Rubric writing references evaluation rubric literature ([11, 4, 2]); the auto-rater references WildBench ([25]) and HealthBench ([2]).

The authors position this paper as a multi-topic, open-ended PSS help-seeking quality evaluation for honest users, using expert rubrics instead of one-off manual inspection.

HelpBench turns real Reddit help-seeking into single-turn English questions, then uses per-question rubrics to see whether responses state key facts, avoid harmful claims, and whether expression fits the question. Scoring is done by an auto-rater.

Question selection and annotation

• Source: 3 million Reddit posts from 2021–2024 already labeled as PSS-related ([49]), with multi-labels across nine topics. Per topic, 100 posts are stratified randomly sampled, then two researchers do inclusion/exclusion; if either excludes, it is not used. Appendix A also requires balancing subtopics and complexity.
• Scale and topics: Final 450 questions, 50 per topic: Accounts, Moderation tools, Security tools, Privacy tools, Compromise, Platform actions, Scams, Harassment, Data concerns.
• Stratification: Length split into short (fewer than 333 characters), medium (333 to fewer than 666), long (666 to fewer than 1000); posts over 1000 characters are not included. Each topic has at least 10 questions per length bucket, whether the user mentions attempted actions, sensemaking, and guidance. Length is computed automatically; the latter two are annotated by one person and verified by another.
• Boundaries: Only single-turn answerable, text-only, English posts are included. Excluded are posts without a clear question, purely relational or therapeutic help-seeking, outdated technology, requests for inappropriate support, or only asking others to report. The authors list multi-turn, multimodal, and multilingual as future work.

Rewriting

• Approach: To reduce the risk of exposing the original post and being memorized by models, and to make it sound more like talking to an LLM. Gemini 2.5 Flash with default parameters produces a draft, prompt in Appendix B; then manual semantic check, and where feasible, changing details like names, ages, platforms.
• Distance from original: The authors report average cosine similarity between original and rewritten as 0.56, average 3-gram overlap as 0.04.

Expert rubrics

• Scoring: Standard scores are 5, 3, 2, or 1, split into positive and negative. Positive are information that should be included, scored only if met; negative are information that should be avoided, scored only if avoided, not a deduction system. Each is a binary judgment. Score equals hit positive points plus avoided negative points, divided by full marks, normalized to 0 and 1. In Appendix D, one question's full mark is 43, an example answer gets 24, i.e., 55.8%.
• Factual standard: Drafted by six researchers each with over ten years of PSS experience, one person drafts the initial version per question, a second same-topic expert verifies; similar questions are reviewed to see if they reward the same advice. During drafting, product documentation, other online sources, hands-on testing, and drafts generated by Gemini 2.5 Flash per Appendix C can be referenced, and two randomly sampled responses from six models can be seen: GPT 5.0, Claude Sonnet 4, DeepSeek 3.1, Gemini 2.5 Pro, Grok 4, Qwen 3 (written as qwen3-vl-235b-a22b-thinking). Rubrics only cover PSS, not rewarding or penalizing relationship advice or general technical support.
• Expression standard: Each question adds the same set of standards, based on prior evaluations of tone, conciseness, and relevance, and common issues the authors saw, such as too much information. Appendix E's instructions cover whether key information is up front, whether information or action items exceed what the question needs, unprofessionalism, overly imperative tone, exaggerating effectiveness. Full prompts in Appendices B, C, E.

Auto-rater

• Expert anchors: Rubric authors first score one random response from one of the six models, then discuss with a second researcher and resolve all disagreements.
• Automatic scoring: Following WildBench's itemized checklist and HealthBench. The rater outputs explanation and yes/no for each criterion. The authors revise expression standard instructions based on differences from humans until no further relevant improvement is seen. Finally, using Gemini 2.5 Pro, temperature 0, other default parameters, Pearson correlation with expert scores is 0.85 (p < 0.001); factual standard 0.96, expression standard 0.78.

Experimental Setup

• Models tested: 18 across seven families: GPT 4.1, GPT 5.0, GPT 5.3; Claude Sonnet 4, Claude Sonnet 4.6, Claude Opus 4.6; Gemini 2.5 Pro, Gemini 3 Flash, Gemini 3.1 Pro; Grok 4, Grok 4.20; DeepSeek 3.1, DeepSeek 3.2; Qwen 3, Qwen 3.6 Plus; GLM 4.6, GLM 5 Turbo, GLM 5.1. Used chat versions when available, with default parameters.
• Data and repetition: HelpBench 450 questions, 50 per nine topics. Each model answered each question 5 times, averaged by question then aggregated. 2,250 responses per model, totaling 40,500 responses.
• Metrics: Rubric score, i.e., points received divided by full marks. Table 3 gives the average standard deviation of the five responses below each average score.
• Evaluation: All responses were scored by the Gemini 2.5 Pro auto-rater according to the rubric for that question; this model is also among the 18 tested models. During development, one expert rating per question was used to estimate correlation, not a full manual re-evaluation of all 40,500 responses.
• Statistics: Version differences using Wilcoxon signed-rank; topic differences using Kruskal-Wallis; whether existing practices are mentioned using Kolmogorov-Smirnov. The paper provides p-values for these tests.
• Baselines: No external method baselines. Table 1 lists only one variant per family; full results in Table 3 in Appendix F. Figures 3–6 show score distributions; the text does not contain point-by-point verifiable values from the figures.

Main Results

According to Table 3. Five responses averaged by question first. Only 8 models listed, including the lowest in the table, Qwen 3.

Swipe horizontally to read the table

Model	Overall	scams	moderation tools
GPT 5.0	87%	93%	81%
GPT 5.3	87%	90%	84%
GPT 4.1	84%	90%	78%
Gemini 3 Flash	84%	87%	82%
Claude Opus 4.6	84%	90%	81%
Grok 4.20	80%	89%	73%
GLM 5 Turbo	78%	83%	72%
Qwen 3	67%	78%	62%

• Overall scores not listed in the table (Table 3): Claude Sonnet 4.6 83%, Qwen 3.6 Plus 83%, Grok 4 83%, Claude Sonnet 4 82%, Gemini 3.1 Pro 82%, GLM 5.1 82%, Gemini 2.5 Pro 81%, DeepSeek 3.2 80%, GLM 4.6 80%, DeepSeek 3.1 79%. Overall average 82%. The authors state GPT 5.0 is highest; in the table GPT 5.3 has the same overall score.
• The authors state that, except for Qwen 3.6 Plus improving 16% relative to Qwen 3, changes within the same family and same scale range from -3% (Grok 4 vs. Grok 4.20) to 2% (GLM 4.6 vs. GLM 5.1), with some versions declining. The authors report Wilcoxon test p < 0.001 for these version differences. On scale, the authors state Claude Opus 4.6 and Claude Sonnet 4.6 differ by 1%, and Gemini 3 Flash and Gemini 3.1 Pro differ by 2% (p < 0.001).
• Overall topic averages: scams 88%, moderation tools 77% (Table 1). The authors report Kruskal-Wallis p < 0.05 for topic differences in 17/18 models; the difference between best and worst topics ranges from 7% (Gemini 3 Flash) to 15% (Grok 4). Spearman correlation between topic mean scores and interquartile range is -0.73, and with median is -0.61 (both p < 0.001). The authors therefore state that models score higher on topics where responses are more consistent. Responses below 65% range from 156 for GPT 5.0 to 800 for Qwen 3 (out of 2,250 each). The abstract says one-tenth; Section 5.2 says 14% of all responses. The authors therefore believe the benchmark is not saturated.

Factual and Expression Standards

• Table 4 extremes: Highest positive factual is Grok 4 at 89%, lowest is Claude Sonnet 4.6 at 75%. Negative factual lower is better: GPT 5.0 and GPT 5.3 at 7%, Grok 4.20 at 13%. Highest positive expression at 92% (GPT 5.3, Gemini 3 Flash, Qwen 3), lowest at GPT 4.1's 75%. Negative expression lower is better: GPT 5.3 and Claude Sonnet 4.6 at 13%, Qwen 3 at 86%.
• Differences from main text: When discussing the Table 2 subset, the authors state GPT 5.0 best avoids negative facts (7%) and negative expressions (14%), yet has the lowest positive expression satisfaction (82%); and state Grok 4.20 misses the most positive facts (79%). These two sentences are inconsistent with the full-table extremes in Table 4.
• Version changes: The authors report Qwen 3's negative expression loss drops to 34% in Qwen 3.6 Plus (p < 0.001); Grok 4.20 worsens relative to Grok 4 on both factual standards (p < 0.001).

Low-Scoring Responses

• Concentrated failures: The authors report 7% of questions average below 65% across all 18 models. One question about whether original files are securely deleted after copying into an encrypted vault averages 53% across models. The authors state responses affirm the encrypted vault but do not point out that copying does not equal secure deletion of the original.
• Behaviors summarized by authors: In high-risk situations, only giving technical steps without following existing security plans; giving non-scalable or costly solutions; on ban issues, some models refuse to help circumvent terms, others leave it to the user to decide if it violates terms; making unfounded speculations about others' motives; and escalating tone relative to neutral questions. Except for 7% and 53%, the paper does not give counts for each category.

Other Ablations and Analyses

• Length and mean score: The authors state no correlation overall or per model; the longest question is still under 1,000 characters.
• Existing practices: Difference between presence and absence is 0–3%, with only 5/18 models having Kolmogorov-Smirnov test p < 0.05.
• Sensemaking vs. not: difference 0–4%; guidance vs. not: difference 0–3%, with only 1/18 models p < 0.05 for the latter. The authors believe improvements are more likely from topic-level facts and expression rather than these complexity dimensions.
• Skewness: Fisher-Pearson skewness ranges from Qwen 3's -1.07 to GPT 5.3's -1.92; excess kurtosis ranges from Grok 4.20's 0.87 to GPT 5.3's 5.2.
• Practice example: The authors give an example of a question about freezing an account, where the rubric requires stating that creating multiple accounts in a short time is a common security marker, and state most model responses do not mention it; the paper does not give the proportion for that question.

Limitations and future directions noted by the authors

• Training exposure (Section 6.1): Models may have seen these questions or close variants during training, inflating performance. The authors state that rewriting only partially mitigates; expert rubrics also rarely rely on Reddit advice.
• Different phrasing (Section 6.1): Users may phrase questions differently if they know the other party is an LLM, compared to on Reddit.
• Domain changes (Section 6.1): Interfaces and platform features change quickly, and advice correct when writing rubrics may become outdated. The authors state they removed questions that require specific OS or app versions to answer correctly.
• Rubrics and expert preferences (Section 6.1): Rubrics penalize known common harmful advice but cannot cover every misconception. Experts' judgments on tone, structure, and expression may also differ from what non-experts find helpful.
• Interaction and language (Sections 3.2, 6.2, 7): The authors list multimodality, multi-turn, and multilingual as extensions, with examples like screenshot-based scam detection, step-by-step guidance in crises, and questions that are not simple translations but better represent local language and culture. Section 7 also proposes determining whether users need a single-turn complete answer or multi-turn follow-up on key context.
• Beyond rubrics (Sections 6.2, 7): The authors state that legal, therapeutic, and other needs common in PSS help-seeking also need consideration, and connecting models to platform support and advocacy organizations. Expressing the relative importance of standards and other unaccounted negative behaviors require user testing.

Experimental coverage

• Tested are the 18 models across seven families in Section 5 and Table 3; 5 runs per question, totaling 40,500; chat versions used when available, default parameters.
• 450 questions, 50 per topic, English, single-turn, text-only, shorter than 1,000 characters; from 2021–2024 Reddit PSS posts and rewritten (Section 3).
• Scoring by Gemini 2.5 Pro; Pearson correlation with one expert score per question is 0.85, factual standard 0.96, expression standard 0.78 (p < 0.001, Section 4.3).
• The paper reports p-values for version, topic, and some stratified factors, and the average standard deviation across five responses (Table 3). The paper does not report inter-coder agreement coefficients.
• The paper does not report which model did not reach p < 0.05 in Kruskal-Wallis, nor the counts of low-score behaviors in Section 5.2.

HelpBench evaluates whether LLM responses to digital privacy, security, and safety requests are accurate and appropriately expressed.

• Problem: Honest users turn to LLMs for help with accounts, scams, harassment, etc. The authors argue that incorrect advice could lead to account, financial, or physical harm, and that guardrail evaluations and multiple-choice knowledge evaluations do not cover this open-ended, case-specific advice.
• Method: 450 English single-turn questions adapted from Reddit, 50 per nine topics. PSS experts wrote factual standards for what should be included and avoided for each question, plus uniform expression standards. Score is points received divided by full marks. Gemini 2.5 Pro scored according to the rubric, with a Pearson correlation of 0.85 with one expert rating per question.
• Main results: 18 models each answered 5 times, totaling 40,500 responses. Overall average 82% (Table 3). GPT 5.0 and GPT 5.3 at 87%, Qwen 3 at 67%. Overall on scams 88%, on moderation tools 77% (Table 1). Section 5.2 states 14% of responses scored below 65%; 7% of questions averaged below 65% across the 18 models.
• Authors' conclusions: The authors believe models can already provide high-quality advice on most questions, but the long tail contains inaccurate or even harmful advice, so the benchmark is not saturated. They believe improvements are more likely to come from topic facts and expression rather than question length or need type; and suggest future work on multi-turn, multimodal, multilingual, and user testing of expression standards.