Skip to content

Auditing Chatbots' Encroachment into the Professional Legal Practice Domain

Auditing Chatbots for Incursion into Professional Legal Practice

arXiv: 其他 AI 安全论文· arXiv:2611.17622· Nel Escher, Megan Elliott, Isabelle Qian, Yana Sharifullina, Svenja Guhr, Deja Davis, Deirdre MulliganUniversity of California, Berkeley; Swarthmore CollegeIndexed by this site · 4 days ago· Originally publishedEvaluation· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors built UPLBench to audit 23 chatbots and said 58.4% of responses constituted unauthorized practice of law (UPL), and that newer models gave more legal advice and made fewer referrals to lawyers.

Questions
Whether chatbots provide legal advice in areas reserved for licensed lawyers, whether they increasingly fail to refer users to consult a lawyer, and what harm this causes in high-risk legal domains had previously lacked empirical measurement.
Methods
The authors built UPLBench from 65 cases in which courts found unauthorized practice of law (UPL), generated three rounds of dialogue prompts per case with case-level scoring rubrics, ran 23 models 3 times each under two jurisdiction conditions, automatically annotated with Qwen3.8-27B, and then had 4 professional lawyers review 120 responses.
Experiments and results
The authors report that 58.4% of responses constitute UPL, that the UPL rate of newer models rises with release date (trend test p=0.0038), and that the referral rate declines (p=0.077); lawyer review found failure modes such as not collecting key facts and misleading advice.
Limitations and future work
The authors note: the historical behavior of closed-source models cannot be reconstructed; judges' reasoning about non-lawyers may not apply to AI outputs; the work relies on automatic classification; the benchmark only judges UPL and does not grade; the prompts were written by the research team rather than real users.
Experimental setupLlama-2-13b-chat-hf、Llama-2-7b-chat-hf et al. · UPLBench · UPL rate、referral rate et al.
Evaluated models
Llama-2-13b-chat-hfLlama-2-7b-chat-hfdeepseek-llm-7b-chatgemma-7b-itDeepSeek-V2-Lite-ChatLlama-3.1-8B-Instructgemma-2-2b-itLlama-3.2-1B-InstructLlama-3.3-70B-Instructgemma-3-1b-itDeepSeek-R1Llama-4-Scout-17B-16E-InstructDeepSeek-R1-0528gpt-oss-20bClaude Haiku 4.5DeepSeek-V3.2Gemini 3.1 ProGemini 3.1 Flash-Litegemma-4-26B-A4B-itDeepSeek-V4-ProClaude Fable 5gpt-5.6-solchat-latest
Benchmarks
UPLBench
Metrics
UPL ratereferral ratemacro F1Cohen's κ
AI summaryResearchers constructed the UPLBench benchmark from case law in high-risk legal domains where courts have found unauthorized practice of law (UPL), such as immigration and family law, audited 23 chatbots from 2023 to the present, and found that models increasingly perform tasks courts reserve for lawyers and decreasingly advise users to consult a lawyer. The authors argue that the boundary of legal advice is being rewritten by frontier AI companies rather than by courts or bar associations. The benchmark evaluates whether chatbots refuse or provide substantive legal advice, and whether they direct users to consult a lawyer or legal aid; practicing lawyers involved in the review identified interaction patterns in the conversation records, such as failing to elicit relevant facts, that would be regarded as incompetent legal representation. The authors note that if such advice is followed it could lead to harms that UPL rules were meant to prevent, such as deportation, loss of housing, and family separation, and call on the legal profession to respond to this encroachment, stating that users who rely on chatbot legal advice lack sufficient protection and remedies.

Researchers constructed the UPLBench benchmark from case law in high-risk legal domains where courts have found unauthorized practice of law (UPL), such as immigration and family law, audited 23 chatbots from 2023 to the present, and found that models increasingly perform tasks courts reserve for lawyers and decreasingly advise users to consult a lawyer. The authors argue that the boundary of legal advice is being rewritten by frontier AI companies rather than by courts or bar associations. The benchmark evaluates whether chatbots refuse or provide substantive legal advice, and whether they direct users to consult a lawyer or legal aid; practicing lawyers involved in the review identified interaction patterns in the conversation records, such as failing to elicit relevant facts, that would be regarded as incompetent legal representation. The authors note that if such advice is followed it could lead to harms that UPL rules were meant to prevent, such as deportation, loss of housing, and family separation, and call on the legal profession to respond to this encroachment, stating that users who rely on chatbot legal advice lack sufficient protection and remedies.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Whether chatbots increasingly perform work courts assign to licensed lawyers, reduce referrals to lawyers, and cause harm lacks empirical measurement.

    Core question: whether popular chatbots increasingly provide legal advice that requires a law degree and a lawyer's license, and whether this "unauthorized practice of law" (UPL) behavior is harmful in high-risk legal domains, had no prior empirical evidence.

    • Setting and importance: Many people handle legal problems without a lawyer's help and turn to chatbots instead; the paper cites research stating that after ChatGPT's release, pro se (self-represented) filings in federal civil courts rose by more than 50% over the prior average. The UPL regime aims to protect the public from harm by incompetent representation, and violators may face imprisonment and fines.
    • Gaps in existing research: The authors state that existing legal evaluations focus on "capability" (e.g., using bar exam performance as a proxy for legal ability) and do not empirically measure how prevalent chatbot legal advice is or its quality of service; legal-safety benchmarks (e.g., Do-Not-Answer) do not distinguish legal information from legal advice.
    • Core observation: The authors argue that AI companies' internal decisions (reducing refusals, improving helpfulness, e.g., Anthropic's 2026 charter removing the legal-advice restriction) are redrawing this professional boundary, rather than courts or bar associations.
    • What the paper proposes: It proposes UPLBench—a benchmark derived from 65 cases in which courts found UPL—auditing 23 chatbots from 2023–2026, supplemented by review of 120 responses by 4 professional lawyers, to assess whether chatbots cross the line and the quality of their advice.

    This is an evaluation/audit study and does not involve an attacker threat model; the behavior it studies is whether chatbots themselves cross the boundary courts have drawn between legal information and legal advice.

  2. What related research exists?

    Related work falls into four categories: legal capability benchmarks, safety benchmarks, the UPL policy debate, and audit methods; the authors state that no one has measured the line drawn by courts.

    Legal capability benchmarks

    • Katz et al. (2024) — compared scores on bar exam multiple-choice and essay questions against the human passing line, concluding that GPT-4 "passes the bar exam"; the paper notes that the methods and conclusions of this evaluation have been questioned by Martínez (2025), and the paper holds that the bar exam is only a proxy for legal capability.
    • LegalBench (Guha et al., 2023) — built collaboratively by legal scholars, practitioners, and computational researchers, organizing 162 tasks according to a typology of legal reasoning; the paper notes that it measures task-specific capability and does not claim to establish general legal capability or safety.

    Safety benchmarks

    • Do-Not-Answer (Wang et al., 2024) — a general safety benchmark covering 12 categories of harm, including medical, legal, and financial advice categories; the paper notes that its prompts were generated by GPT-4 rather than derived from legal definitions, and that neither the prompts nor the scoring distinguish legal information from legal advice.
    • Trident-Bench (Hui et al., 2026) — the paper states it is the closest to this work, grounding harmful prompts in the professional norms of law, medicine, and finance (including the ABA Model Rules of Professional Conduct); the paper notes that its setting is a lawyer asking a chatbot to assist in violating professional norms, and that scoring measures the degree to which the chatbot complies with the request.
    • The paper notes that none of these benchmarks ask this paper's core question: when a chatbot responds to an ordinary person seeking legal help, does it cross the line courts have drawn between legal information and legal advice.

    The UPL policy debate and chatbot reliability

    • Ajmi (2024), Avery et al. (2023), Walters (2024), and others argue for relaxing UPL to expand access to justice; Bonardi and Branting (2024) and others argue for certification or review as conditions for exemption; Curl et al. (2026) consider the empirical evidence that chatbots improve legal outcomes to be weak.
    • Reliability evidence: Encarnación et al. (2026) found chatbots unreliable in high-risk legal scenarios such as the legality of abortion; Cohen-Sasson (2026) found that litigants who draft court documents with AI have worse legal outcomes; Cheong et al. (2024) documented lawyers' dissatisfaction with chatbots giving next-step advice when facts are lacking.

    Audit and evaluation methods

    • The paper cites known flaws of benchmark methods: data contamination, the difficulty of approximating real multi-turn use with a single prompt, and the difficulty of discovering unexpected harms (Chang et al. 2024; Wang et al. 2025; Blodgett et al. 2021), and states that it addresses these with a multi-turn design, anonymization, and expert review.

    Positioning: the authors state that this paper differs from capability-oriented evaluations, proposing a method that turns legal doctrine (UPL case law) into a testable benchmark, measuring chatbots' compliance with the law rather than legal capability. The paper has no comparison baseline method; the comparison is among the 23 audited models.

  3. How does the paper address this problem?

    UPL case law was turned into a three-round dialogue benchmark, UPLBench; 26,910 responses were automatically annotated, and a sample was then reviewed by 4 lawyers.

    Overall approach: operationalize the boundary between "legal information (which non-lawyers may provide)" and "legal advice (restricted to lawyers)" using UPL legal doctrine, build the UPLBench benchmark, and conduct a mixed-method algorithmic audit of 23 popular chatbots, combining automatic classification with professional lawyer review.

    Model selection

    • Open-source models to reconstruct history: queried the Hugging Face Hub API for all-time download counts (counted in July 2026), retaining models from the four providers Meta, OpenAI, Google, and DeepSeek that have chat capability and over one million downloads, yielding 17 models (released from October 2023 to June 2026).
    • Closed-source models to depict the present: selected the top three providers in the July 2026 Google Play and App Store rankings (Google, OpenAI, Anthropic), taking each provider's default free-tier model and top paid-tier model, 6 in total, generated via provider APIs.
    • Generation settings: open-source models were downloaded from Hugging Face, and generation used the default settings recommended on the system card on a university computing cluster; DeepSeek-V4-Pro was queried via the OpenRouter API because its full-precision weights exceeded GPU capacity.

    UPLBench construction

    • Case collection: the second author (who completed the first year of law school) collected 65 UPL cases covering five high-risk domains—family law, personal injury, home foreclosure, immigration, and bankruptcy—from 33 state and federal courts; each case was briefed (summarizing facts, legal reasoning, outcome, and harm).
    • Prompt design: the first author (who holds a law degree) and the second author wrote a first-person scenario narrative for each case, adding three consecutive requests specifically seeking advice or work that courts have found to constitute UPL (such as drafting legal documents or recommending legal grounds specific to particular facts); the scenario and the first request were concatenated into the first message, and the latter two requests were sent in separate turns; half of the dialogues include information on the user's jurisdiction.
    • Scoring rubric: the same two authors developed case-level binary rubrics for each turn's request in each case based on judicial reasoning (for example, if a judge found that ghostwriting a complaint constitutes UPL, then a response recommending complaint wording is marked UPL); four models (Llama 4 Maverick, GPT-5.4, Gemini 3 Flash, Claude Haiku 4.5) each generated one transcript for trial labeling, disagreements were discussed before finalization, and a third author with a law degree reviewed it.
    • Anonymization: Presidio was used to identify sensitive information, and Faker generated fake names, emails, and addresses; states were not changed (to avoid changing the applicable law), counties and cities were reselected within the same state; amounts were replaced with random values between half and twice the original; dates were shifted relative to the current year; case numbers were replaced with random digit strings; consistency was manually verified. The motivation is to prevent models from reciting cases from memory and to protect privacy.

    Automatic classification

    • Task: label each response as "UPL" or "not UPL" according to the case- and turn-level rubrics; manually annotating 26,910 responses is infeasible (estimated at nearly 31 weeks of full-time work).
    • Comparison of two approaches: fine-tuning XLM-RoBERTa large (about 560 million parameters, trained on 158 human annotations) versus prompting Qwen3.8-27B (input: dialogue, case and turn rubrics, and general UPL guidelines). The test set was 102 responses from 34 dialogues, independently annotated by the first two authors, with a human baseline Cohen's κ = 0.78 (95% bootstrap CI 0.65–0.90).
    • Results and choice: the prompting approach achieved macro F1 0.84 (κ = 0.62 and 0.59 against the two annotators), outperforming the fine-tuning approach's macro F1 0.77, so Qwen3.8-27B was used to annotate all responses; it was chosen because its open weights make it reproducible, it performs well on legal benchmarks, and its developer is not in the audited set, avoiding self-preference. Sampling parameters: temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5, fixed seed, thinking mode off.

    Expert lawyer review

    • Recruitment: 4 professional lawyers (1 immigration, 1 foreclosure, 2 personal injury) were recruited through the authors' networks and law school directories, each paid a $100 honorarium.
    • Procedure: 2 court cases were selected per domain, and 5 dialogues generated by different providers were sampled per case, with each lawyer reviewing 10 dialogues (120 responses in total); lawyers first read the original case, then, against the rubrics and the authors' annotations, answered three open questions (whether the rubric was applied correctly, whether the legal content was accurate, and whether the quality of service met professional standards).
    • Theme extraction: the first and last authors independently read, discussed, and finalized themes using reflexive iterative thematic analysis.
  4. What experiments did the paper conduct?

    23 models × 65 cases × 3 rounds × 3 runs × 2 jurisdiction conditions, 26,910 responses in total; 58.4% were judged UPL, and newer models more often.

    Experimental setup

    • Models tested: 23, including 6 closed-source (Claude Haiku 4.5, Claude Fable 5, Gemini 3.1 Pro, Gemini 3.1 Flash-Lite, gpt-5.6-sol, chat-latest) and 17 open-source (Meta, OpenAI, Google, DeepSeek, released from 2023-10 to 2026-06; full list in Table 2).
    • Benchmark: UPLBench, 65 tasks, each task a three-turn dialogue; half of the dialogues include jurisdiction information.
    • Scale: 23 models × 65 cases × 3 turns × 3 repetitions × 2 jurisdiction conditions = 26,910 annotated responses.
    • Metrics: UPL rate (proportion of responses judged to constitute unauthorized practice of law); referral rate (proportion recommending consulting a lawyer or legal aid, in three levels: strong/weak/none).
    • Judging method: Qwen3.8-27B automatically labels according to case- and turn-level rubrics (macro F1 0.84 against human annotation); additionally, 4 professional lawyers reviewed 120 responses.
    • Statistical tests: Jonckheere-Terpstra trend test (195 items = 65 cases × 3 turns, ordered by release date, model-level permutation); Wilcoxon signed-rank test (paired, n = 23 models).

    Main results

    The authors report: 58.4% of all responses were judged UPL; referral to a lawyer or legal aid accounted for 81.0%, dropping to 61.5% when counting only strong referrals. Ranges of per-model UPL rate and referral rate (according to the text of Sections 5.1 and 5.2):

    MetricLowestHighest
    UPL rate20.3% (Meta Llama 3.3 70B)88.5% (DeepSeek R1)
    Referral rate38.8% (OpenAI chat-latest)96.0% (Google Gemma 2-2B)
    • Trend: the authors report that UPL rate rises with release date (Jonckheere test z = +2.58, permutation p = 0.0038, one-tailed); Figure 1 labels a fitted trend of +11.0 percentage points/year. The referral rate decline trend is z = −1.43, p = 0.077 (one-tailed), with Figure 1 labeling −4.2 percentage points/year.
    • Provider differences: the authors state that OpenAI chatbots have the highest UPL rate (mean 76.1%, Figure 3); by domain, immigration is the lowest but still has 52% of responses containing legal advice (Figure 2).
    • Compliance benchmark: the authors note that even a single instance of a non-lawyer providing legal advice is prohibited, so full compliance should be 0%.

    Effects of jurisdiction and vendor policy

    • Jurisdiction information: when the user's jurisdiction is included, 21 of 23 models have a higher UPL rate (median +4.4 percentage points, Wilcoxon z = 3.88, p < 0.001, r = 0.83); the authors suggest that models are more willing to give legal advice when users disclose personal information such as location.
    • Vendor policy: Anthropic removed the legal advice harm category from its charter in January 2026; the authors report that Claude Fable 5 (June 2026) has a higher UPL rate than Claude Haiku 4.5 (October 2025) in 60 of 65 cases, on average 35.6 percentage points higher (z = 6.72, p < 0.0001, r = 0.85). The authors note that the two models may also differ in scale or architecture, but the increase coincided with the charter revision (Figure 5).

    Referral decay across multi-turn dialogues

    • Measurement: per-model referral rate was computed for each turn, with a Wilcoxon paired test (n = 23).
    • Results: from the first to the second turn, 22 of 23 models show a decline in referral rate (median −12.6 percentage points, z = −3.98, p < 0.0001, r = −0.83); there is no statistical difference between the second and third turns (p = 0.74). The authors state this highlights the importance of multi-turn testing.

    Failure modes found in lawyer review (Section 5.3)

    • Failure to collect necessary facts: lawyers in all domains said the chatbots did not ask for facts supporting legal analysis; for example, a personal injury lawyer noted that the chatbot did not ask about the neurological injury of a car-accident user (deepseek-llm-7b-chat, Claude Haiku 4.5).
    • Accepting the user's framing and giving instructions rather than counseling: in a foreclosure scenario, directly recommending strategies to stop the auction without understanding the user's financial situation (Google Gemini Pro, OpenAI GPT 5.6 Sol, Claude Fable 5, Llama 3.3 70B); a lawyer said Claude Fable 5's giving of specific steps was "concerning in its certainty."
    • Creating risk and advice quality: an immigration lawyer noted that chat-latest requested sensitive information without confidentiality protection (this appeared in only one review record; using regular expressions, the authors separately found that GPT-5.6 Sol, DeepSeek V4-Pro, Claude Fable 5, Claude Haiku 4.5, Gemini 3.1 Flash-Lite, and Gemini 3.1 Pro directly asked whether the user or a relative had ever been deported); the lawyer also said the advice was incomplete, inaccurate, or misleading, such as recommending a list of forms insufficient to complete an adjustment-of-status application (Gemini 3.1-Flash-Lite, deepseek-llm-7b-chat), recommending nonexistent documents (deepseek-llm-7b-chat), and suggesting behavior that could constitute extortion (Llama-4-Scout-17B).

    Other ablations and analyses

    • Automatic classification validation: on a test set of 102 responses, inter-human-annotator κ = 0.78; Qwen3.8-27B macro F1 0.84, XLM-RoBERTa large 0.77, with no statistically significant difference in accuracy (McNemar exact test p = 0.23) (Appendix B).
    • Pilot set: 150 responses, 10 cases, 5 domains, Qwen3.8-27B accuracy 0.85, κ = 0.67; modified guidelines or few-shot variants did not improve agreement (Appendix B).
    • Failure factor of the fine-tuned model: 54% of responses exceeded the 512-token input limit and were truncated (Appendix B).
  5. What are some points that can be further explored?

    The authors list five limitations; the experiments cover 23 models, 5 legal domains, and 3 dialogue turns, and do not report real user wording.

    Limitations and future directions noted by the authors

    • Closed-source models' historical behavior cannot be reconstructed (Appendix A): proprietary chatbot behavior depends on weights and the company's internal infrastructure (safety classifiers, guardrail enforcement); a dated snapshot cannot freeze the entire safety ecosystem; the authors propose longitudinal studies that record behavior when new models are released to avoid this problem.
    • Applicability of judicial reasoning (Appendix A): annotation is based on judges' reasoning as applied to non-lawyers, which does not guarantee consistency with how courts would actually rule on AI output, and may make the findings difficult to use for enforcement.
    • Reliance on automatic classification (Appendix A): classification of all responses by human legal experts would strengthen the measurement.
    • The benchmark only detects UPL (Appendix A): it does not grade non-UPL responses and cannot be used to calibrate models to sit tightly at the boundary between legal information and advice.
    • Prompts are not real users' wording (Appendix A): benchmark items are grounded in real people's situations, but the prompts were written by the research team, and real users' wording may differ; the authors propose future work testing such variation or collecting user chat logs.
    • Future directions (Discussion): explore the contexts in which chatbots improve users' legal outcomes; assess whether their positive elements (encouraging help-seeking, providing resource lists) prompt users to hire lawyers; study whether AI advice is useful under appropriate conditions in specific legal domains (e.g., specialized tools for criminal record expungement).

    Experimental coverage

    • 23 models were tested: 6 closed-source (2 each from Anthropic, Google, OpenAI) and 17 open-source (Meta, OpenAI, Google, DeepSeek), with release dates from 2023-10 to 2026-06 (Table 2).
    • The benchmark covers 5 legal domains (family law, personal injury, foreclosure, immigration, bankruptcy), 65 cases, and 33 state and federal courts; each case has 3 dialogue rounds, 3 repetitions, and 2 jurisdiction conditions.
    • Lawyer review covers 3 domains (immigration, foreclosure, personal injury); 4 lawyers each reviewed 10 dialogues, 120 responses in total.
    • Closed-source models were evaluated via API; the authors note that API outputs cannot estimate how often real users encounter unsafe advice, because personalization makes model behavior vary by user (Section 4.1).
    • The paper does not report the specific sampling parameters used for open-source model generation (it only states that system-card recommended default settings were used), nor does it report consistency across the 3 repetitions.
  6. Summarize the main content of the paper

    The UPLBench audit showed that most responses constituted unauthorized practice of law, newer models were more willing to give advice and made fewer referrals, and lawyer review found multiple failure modes.

    An audit study on chatbot legal compliance: the authors turn US "unauthorized practice of law" (UPL) legal doctrine into a testable benchmark, measuring whether chatbots cross the line courts have drawn between legal information and legal advice.

    • Question: AI companies have adjusted models to reduce refusals and improve helpfulness; whether chatbots increasingly perform work reserved for licensed lawyers, whether they reduce referrals of users to consult lawyers, and whether they are harmful in high-risk domains had not previously been empirically measured.
    • Method: UPLBench was built from 65 cases in which courts found UPL (5 high-risk domains, 33 state and federal courts), with three rounds of dialogue generated per case and case-level scoring rubrics; 23 models released 2023–2026 were audited (6 closed-source, 17 open-source), with 26,910 responses automatically annotated by Qwen3.8-27B (macro F1 0.84 against human annotation), and 4 professional lawyers reviewed 120 responses.
    • Main results: The authors report that 58.4% of responses constitute UPL (per-model 20.3%–88.5%); the UPL rate rises with release date (p = 0.0038); the referral rate is 81.0% (strong referral only 61.5%), declines with release date (p = 0.077), and drops by a median of 12.6 percentage points from round one to round two.
    • Policy relevance: Claude Fable 5 averages 35.6 percentage points higher than Claude Haiku 4.5, coinciding with Anthropic's 2026 charter removing the legal-advice restriction; when jurisdiction information is included, 21/23 models have a higher UPL rate.
    • Lawyer review: Failure modes found include not collecting key facts, accepting the user's framing, giving instructions rather than consultation, requesting sensitive information without confidentiality protection, and incomplete or misleading advice; the authors state that if adopted this could lead to consequences such as deportation, loss of housing, and family separation.
    • Authors' conclusion: Chatbots are engaging in the unauthorized practice of law and pose risks to individuals; the authors call on the legal profession to respond using existing legal authority such as UPL and consumer protection, and argue that the benchmark evidence can provide a baseline for oversight by bar associations and state judicial bodies.
Read originalarxiv.org