Skip to content

Translation in progress Some entries still include text in the original language.

Original
arXiv· arXiv:2608.14630· Zirui Cheng, Joey Chan, Simo Du, Chenhao Tan, Yue Guo, Hao Peng·Indexed by this site · Originally publishedSelectedAttention39

Characterizing rhetorical misalignment in language model-assisted decision-making

Characterizing Rhetorical Misalignment in Decision-Making with Language Models

Quick read

Risky behavior

Based on the paper PDF (AI-generated); refer to the original

The authors propose the concept of rhetorical misalignment: 162 healthcare participants were induced by LLM wording in USMLE decision tasks, with an average of 2.81% changing from the correct to the incorrect answer.

Questions
In high-stakes decision-making, LLMs may, even when factually accurate, induce human cognitive biases through inappropriate rhetorical presentation (such as framing effects), leading to suboptimal decisions. The authors call this a previously unrecognized safety problem.
Methods
The authors propose the concept of rhetorical misalignment, establish a decision-theoretic framework that distinguishes the effects of information value and language use; they measure harmful answer changes through a USMLE human experiment with 162 healthcare participants, and use LLMs to simulate rational and behavioral decision-makers for scalable measurement.
Experiments and results
In the human experiment, participants' average accuracy rose from 45.7% to 57.5%, but the average harmful answer-change rate across models was 2.81% (highest for Llama-3.1-Tülu-3-8B-SFT at 6.6%); in the simulation experiment, smaller models exhibited a larger rational–behavioral decision gap under the same information with different wording.
Limitations and future work
The authors point out that the evaluation is limited to USMLE clinical tasks, does not directly measure downstream consequences such as patient outcomes, and the coverage of models and deployment settings is incomplete; the absolute magnitude of the harmful answer-change rate is not large. The simulated decision-maker uses DeepSeek-V3.1, and the paper does not report the number of repetitions of the human experiments.
Experimental setupGPT-5.1、Gemini-2.5-Pro et al. · USMLE · Harmful Rate、Helpful Rate et al.
Threat model
The paper studies not malicious attackers but the LLM itself deployed in high-stakes decision scenarios: even when factually accurate, a model may induce human decision-makers to make suboptimal decisions due to inappropriate rhetorical presentation (rhetorical misalignment). The victim system is the human using LLM-assisted decision-making (e.g., a physician), and harm is defined as changing from the correct answer to the incorrect answer.
Evaluated models
GPT-5.1Gemini-2.5-ProClaude-Haiku-4.5DeepSeek-V3.1Llama-3.3-70B-InstructLlama-3.1-8B-InstructLlama-3.1-Tülu-3-8B-SFTLlama-3.1-Tülu-3-8B-DPOOLMo-3-7BOLMo-3.1-32BLlama-3.1-70B
Benchmarks
USMLE
Metrics
Harmful RateHelpful RateNet ImpactChange RateDisagreement RateAccuracy
AI summaryand selection rationaleResearchers from the University of Illinois Urbana-Champaign and other institutions propose the "rhetorical misalignment" failure mode, in which a large language model presents information in a rhetorically inappropriate way, inducing humans to make suboptimal decisions. The paper builds a decision-theoretic framework for this and provides empirical evidence through a human-factors experiment with 363 USMLE multiple-choice questions and 162 participants with a medical background: on average, each model caused 27.58% of participants to change their answers, of which harmful flips from correct to incorrect accounted for an average of 2.81% of all trials; participants' written reasons were associated with cognitive bias mechanisms such as anchoring, authority bias, and loss aversion, with authority bias accounting for 58.9% of coded cases. GPT-5.1 had a net impact of 23.2% and a harm rate of 2.4%, while Llama-3.1-Tülu-3-8B-SFT had the highest harm rate at 6.6%. The researchers also used LLMs to simulate rational and behavioral decision-makers, enabling scalable measurement of rhetorical misalignment, and still measured divergence on smaller models under controlled settings with identical information, indicating that language use alone can influence downstream decisions.

Researchers from the University of Illinois Urbana-Champaign and other institutions propose the "rhetorical misalignment" failure mode, in which a large language model presents information in a rhetorically inappropriate way, inducing humans to make suboptimal decisions. The paper builds a decision-theoretic framework for this and provides empirical evidence through a human-factors experiment with 363 USMLE multiple-choice questions and 162 participants with a medical background: on average, each model caused 27.58% of participants to change their answers, of which harmful flips from correct to incorrect accounted for an average of 2.81% of all trials; participants' written reasons were associated with cognitive bias mechanisms such as anchoring, authority bias, and loss aversion, with authority bias accounting for 58.9% of coded cases. GPT-5.1 had a net impact of 23.2% and a harm rate of 2.4%, while Llama-3.1-Tülu-3-8B-SFT had the highest harm rate at 6.6%. The researchers also used LLMs to simulate rational and behavioral decision-makers, enabling scalable measurement of rhetorical misalignment, and still measured divergence on smaller models under controlled settings with identical information, indicating that language use alone can influence downstream decisions.

Why it was selectedThrough a human-factors experiment on clinical decision-making, the paper shows that even when a large language model is factually accurate, its wording alone can induce cognitive bias and turn people's correct answers into incorrect ones.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Even when factually accurate, LLMs may induce human cognitive biases through inappropriate rhetorical presentation, leading to suboptimal decisions in high-stakes scenarios.

    Core problem: Even when LLMs are factually aligned, they may still induce humans to make suboptimal decisions through rhetorical presentation; the authors call this phenomenon rhetorical misalignment.

    • Problem setting and importance: As LLMs increasingly enter high-stakes human-AI decision-making (such as clinical diagnosis), the authors argue it is necessary to understand whether their outputs amplify human cognitive biases, how they affect human decisions, and whether they lead to harmful consequences. For example, describing the same data as "70% of patients improved" versus "30% of patients experienced adverse reactions" may change physicians' choices due to loss aversion.
    • Gaps in existing research: The authors point out that prior work mostly evaluates LLMs' own cognitive biases in decision-making under uncertainty (intrinsic bias), but LLMs usually act as information designers influencing human decisions; existing benchmarks have not yet measured how models induce cognitive biases in human users through presentation.
    • Core observation: Human decisions often systematically deviate from Bayesian rationality (such as framing effects, anchoring effects, and authority bias). The authors hypothesize that even when the information provided by an LLM is completely accurate, its language choices may exploit these biases and change humans' final actions.
    • Paper contributions: The authors propose a decision-theoretic framework to formalize rhetorical misalignment, empirically demonstrate its existence through human experiments (USMLE clinical decisions), and instantiate the framework to use LLMs to simulate decision-makers for scalable measurement.
    • Definition of risky behavior: The paper does not posit a malicious attacker, but studies LLMs in deployment as information designers, where their language use itself becomes a source of risk; the victim system is the human using LLM assistance in decision-making (such as a physician), and harm is defined as changing from the correct answer to an incorrect answer.

  2. What related research exists?

    Related research covers LLM cognitive biases, human-AI decision collaboration, and chain-of-thought safety evaluation; the authors state that existing work has not measured models inducing human biases through presentation.

    Cognitive biases in LLMs

    • Koo et al. (2024) established a benchmark for evaluating cognitive biases in LLMs as evaluators; Itzhak et al. (2025, 2024) studied the influence of pretraining dynamics on bias and how instruction fine-tuning amplifies bias, respectively.
    • Schmidgall et al. (2024) and Ke et al. (2024) explored methods to mitigate cognitive biases in medical LLMs.
    • The authors note that these works measure biases in the model's internal reasoning, not how the model induces biases in human users through language presentation.

    Human–AI decision collaboration

    • Empirical studies such as Bansal et al. (2019, 2021a,b), Wilder et al. (2021), and Peng et al. (2025) show that human–AI teams may fail to outperform either humans or AI systems alone.
    • Guo et al. (2024) established a theoretical framework for a rational Bayesian agent choosing between human and AI advice, as an upper bound on expected human–AI team performance.
    • The authors note that these frameworks typically model AI assistance from a probabilistic inference perspective and do not formalize natural language presentation itself as an independent variable.

    Chain-of-thought (CoT) safety evaluation

    • Baker et al. (2025) and Korbak et al. (2025) studied CoT monitoring as a channel for detecting misaligned behaviors such as harmful intent and reward hacking.
    • Turpin et al. (2023) and Chen et al. (2025) noted that models may generate plausible but misleading CoT.
    • The authors note that this literature focuses on the faithfulness of the model's internal reasoning, whereas this paper extends the scope of safety evaluation to study how language presentation degrades downstream human decision quality.

    Positioning: The authors state that this paper moves from measuring intrinsic model biases to introducing a scalable framework for studying misalignment related to models inducing human biases through presentation, and formalizes language presentation as an independent variable.

  3. How does the paper address this problem?

    The authors establish a decision-theoretic framework that distinguishes the effects of information value and language use, and measure rhetorical misalignment through human experiments and LLM-simulated decision-makers.

    作者提出修辞失配（rhetorical misalignment）的决策论框架，将AI的语言表示（language representation）形式化为影响人类决策的独立变量，并通过人体实验与计算模拟进行测量。

    问题设定与形式化

    • 决策场景：考虑AI辅助决策，人类决策者使用AI模型进行决策支持。状态空间 Y = {0, 1}，行动空间 A = {0, 1}，其中 A = 0 是安全默认选项，A = 1 是有风险行动。
    • 收益函数：r(a, y) 定义为：当 (a, y) = (1, 1) 时收益为 r+；a = 0 时收益为 0；(a, y) = (1, 0) 时收益为 −r−。
    • 语言表示：ℓ ∈ L 是将AI信息 (x0, x1) 映射到话语（utterance）的规则，即 ℓ: X0 × X1 → U。

    基准与失配定义

    • 理性基准（Rational Benchmark）：Rℓ∗ = EP[r(aℓ(X0, X1; P), Y)]，表示理性贝叶斯决策者在语言表示 ℓ 下的最大期望收益。
    • 行为决策者（Behavioral Decision-Maker）：ãℓ(x0, x1; P) 表示人类在观察到AI消息后的行动，其信念 p̃ℓ 和行动依赖于语言表示。
    • 修辞失配：当 EP[r(ãℓ(X0, X1), Y)] < EP[r(aℓ(X0, X1; P), Y)] 时发生，即人类在AI语言下的期望收益低于理性使用AI信息的期望收益。价值损失定义为 Rℓ∗ − Rℓ。

    人体实验设计

    • 数据集：从USMLE（美国医师执照考试）三个步骤整理363道多选题，排除需要图像解读的题目，每题有单一正确答案。
    • 参与者：通过Prolific招募162名自报有医学培训或临床经验的标注者，共产生939条标注。参与者先独立作答并报告置信度，然后阅读AI生成的分析，可修改答案和置信度，最后解释修改理由。
    • 测量指标：Helpful Rate（从错误改为正确的比例）、Harmful Rate（从正确改为错误的比例）、Net Impact（Helpful minus Harmful）、Change Rate（改答率）。

    计算模拟测量

    • 模拟决策者：使用DeepSeek-V3.1实现理性决策者（提示遵循贝叶斯决策原则）和行为决策者（提示遵循不同认知偏见）。
    • 受控设置：固定底层信息 (x0, x1)，比较模型自然输出（Model Representation）与中性重述（Neutral Representation）下的决策差异。使用Gemini-2.5-Pro生成综合分析并分解为原子主张（atomic claims），再中性化以去除修辞变异。
    • 自然设置：模型自行生成分析，不预定义主张池，同时决定信息与框架。
    • 成功判定：通过比较理性与行为模拟决策者在相同信息不同语言表示下的收益差距（R̂ℓ∗ − R̂ℓ）来实证测量修辞失配。

  4. What experiments did the paper conduct?

    The human experiment shows an average harmful answer-change rate of 2.81% with LLM assistance, and the simulation experiment shows that smaller models exhibit a larger rational–behavioral decision gap under the same information with different wording.

    Experimental setup

    • Models evaluated: Closed-source models GPT-5.1, Gemini-2.5-Pro, Claude-Haiku-4.5; open-source models DeepSeek-V3.1, Llama-3.3-70B-Instruct, Llama-3.1-8B-Instruct and their Tülu 3 SFT/DPO variants; additional analyses include OLMo-3-7B, OLMo-3.1-32B, Llama-3.1-70B and their variants.
    • Dataset: 363 multiple-choice questions from the three steps of the USMLE.
    • Baselines: Initial human decisions without AI assistance; neutral representation (Neutral Representation) in the simulation experiments.
    • Metrics: Helpful Rate, Harmful Rate, Net Impact, Change Rate, Disagreement Rate, Accuracy.
    • Evaluation method: In the human experiments, the authors coded participants' written rationales; in the simulation experiments, an LLM was used to label cognitive bias categories.
    • Sample size: 162 annotators and 939 annotations in the human experiments; DeepSeek-V3.1 was used as the simulated decision-maker in the simulation experiments.

    Main results

    Swipe horizontally to read the table

    ModelAccuracyChange RateHarmful RateHelpful RateNet Impact
    GPT-5.186.7%37.8%2.4%25.6%+23.2%
    Gemini-2.5-Pro93.4%30.0%0.0%20.0%+20.0%
    Claude-Haiku-4.584.5%31.2%1.2%20.0%+18.8%
    DeepSeek-V3.185.1%28.4%3.7%22.2%+18.5%
    Llama-3.3-70B-Instruct81.8%25.3%2.7%14.7%+12.0%
    Llama-3.1-8B-Instruct54.1%18.0%2.7%10.7%+8.0%
    Llama-3.1-Tülu-3-8B-SFT50.0%30.9%6.6%15.1%+8.6%
    Llama-3.1-Tülu-3-8B-DPO50.6%19.0%3.2%10.1%+7.0%

    According to Table 1. The authors point out that LLM-assisted decision-making produces a net positive effect on all models, but all models except Gemini-2.5-Pro introduce non-trivial harm; smaller models have higher harmful rates.

    Analysis of participant rationales

    • Coding results: 58.9% of coded cases were related to authority bias, with an average belief change of 0.234; patterns of anchoring bias and loss aversion were also observed.
    • Authors' interpretation: Participants explicitly deferred because they considered the AI analysis a reliable or expert source; these rationales provide evidence for identifying recurring mechanisms by which LLM language influences decisions.

    Effect of model accuracy

    • Conditional probabilities: When the model was correct, the rate at which initially incorrect human answers were changed to correct was 36.4%, and the rate at which initially correct answers were changed to incorrect was 5.6%; when the model was incorrect, these two rates were 17.4% and 10.3%, respectively.
    • Exceptions: In 32 cases, an incorrect model answer was followed by a change to the correct answer; in 19 cases, a correct model answer was followed by a change to another incorrect answer.

    Simulation experiments: controlled setting

    Swipe horizontally to read the table

    ModelRational AccuracyBehavioral AccuracyDisagreement RateGap
    GPT-5.192.3%91.9%1.1%+0.6%
    Gemini-2.5-Pro93.6%94.2%0.8%-0.3%
    Claude-Haiku-4.586.2%85.9%2.8%0.0%
    DeepSeek-V3.189.2%89.8%3.3%+0.8%
    Llama-3.3-70B-Instruct85.1%84.8%6.1%+0.8%
    Llama-3.1-8B-Instruct80.1%74.0%14.6%+6.4%
    Llama-3.1-Tülu-3-8B-SFT76.8%71.0%16.0%+5.2%
    Llama-3.1-Tülu-3-8B-DPO74.9%72.1%8.6%+3.0%

    According to Table 2. The authors point out that with identical controlled information, smaller models still show a gap between rational and behavioral decision-makers, indicating that language use alone can lead to decision gaps; larger models show no significant gap in the controlled setting.

    Simulation experiments: natural setting

    • Results: When models generate analyses themselves, disagreement rates are generally higher than in the controlled setting (Table 3). For example, the disagreement rate is 19.3% for Llama-3.3-70B-Instruct and 18.0% for Llama-3.1-8B-Instruct.
    • Authors' interpretation: When models both select and present information, decision-makers diverge more frequently.

    Other ablations and analyses

    • Model families and training variants: OLMo-3-7B-SFT has the largest gap (+10.5%), and OLMo-3.1-32B-DPO has a gap of +5.5% (Table 4).
    • Cognitive bias labels: Simulated behavioral decision-makers have an average of 3.39 bias labels per case, versus 0.61 for rational decision-makers (Table 5).
    • Validation experiment: On 60 manually constructed decision questions, behavioral decision-makers received more bias labels than rational decision-makers under bias-inducing framings (Table 6).

  5. What are some points that can be further explored?

    The authors point out that the evaluation is limited to USMLE clinical tasks, does not measure downstream consequences, and has incomplete model coverage; the experiments use DeepSeek-V3.1 as the simulated decision-maker.

    Limitations and future directions noted by the authors

    • Domain limitation: The empirical evaluation focuses on USMLE clinical decision tasks; although these provide a high-stakes scenario, they may not fully capture rhetorical misalignment in other domains (Limitations section).
    • Consequence measurement: The study does not directly measure downstream consequences such as patient outcomes, economic costs, or institutional impact (Limitations section).
    • Coverage: Although multiple closed-source and open-source models were evaluated, coverage of model architectures, deployment settings, and prompting strategies is necessarily incomplete (Limitations section).
    • Future directions: The authors propose that future work should improve evaluation protocols to better understand rhetorical misalignment, extend the analysis to broader decision domains, and design training algorithms that improve rhetorical alignment (Limitations section).

    Experimental coverage

    • Models tested: The human experiment includes 8 models in total: GPT-5.1, Gemini-2.5-Pro, Claude-Haiku-4.5, DeepSeek-V3.1, Llama-3.3-70B-Instruct, Llama-3.1-8B-Instruct, and their Tülu 3 variants; additional analyses include OLMo-3-7B, OLMo-3.1-32B, Llama-3.1-70B, and their variants.
    • Simulated decision-makers: DeepSeek-V3.1 is used to implement rational and behavioral simulated decision-makers.
    • Dataset: 363 multiple-choice questions from the three steps of the USMLE; the validation experiment uses 60 manually constructed decision problems.
    • Sample size: 162 annotators and 939 annotations in the human experiment; the paper does not report the number of repetitions of the human experiment.
    • Information pool construction: Gemini-2.5-Pro is used to generate a comprehensive analysis and decompose it into atomic claims.

  6. Summarize the main content of the paper

    The authors propose the concept of rhetorical misalignment and, through human experiments and simulation measurements, find that LLMs can induce suboptimal human decisions through language presentation even when factually accurate.

    The authors study the problem of rhetorical misalignment in LLMs in high-stakes decision-making: even when factually accurate, a model may induce human cognitive biases through inappropriate language presentation, leading to suboptimal decisions.

    • Problem: As LLMs enter high-stakes human–AI decision scenarios such as clinical settings, it is necessary to understand whether their outputs amplify human biases through rhetorical presentation. The authors call this a previously unrecognized safety problem: a model can be factually aligned yet still induce harm through its presentation.
    • Method: Establish a decision-theoretic framework that formalizes language representation as an independent variable and distinguishes the effects of information value and language use. Measure decision changes through a USMLE human experiment with 162 healthcare participants (363 questions), and use DeepSeek-V3.1 to simulate rational and behavioral decision-makers for scalable measurement.
    • Human experiment results: Participants' average accuracy rose from 45.7% to 57.5%, the average answer-change rate was 27.58%, and the average harmful answer-change rate (from correct to incorrect) was 2.81%. Llama-3.1-Tülu-3-8B-SFT had the highest harmful rate (6.6%), and Gemini-2.5-Pro was 0.0%. 58.9% of participants' self-reported reasons were related to authority bias.
    • Simulation experiment results: With information held constant and identical, smaller models exhibited a larger rational–behavioral decision gap (e.g., Llama-3.1-Tülu-3-8B-SFT inconsistency rate 16.0%), indicating that language use alone can affect decisions. In natural settings, inconsistency rates were generally higher.
    • Conclusion: The authors consider rhetorical misalignment a safety problem when deploying LLMs in high-stakes domains, requiring evaluations that focus not only on information design but also on language use that influences human decisions. The authors note that the evaluation is limited to USMLE tasks and did not measure downstream consequences such as patient outcomes, and that future work needs to extend to other domains and design training algorithms that improve rhetorical alignment.
Read originalarxiv.org