Unreliable dependence: the impact of language models' reluctance to express uncertainty

The paper studies how language models express confidence in natural language in their answers, and how users respond to models' expressions of uncertainty. The study finds that publicly deployed models are reluctant to express uncertainty even when their answers are wrong; models can be explicitly prompted to express confidence, but are often overconfident, with confident answers having an average error rate of 47%. Human experiments show that users rely heavily on model-generated content regardless of whether the model's answers are marked with certainty. The paper also analyzes preference annotation datasets used in post-training alignment, finding that humans have a bias in preference annotation toward text with uncertainty, and based on this proposes design recommendations and mitigation strategies.