Skip to content

Uncertainty-based LLM abstention can improve safety and reduce hallucination

Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations

arXiv· arXiv:2404.10960· Christian Tomani, Kamalika Chaudhuri, Ivan Evtimov, Daniel Cremers, Mark IbrahimMeta; Technical University of MunichIndexed by this site · 23 days ago· Originally publishedDefenses

Translation in progress — untranslated text is omitted.

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

AI summaryResearch shows that when LLMs choose not to answer in question answering under uncertainty, correctness improves, hallucination is reduced, and safety is enhanced. The research compares statistical uncertainty metrics with a verbal uncertainty measure called In-Dialogue Uncertainty (InDU), and tests on models with and without RLHF. Sacrificing only a small number of high-uncertainty samples, accuracy improves by 2% to 8%, 50% of hallucinations are avoided by identifying unanswerable questions, safety improves by 70% to 99%, and computational overhead is almost not increased.

Questions
LLMs should abstain when they answer incorrectly, hallucinate on unanswerable questions, or are induced to give unsafe responses; the authors explore whether the uncertainty-based abstention mechanism from classification can be borrowed to improve question-answering reliability.
Methods
Compares three types of statistical uncertainty (negative log-likelihood, predictive entropy, semantic entropy) with In-Dialogue Uncertainty (InDU) based on hedge word counts, performing threshold-based abstention on Base and RLHF models, evaluated with AUROC and ARC.
Experiments and results
Statistical uncertainty distinguishes correct from incorrect (TriviaQA RLHF AUROC 0.85), while InDU distinguishes answerable from unanswerable (AUROC 0.75, detecting about 50% of unanswerable and falsely refusing 10%); after the RLHF model refuses the 10% most uncertain samples, the AutoDAN safety rate rises from 92.5% to 99.4%.
Limitations and future work
The paper does not specifically discuss limitations. The experiments use only the Llama2 series and Vicuna; safety judgment mainly uses the keyword method; human validation of fuzzy exact match and the safety metric is based on 200 samples and one dataset, AutoDAN, respectively.

Related papers

Show 3 more papers
Read originalarxiv.org