Uncertainty-based LLM abstention can improve safety and reduce hallucination
Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations
Translation in progress — untranslated text is omitted.
Research shows that when LLMs choose not to answer in question answering under uncertainty, correctness improves, hallucination is reduced, and safety is enhanced. The research compares statistical uncertainty metrics with a verbal uncertainty measure called In-Dialogue Uncertainty (InDU), and tests on models with and without RLHF. Sacrificing only a small number of high-uncertainty samples, accuracy improves by 2% to 8%, 50% of hallucinations are avoided by identifying unanswerable questions, safety improves by 70% to 99%, and computational overhead is almost not increased.
- LLMs should abstain when they answer incorrectly, hallucinate on unanswerable questions, or are induced to give unsafe responses; the authors explore whether the uncertainty-based abstention mechanism from classification can be borrowed to improve question-answering reliability.
- Compares three types of statistical uncertainty (negative log-likelihood, predictive entropy, semantic entropy) with In-Dialogue Uncertainty (InDU) based on hedge word counts, performing threshold-based abstention on Base and RLHF models, evaluated with AUROC and ARC.
- Statistical uncertainty distinguishes correct from incorrect (TriviaQA RLHF AUROC 0.85), while InDU distinguishes answerable from unanswerable (AUROC 0.75, detecting about 50% of unanswerable and falsely refusing 10%); after the RLHF model refuses the 10% most uncertain samples, the AutoDAN safety rate rises from 92.5% to 99.4%.
- The paper does not specifically discuss limitations. The experiments use only the Llama2 series and Vicuna; safety judgment mainly uses the keyword method; human validation of fuzzy exact match and the safety metric is based on 200 samples and one dataset, AutoDAN, respectively.
Related papers
Both use: AutoDAN
Show 3 more papers
Both use: AutoDAN