LLM confidence calibration should not stop at single-turn dialogue

The researchers propose the multi-turn calibration task, reframing confidence calibration of large language models from a static property into a dynamic problem that incorporates conversation history at each turn, and point out that user feedback (such as persuasion) may weaken multi-turn calibration. This work targets multi-turn interaction of LLMs in high-stakes domains such as finance, healthcare, and education, proposes the metric ECE@T for tracking per-turn calibration changes, and the method MTCal that minimizes ECE@T via an agent calibration objective, and uses the calibrated confidence in the decoding strategy ConfChat to improve the factuality and consistency of responses. Experiments in the paper show that MTCal performs stably on multi-turn calibration, and ConfChat maintains or even improves model performance in multi-turn interaction; the authors state that multi-turn calibration is the missing link for extending LLM calibration to safe and reliable real-world applications.