Knowing what you know is not enough: large language models' confidence is inconsistent with their actions
Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
Quick read
Based on the paper's full text (AI-generated); refer to the original
Through three types of experiments—prediction market betting, tool calling, and user questioning—the researchers find that large language models' actions often contradict their own confidence estimates, revealing a systematic action-belief gap. The experiments cover 7 models from 5 model families (including GPT-4o, Gemini 2.5 Pro, Llama 3.1 8B, etc.), and elicit confidence using three methods: logit, sampling, and verbal expression. On the Metaculus prediction market, models often bet in the opposite direction of their own high-confidence predictions; no scenario has a direction consistency rate above 79%, and the GPT series is inconsistent with its beliefs most of the time; when confidence is low, models also fail to reliably call the search tool; when faced with user questioning, models change answers they have high confidence in but persist with answers they have low confidence in. Analysis shows that static calibration (ECE) is insufficient to predict behavioral consistency in dynamic scenarios; the well-calibrated Gemini 2.5 Pro is sometimes less consistent than smaller open-source models. The authors argue that in agent deployment scenarios, action-belief consistency should be used as a supplementary metric for evaluating confidence elicitation methods.
- LLMs are increasingly deployed in agentic, multi-turn workflows to perform consequential actions, but confidence extraction methods are usually evaluated only on static question-answering datasets by calibration (such as ECE). The authors point out that this implicitly assumes models act according to their own beliefs, and this assumption has not yet been tested.
- Three minimal interactive experiments are designed: prediction market betting (linear/log utility), tool calling with an oracle search tool provided, and answer persistence/changing under user questioning. On 7 models (5 families, including open-source and closed-source), three confidence extraction methods—logit, sampling, and verbal—are used to compare the consistency between actions and confidence.
- An action-belief gap is observed in all three settings: models often bet in the opposite direction of their own high-confidence predictions, fail to reliably call the search tool when confidence is low, and instead change their answers when confidence is high and persist when confidence is low. Static calibration cannot predict consistency; some closed-source models that emphasize calibration more (such as Gemini 2.5 Pro) are sometimes less consistent than smaller open-source models; consistency is only moderately positively correlated with task performance.
- The authors suggest that the gap may partly stem from the confidence extraction methods themselves failing to point to true internal beliefs, and may also reflect instability in LLM behavior and internal world models; how much each accounts for is left to future work. Future work could seek confidence extraction methods specifically optimized for agentic consistency and test new methods such as GAC on this experimental design.
The paper finds that model actions often contradict their own confidence and that static calibration is insufficient to predict this consistency, providing a dimension beyond calibration for evaluating agent reliability.