RL-ARC: Reasoning-Guided Uncertainty Calibration for Large Reasoning Models

RLVR training of reasoning models does not include confidence in the reward, making them prone to overconfidence; existing calibration training remains overconfident under distribution shift and often sacrifices reasoning accuracy.

• Scenario: The capability of large reasoning models mainly comes from reinforcement learning, with the representative paradigm being RLVR, which gives binary rewards based only on whether predictions are correct (Lambert et al., 2025). The authors claim such training reinforces overconfident reasoning (Kirichenko et al., 2025; Mei et al., 2026), thereby limiting high-risk domains such as healthcare and finance (Lin et al., 2022).
• Existing limitations: Incorporating uncertainty estimation into training can improve calibration (Leng et al., 2025; Liu et al., 2025; Damani et al., 2026), but the paper claims these methods mainly calibrate answer confidence without checking whether the reasoning process is correct (Khalid et al., 2025; Chen et al., 2026). The authors claim that directly optimizing calibration objectives often leads to an accuracy-calibration trade-off; the caption of Figure 1 depicts this phenomenon as more pronounced on OOD.
• Core observation: The authors divide confidence into two types. Reasoning confidence is a self-assessment of whether the reasoning process is correct or truthful, while answer confidence is a self-assessment of whether the prediction is correct; the former is treated as an auxiliary signal for calibrating the latter.
• Proposed method: RL-ARC uses reasoning confidence in reinforcement learning based on whether predictions are correct: it regularizes answer confidence on correct samples and penalizes overconfidence on incorrect samples. The authors hope the model adaptively estimates confidence per question, improving calibration on ID and OOD without significantly sacrificing reasoning performance.

Verbalization and Other Uncertainty Estimation

• Tian et al. (2023), Xiong et al. (2024), and Dong et al. (2024) have models explicitly state their confidence in their own answers. The paper states that this is simple, human-interpretable, and correlated with performance and answer quality, but it remains overconfident and has limited ability to identify its own errors.
• Methods based on accuracy from multiple samples (Lin et al., 2024; Xue et al., 2025) estimate uncertainty using the accuracy of multiple generations. The paper states that inference cost is high and that estimates are unstable with prompt design (Zhao et al., 2021; Tonolini et al., 2024).
• Token-probability-based methods (Kadavath et al., 2022; Kuhn et al., 2023) estimate uncertainty from generation probabilities. The paper states that applicability is limited in open-ended generation and that mitigation often requires additional inference cost.

Calibration-Aware Training

• RLCR by Liu et al. (2025) and Damani et al. (2026), and Behavioral Calibration by Wu et al. (2026), incorporate uncertainty objectives into reasoning training to align self-evaluated confidence with prediction correctness. The paper states that they still have an accuracy-calibration trade-off and remain prone to overconfidence on OOD.
• Yoon et al. (2025) is used by the paper to illustrate that existing LMs can self-assess reliability, but confidence is often misaligned with true correctness and accompanied by overconfidence (Tian et al., 2023; Xiong et al., 2024; Mei et al., 2026).

Overconfidence in Reasoning Training

• RLVR (Lambert et al., 2025) improves reasoning using only prediction correctness. The paper states that such RL does not account for output confidence and reinforces overconfident reasoning (Kirichenko et al., 2025; Mei et al., 2026).
• Works such as Leng et al. (2025) incorporate uncertainty estimates into rewards. The paper lists this as a promising direction but argues that calibrating only answer confidence is still insufficient.

Baseline Methods and Benchmarks

• Baseline methods: Base (verbalized confidence of the pretrained model), RLVR w/ Confidence (Xiong et al., 2024), RLVR w/ Probability (Kuhn et al., 2023), RLVR w/ Post-hoc (Li et al., 2025, external confidence estimator), Behavioral Calibration, RLCR.
• Benchmarks/datasets: Training mainly uses Big-Math (Albalak et al., 2025), with HotpotQA also used; evaluation includes arithmetic reasoning, complex reasoning, and factual QA.

The authors position this work as: aligning self-evaluated confidence by considering not only prediction correctness but also the correctness or truthfulness of the reasoning process, so that the model accounts for errors identified in reasoning when estimating confidence.

RL-ARC (Reinforcement Learning with Auxiliary Reasoning Confidence) on verifiable-reward reinforcement learning elicits both reasoning confidence and answer confidence, and adjusts the calibration term based on whether the prediction is correct.

Problem Setting and Formalization

• The answer reward in standard RLVR is binary correctness. The Brier score is the squared error between confidence and correctness: Brier(y, c_a, y^∗)=(1_y=y^∗−c_a)². The authors follow the practice of writing the reward as correctness minus Brier, so that a high score requires both a correct answer and appropriate confidence.
• The total reward of RL-ARC is R(z, ca) minus the reasoning term. z indicates whether the prediction is correct, and R(z, ca)=z-(z-ca)^2. When correct, the reasoning term is Ω_arc(z, c_a, c_r)=λ_pos(c_a−c_r)²; when incorrect, the term is λneg times the square of cr. λpos and λneg control the strength of cr.

Confidence Elicitation

• Reasoning confidence cr: The model's verbalized self-assessment of whether the reasoning process is correct or truthful.
• Answer confidence ca: The model's verbalized self-assessment of whether the prediction is correct.
• Both are generated together with the reasoning and the prediction. The training prompt requires explicitly estimating confidence when solving problems; Figure 9 provides the template, and the full text is not transcribed in the main text.

Objective Modulated by Correctness

• Correct samples: R encourages high ca while reducing the gap between ca and cr. The authors claim this avoids giving high confidence merely because the prediction is correct, requiring the reasoning process to also be correct.
• Incorrect samples: When z=0, R penalizes high ca; then subtracting λneg cr^2 makes high reasoning confidence that leads to incorrect predictions also reduce the reward. The authors claim this mitigates overconfidence.
• Default λpos=0.4, λneg=0.15 (Appendix A.2).

Training Setup

• The algorithm is GRPO (Shao et al., 2024). Backbones are Qwen2.5-7B, Qwen3-8B, and Llama-8B distilled from DeepSeek-R1. All methods share hyperparameters.
• The main training set is 30,000 moderately difficult problems from Big-Math. In implementation, easy problems for LLaMA-8B are excluded, keeping only numerical answers (Appendix A.1–A.2). A second training distribution uses HotpotQA.
• Table 6: input max length 1024, output max length 4096, rollout group size 8, rollout temperature 0.7, 1 epoch, batch size 1536, learning rate 5e-6, linear decay, warmup ratio 0.2. Training is done on four NVIDIA H100 or A100 GPUs.

How Correctness and Calibration Are Determined

• Math problems use math-verify; HotpotQA uses exact match; StrategyQA, GPQA, SimpleQA, NQ-Open, and TriviaQA use LLM-as-a-judge. The paper does not name the judge model.
• Calibration metrics are Brier, ECE (10 bins), and AUROC. Selective prediction uses AURC and risk-coverage curves.
• Reasoning Calibration: First, an LLM judges whether the reasoning supports the standard answer, then ECE is computed by binning cr. Shallow Reasoning is the proportion of OOD cases where the prediction is correct but the reasoning is incorrect.

Experimental Setup

• Backbone: Qwen2.5-7B, Qwen3-8B, Llama-8B distilled from DeepSeek-R1. The optimization algorithm is GRPO for all, with the same hyperparameters (Table 6).
• Training data: The main experiments use 30,000 problems from Big-Math; HotpotQA is also used as a complex reasoning training distribution. Appendix A.2 states that easy problems for LLaMA-8B are excluded, leaving only numerical answers.
• Evaluation sets: The arithmetic evaluation in §4.1 includes Big-Math (eval 1,000), GSM8K (1,319), MATH-500 (500), AMC23 (40), AIME24 (30), AIME25 (30). OOD includes StrategyQA (229), GPQA (448), HotpotQA (1,000), SimpleQA (4,326), NQ-Open (3,610), TriviaQA (2,000). The per-item table for Qwen2.5-7B (Table 11) does not list AIME25, while Table 13 for Qwen3-8B lists AIME25.
• Baselines: Base, RLVR w/ Confidence, w/ Probability, w/ Post-hoc, Behavioral Calibration, RLCR. Tables 2 and 3 only report Base, RLVR, RLCR, RL-ARC.
• Metrics and criteria: Accuracy, Brier, ECE (10 bins), AUROC, AURC, as well as Reasoning Calibration and Shallow Reasoning. Math uses math-verify, HotpotQA uses exact match, and several tasks use LLM-as-a-judge. The paper does not specify the judge model name or report the number of repetitions. Acc in appendix tables uses proportions, while Acc in Tables 1–4 uses percentages.
• Default coefficients: λpos=0.4, λneg=0.15.

Main Results

Swipe horizontally to read the table

Model	Condition	RL-ARC Acc	RL-ARC ECE	RLVR Acc	RLCR ECE
Qwen2.5-7B	ID average	56.53%	0.18	56.86%	0.22
Qwen2.5-7B	OOD average	47.27%	0.16	47.58%	0.20
Qwen3-8B	ID average	66.10%	0.15	66.58%	0.16
Qwen3-8B	OOD average	51.61%	0.17	47.24%	0.22
Llama-8B (DeepSeek-R1 distilled)	ID average	56.69%	0.25	59.49%	0.29
Llama-8B (DeepSeek-R1 distilled)	OOD average	43.30%	0.19	40.48%	0.28

According to Table 1 (for the two Qwen models) and Table 2 (for Llama). The RLVR Acc for Qwen in Table 1 is the same across the three confidence extraction methods.

• The authors state that RL-ARC improves calibration on both ID and OOD without significantly sacrificing the accuracy of RLVR, which only optimizes correctness. In Table 1, the ID AUROC for Qwen2.5-7B is 0.72 for post-hoc and 0.60 for RL-ARC; the OOD AUROC for that post-hoc is 0.51. The authors state that external estimators trained on ID do not necessarily transfer to OOD.
• For Qwen's RLVR w/ Confidence, the ECE in Table 1 is ID 0.40 and OOD 0.50 for Qwen2.5-7B, and ID 0.32 and OOD 0.48 for Qwen3-8B. The OOD Base Acc for Qwen3-8B is 51.37%.
• The OOD Base Acc for Llama-8B is 45.96% (Table 2). The authors state that with this backbone, OOD accuracy still decreases with RL-ARC, but the decrease is mitigated compared to other training methods. Its ID AUROC is 0.81, and RLCR is 0.80.

Training Distribution

• What was tested: Qwen2.5-7B was trained on arithmetic reasoning and complex reasoning separately, reporting ID (Avg.) and OOD (Avg.) (Table 3). The paper does not specify item-by-item which benchmarks are included in the ID average for complex reasoning training; Appendix Tables 17 and 18 mark both arithmetic and factual QA evaluations under that setting as OOD.
• Results: Under complex reasoning training, RL-ARC has ID Acc 55.10% and ECE 0.06, RLCR 54.70% and 0.07, RLVR 56.30% and 0.43. On OOD, RL-ARC Acc is 45.42% and ECE 0.16, Base Acc is 45.41%, RLVR Acc is 39.51% and ECE 0.60.
• Authors' interpretation: The authors state that on both training distributions, RL-ARC improves both calibration and accuracy on OOD relative to other training methods. The arithmetic reasoning training row is the same as the Qwen2.5-7B row in Table 1.

OOD Tasks and Confidence Patterns

• What was tested: Qwen2.5-7B averaged over three types of complex reasoning and three types of factual QA (Table 4), and OOD confidence binning (Figure 3).
• Results: On complex reasoning, RL-ARC has Acc 53.36% and ECE 0.14, RLVR 54.86% and 0.45, RLCR 51.83% and 0.16. On factual QA, RL-ARC has Acc 41.19%, Brier 0.27, ECE 0.19, and AUROC equal to RLCR at 0.54. Exceptions: on GPQA, RL-ARC Acc is 0.38 and RLVR w/ Confidence is 0.48 (Table 15); on SimpleQA, RL-ARC AUROC is 0.45 and Base is 0.50 (Table 16); on GSM8K, RL-ARC ECE is 0.25, RLVR w/ Confidence is 0.10, and Base is 0.22 (Table 11, Acc as proportion).
• Authors' interpretation: The authors state that accuracy and calibration are best on factual QA. They state that in Figure 3, RL-ARC assigns OOD samples to low and medium confidence rather than concentrating them at high confidence; the converted text does not show bar heights. Appendix B.2 states that on ID the distribution is similar to RLCR, while on OOD the differences are larger.

Overconfidence Mitigation and Reasoning Process

• What was tested: Risk-coverage curves in Figure 4, AURC in Table 8, and reasoning calibration and shallow reasoning in Figure 6.
• Results: The authors state that in Figure 4, across most coverage levels on AIME24, GSM8K, GPQA, and NQ-Open, RL-ARC has lower or comparable selective risk, with more pronounced differences at low coverage; the text does not give curve endpoints. Table 8 does not specify the backbone; AURC is ID 0.16 for RL-ARC, 0.20 for RLCR, 0.22 for RLVR, and OOD 0.34, 0.35, 0.44; its Acc matches the Qwen3-8B row in Table 1. The y-axis of Figure 6 is percentage points relative to Base; the converted text does not show bar heights, and the main text does not give raw values for Reasoning Calibration and SR.
• Authors' interpretation: The authors state that high-confidence outputs are more likely to be correct. They state that in Figure 6, RL-ARC shows larger improvements in answer calibration and reasoning calibration, with reasoning calibration gains greater than answer calibration, the largest reduction in shallow reasoning, and that there was no direct supervision on reasoning correctness at the time.

Other Ablations and Analyses

• Table 5 (Qwen2.5-7B, OOD): with only λneg enabled, Acc 47.11%, ECE 0.15, AURC 0.46; with only λpos enabled, Acc 47.51%, ECE 0.21, AUROC 0.55; with both disabled, Acc 46.02%, ECE 0.20.
• Table 7: After shuffling RL-ARC confidence while preserving the histogram, ID ECE/Brier is 0.19/0.25 and OOD is 0.19/0.27; without shuffling, ID is 0.18/0.23 and OOD is 0.16/0.26.
• Table 9: Arithmetic mean OOD Acc 48.61%, ECE 0.25; geometric mean Acc 47.65%, ECE 0.23; the paper's objective Acc 47.27%, ECE 0.16.
• Table 10: Default λneg=0.15, λpos=0.40 gives ID Acc 56.53% and ECE 0.18; the highest ID Acc in the same table is 57.64% with λneg=0.20, λpos=0.60, whose OOD ECE is 0.17.
• Appendix B.6: The authors state that configurations that better mitigate the accuracy-calibration trade-off mostly have λpos/λneg>1, and increasing the influence of reasoning confidence does not necessarily continue to improve calibration.
• Qwen3-8B on AMC23 (Table 13, Acc as proportion): RL-ARC Acc is 0.85, RLVR is 0.90; ECE is 0.18, RLVR w/ Confidence is 0.09, and Brier is 0.06, the lowest in that table.

Limitations and future directions noted by the authors

• Experiments are limited to two training settings: limited math reasoning samples, or limited complex reasoning samples (Limitations).
• For fair comparison with baselines, all LMs use only GRPO. The authors believe that adding more diverse training data and stronger RL algorithms could further improve performance (Limitations).
• The main focus is on LMs, especially the Qwen family. The authors believe extending RL-ARC to multimodal reasoning models is a future direction (Limitations).
• Reasoning calibration and shallow reasoning frequency use LLM-as-a-judge. The authors observe that using it to evaluate reasoning correctness introduces errors, and leave more accurate reasoning correctness review as future work (Limitations).

Experimental coverage

• The tested backbones are Qwen2.5-7B, Qwen3-8B, and Llama-8B distilled from DeepSeek-R1, 3 in total (§4.1, Table 2).
• The training distributions are Big-Math (30,000 problems in the main text) and HotpotQA; the optimization algorithm is GRPO, and all methods share the hyperparameters in Table 6 (§4.1, Appendix A.2).
• The evaluations listed in the main text include the arithmetic benchmarks Big-Math, GSM8K, MATH-500, AMC23, AIME24, AIME25, and the OOD StrategyQA, GPQA, HotpotQA, SimpleQA, NQ-Open, TriviaQA (§4.1).
• Metrics include Accuracy, Brier, ECE (10 bin), AUROC, AURC, Reasoning Calibration, Shallow Reasoning; math uses math-verify, HotpotQA uses exact match, and several tasks use LLM-as-a-judge (Appendix A.1, A.3).
• The paper does not report the number of repetitions, nor does it specify the exact model name of the LLM-as-a-judge (Appendix A.1, A.3).

RL-ARC is a calibration-aware reinforcement learning framework that uses verbalized confidence in the reasoning process to modulate answer confidence.

• Problem: RLVR only rewards whether answers are correct. The authors claim this reinforces overconfidence; training that only calibrates answer confidence remains overconfident on OOD and often loses reasoning accuracy.
• Method: For the same question, elicit reasoning confidence cr and answer confidence ca. When the prediction is correct, reduce the gap between them; when the prediction is incorrect, penalize high ca, and let high cr further reduce the reward. The optimizer is GRPO, with default λpos=0.4 and λneg=0.15.
• Qwen2.5-7B: On OOD average, RL-ARC has Acc 47.27% and ECE 0.16; in the same table, RLVR w/ Confidence has Acc 47.58% and ECE 0.50, and RLCR has ECE 0.20 (Table 1).
• Qwen3-8B: On OOD average, RL-ARC has Acc 51.61% and ECE 0.17; Base has Acc 51.37%, and RLVR w/ Confidence has Acc 47.24% (Table 1).
• Llama-8B: The DeepSeek-R1 distilled model has OOD average Acc 43.30%, Base 45.96%, RLVR 40.48%, and RL-ARC has ECE 0.19 (Table 2). The authors claim OOD accuracy still decreases for this backbone, but is mitigated relative to other training methods.
• Authors' conclusion: Reasoning confidence can serve as an auxiliary signal for calibration training, enabling the model to estimate confidence per question and improve ID and OOD calibration without significantly sacrificing reasoning performance. The breakdown is not better in every cell: Qwen2.5-7B has ECE 0.25 on GSM8K, while Table 11 shows RLVR w/ Confidence at 0.10.