Skip to content

Translation in progress Some entries still include text in the original language.

Original
arXiv· arXiv:2610.21214· Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang, Zhixuan Chu, Kui Ren·Indexed by this site · Originally published

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Quick read

Evaluation and benchmarks

Based on 论文 PDF （AI 生成）; refer to the original

The authors propose URW, using paraphrase templates and conformal p-value weighted rubric judgment; the authors claim it improves preference Acc and RLVR reward.

Questions
Rubric evaluation of open-ended responses compresses each criterion into a binary judgment and sums them uniformly. The authors state that judgment difficulty varies, deterministic verifiers provide no confidence, and uncertain subjective criteria may flip preference near ties and give misleading rewards to downstream RL.
Methods
URW does not change deterministic decoding. Fixed paraphrase templates repeatedly ask the same criterion, and the average pass rate serves as a soft score; then split-conformal p-values are computed on a calibration set disjoint from test, normalized to serve as criterion weights. The authors give a finite-sample bound under exchangeability.
Experiments and results
In Table 1, across three verifiers and five benchmarks, URW's Acc is higher than the baseline in the same cell. In downstream GRPO, Qwen 3 8B's Arena-Hard WR is 76.5, and GLM 4 9B's is 48.7.
Limitations and future work
The authors point out that the theoretical guarantee relies on exchangeability between calibration and test, and distribution shift in RLVR stresses it; periodic recalibration is a future direction. URW only quantifies uncertainty exhibited by fixed paraphrase templates. Experiments include five preference benchmarks and 150 steps of GRPO for each of two policies.
Experimental setupQwen 3 8B、GLM 4 9B · RubricBench、ResearchQA et al. · Acc、RI et al.
Evaluated models
Qwen 3 8BGLM 4 9B
Benchmarks
RubricBenchResearchQARM-BenchRewardBench (Chat-Hard)RewardBench 2 (Precise-IF)RUBRIC-ARROW-Judge-SFTRubricHubArena-HardGPQA-Diamond
Metrics
AccRIWR
AI summaryThe paper proposes a training-free framework, Uncertainty-aware Rubric Weighting (URW), which quantifies rubric-level judgment uncertainty via conformal calibration and converts it into per-criterion weights. Existing rubric-based evaluation aggregates the binary judgments given by an LLM verifier on each criterion into a final score, ignoring differences in judgment certainty across criteria, which can provide misleading training signals to downstream reinforcement learning. URW extracts soft uncertainty signals from a set of rewritten judgment templates, converts them into per-item weights using split-conformal p-values, and provides provable coverage guarantees. The authors state that experiments show the method consistently improves pairwise preference accuracy and leads to stronger downstream RL alignment.

The paper proposes a training-free framework, Uncertainty-aware Rubric Weighting (URW), which quantifies rubric-level judgment uncertainty via conformal calibration and converts it into per-criterion weights. Existing rubric-based evaluation aggregates the binary judgments given by an LLM verifier on each criterion into a final score, ignoring differences in judgment certainty across criteria, which can provide misleading training signals to downstream reinforcement learning. URW extracts soft uncertainty signals from a set of rewritten judgment templates, converts them into per-item weights using split-conformal p-values, and provides provable coverage guarantees. The authors state that experiments show the method consistently improves pairwise preference accuracy and leads to stronger downstream RL alignment.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The problem is that rubric binary judgments lack criterion-level uncertainty; the authors state that uniform aggregation may flip preference and mislead RL.

    The deterministic verifier's binary rubric judgments carry no criterion-level uncertainty, and uniform aggregation cannot weight by judgment certainty.

    • Scenario: Rubrics are commonly used to evaluate open-ended outputs. Given a prompt and response, quality is decomposed into K prompt-specific binary criteria, the LLM verifier outputs 0/1 for each, and a weighted sum yields the response-level score (Eq. (1), Section 2.1). The authors state this score is used for candidate ranking, RL rewards, and agent verification, and the aggregation method affects both evaluation and policy optimization.
    • Current limitation: The authors state that rubric generation has advanced, but aggregation still commonly uses wk=1/K. Binary outputs have no probabilities, logits, or confidence; raising temperature to estimate confidence deviates from the deterministic decoding commonly used in reference verification and is detrimental to policy optimization that needs stable rewards. The authors state that tone, helpfulness, and content quality are harder to judge than format or length constraints, and near ties a single uncertain judgment can flip overall preference.
    • Case and hypothesis: In Figure 1, two interview suggestions tie on objective criteria, and preference depends on the empathetic tone criterion c5; the verifier assigns that criterion to the blunt response, and uniform aggregation contradicts human preference. The authors believe reliable judgments should be stable under meaning-preserving paraphrases, and uncertain judgments are more likely to flip, so raising decoding temperature is unnecessary.
    • Proposed: A training-free framework Uncertainty-aware Rubric Weighting (URW). A fixed paraphrase template extracts criterion-level soft signals, which are converted into aggregation weights via split-conformal p-values. The authors state that when calibration and test are exchangeable, the probability of assigning too low a weight to a correct vote is bounded; and that no existing method simultaneously provides training-free, coverage-controlled, inference-time criterion-level weights applicable to any binary rubric verifier.

  2. What related research exists?

    The authors position this work between rubric evaluation and judge uncertainty estimation, with the distinction being criterion-level conformal weights.

    The introduction and Appendix A divide related work into rubric evaluation and judge uncertainty estimation. The authors state that no existing method simultaneously provides training-free, coverage-controlled, inference-time criterion-level weights applicable to any binary rubric verifier.

    Rubric evaluation and fixed-weight aggregation

    • Checklists and fine-grained criteria: Cook et al. (2024) use generated checklists to guide evaluation and generation; Arora et al. (2025), Li et al. (2026b) build instance-level fine-grained criteria for open-ended outputs.
    • Generation, rewards, and agents: Chen et al. (2026b) do self-evolving rubric generation, Chen et al. (2026a) use meta-rubrics to compile requirements into binary checks. Mu et al. (2024), Gunjal et al. (2026), Shan and Shao (2026) use rubrics as training rewards; Wang et al. (2026c), Liu et al. (2026) use them for agent improvement lacking precise verifiers. The authors state these works improve construction, coverage, or downstream use, but aggregation still commonly uses fixed weights; URW keeps the original rubric and verifier, only calibrating each judgment before aggregation.

    Response-level conformal and calibration tools

    • Score intervals or confidence labels: Wagner et al. (2024) do black-box uncertainty labeling; Sheng et al. (2025) give intervals for aggregated scores; Liu (2026) does multi-agent conformal consensus intervals. The authors state these characterize the final score, not criterion-level weights.
    • Split conformal: Lei et al. (2018) give finite-sample marginal coverage under exchangeability. The authors contrast URW with "outputting response-level intervals, sets, or abstention": URW evaluates the plausibility of each vote label and directly uses conformal p-values as continuous weights.
    • Calibration under shift: Tibshirani et al. (2019), Gibbs and Candes (2021) extend calibration to covariate shift or distribution shift.

    Decoding consistency and heterogeneous committees

    • Sampling consistency: Li et al. (2026a), Cao et al. (2026) derive confidence from decoding-time consistency. The authors state this requires nonzero temperature, incompatible with the common deterministic, single-verifier setting.
    • Committee disagreement: Fathullah and Gales (2025) use a heterogeneous verifier committee. The authors state this requires multiple verifiers, also deviating from single-verifier deterministic reference decoding. URW keeps a single verifier frozen and deterministic, using only fixed meaning-preserving templates to probe stability.

    Trained uncertainty modules

    • Evaluators and probes: Xie et al. (2025) fine-tune evaluators with uncertainty; Radharapu et al. (2026) use linear probes for calibration. The authors state both require labeled training and output extra labels, not scalar weights that can directly enter a weighted sum with coverage control. URW calibrates on held-out annotations without training an additional evaluator.

    Baseline methods and benchmarks: Comparisons are Uniform (wk=1/K), Beta-Binomial, Sampling Ensemble (Wang et al., 2023), SCP, and Mondrian-SCP. Pairwise preference benchmarks are RubricBench, ResearchQA, RM-Bench, RewardBench's Chat-Hard, and RewardBench 2's Precise-IF.

    The authors place URW alongside the above works: it does not change criterion construction, nor make uncertainty into response-level intervals or extra labels, but gives a frozen binary verifier criterion-level p-value weights that can enter a weighted sum.

  3. How does the paper address this problem?

    URW uses vote stability across fixed paraphrase templates to obtain soft scores, then converts them into conformal p-values as weights.

    URW 不训练 verifier，也不提高解码温度。它用语义保持的释义模板看二元判断是否稳定，再把该信号经 split conformal 校准成准则权重。

    问题设定

    • 对象：prompt q、回答 y、准则集 r(q)={c1,…,cK}。verifier v 对每个三元组独立输出 v(q,y,ck)∈{0,1}。作者把 verifier 视为 API 黑盒。
    • 均匀聚合：权重非负且和为 1 时，响应分数是各准则判断的加权和。作者称现有 rubric judge 通常取 wk=1/K。
    • 要构造的量：随实例变化的权重 ŵk(q,y)，使更不确定的判断对响应级分数影响更小。作者称确定性 verifier 不直接暴露确定性，权重须从可观察行为得到。

    模板集成

    • 模板池：T={t1,…,tN}。作者团队交叉检查释义是否保持原准则含义；附录 C.2 给出完整模板。实验前冻结 N=6。一条模板若在 50 对抽查中有超过 2 对被判定改变语义，则丢弃。
    • 软分数：温度 0 下对每条模板解码，再取平均，v̂k∈{0, 1/N, …, 1}。作者称接近 0 或 1 表示确信，接近 0.5 表示最不确定。
    • 校准分布：在与评测划分不相交的 Dcal 上记录软分数和非符合分。Section 3.2 写 si=v̂i−vi*∈[0,1]；式(4)与附录证明使用绝对偏差。数据来自 RUBRIC-ARROW-Judge-SFT，按领域分层抽 200 个 prompt，约 2,000 条准则级标注。

    p 值权重

    • p 值：候选标签 u∈{0,1} 的非符合分为 s(v̂k,u)=|v̂k−u|。其校准排名写成 p(u|vhatk)=(1+|i:si>=s(vhatk,u)|)/(Ncal+1)。作者称它表示 u 作为真标签的合理性。
    • 投票权重：ṽk=round(v̂k)。v̂k≥0.5 时原始权重为 p(1|v̂k)，否则为 p(0|v̂k)。作者称该权重随 |2v̂k−1| 单调不减，在 0.5 最小、在 0 或 1 最大，不确定判断会被降权。Proposition B.4 给出这一单调性。
    • 归一化：wk 为原始权重在 K 条准则上归一后的值。最终分数使用加权后的投票标签，而不是软分数：对模板输出取 round 后再加权求和（式(6)）。URW 直接使用校准分数集，不再设误覆盖率 α。

    保证范围

    • Theorem 3.1：若校准与测试在同一 verifier 和模板池下可交换，则对任意 α∈(0,1)，正确投票且权重不超过 α 的概率不超过 α。作者称因此不必预先固定 α。
    • 未覆盖的方向：Remark B.3 写明，定理只约束正确判断很少被赋过低权重，并不保证错误判断得到低权重；后者取决于多模板分数能否分开这两类判断。作者将其放到 Section 4.2 的实验中查看。

  4. What experiments did the paper conduct?

    In Table 1, URW's Acc is higher than the baseline in the same setting across 3 verifiers and 5 benchmarks.

    Experimental setup

    • Main experiment verifiers: DeepSeek V4 Pro, Qwen 3.8 Max, GLM 5.3, all frozen. Calls use temperature 0, JSON mode, high reasoning effort, and max output of 32,768 tokens. Main experiment template count N=6.
    • Other models: Figure 2 uses Qwen 3 and GLM 4, from 8B/9B to 32B (denoted Q3-8B, Q3-32B, G4-9B, G4-32B in the text). RL policies are Qwen 3 8B and GLM 4 9B, with verifiers Qwen 3 32B and GLM 4 32B, respectively. Benchmarks without human rubrics use RubricARM-8B-Rubric to generate criteria. Arena-Hard judge is DeepSeek V4.1 Flash, temperature 0.0, with response positions swapped.
    • Benchmark sizes: RubricBench 1,147 pairs, ResearchQA 169 pairs, both with human rubrics. RM-Bench uses diagonal pairing to get 387 pairs. RewardBench takes Chat-Hard 456 pairs, RewardBench 2 takes Precise-IF 480 pairs. The latter three have model-generated rubrics.
    • Calibration: RUBRIC-ARROW-Judge-SFT, stratified by domain to sample 200 prompts, about 2,000 annotations, disjoint from all evaluation splits.
    • Baselines and metrics: Uniform; Beta-Binomial with prior Beta(1,1), posterior mean as weight; Sampling Ensemble, each criterion independently decoded 6 times at temperature 0.7; SCP and Mondrian-SCP, α=0.1, Mondrian with 6 bins. Acc is pairwise accuracy, ties count as half a point. RI is relative improvement over Uniform. The paper does not report repetitions for main-table Acc.
    • RL: RubricHub's Chat and Science, 1,000 prompts each (first removing prompts with fewer than 4 or more than 20 rubric criteria), GRPO for 150 steps each. Zero-shot evaluation on Arena-Hard (500 items, against official gpt-4-0314 responses, reporting WR) and GPQA-Diamond (198 questions, reporting Acc). Policy evaluation temperature 1.0, max output 16,384 tokens. All methods share the same training set and framework, changing only reward aggregation.

    Main results

    URW's Acc, with RI relative to Uniform in parentheses. From Table 1.

    Swipe horizontally to read the table

    VerifierRubricBenchResearchQARM-BenchRewardBenchRewardBench 2
    DeepSeek V4 Pro0.802 (2.79%↑)0.698 (10.28%↑)0.792 (8.69%↑)0.810 (5.72%↑)0.759 (21.50%↑)
    Qwen 3.8 Max0.819 (3.58%↑)0.692 (13.59%↑)0.786 (5.56%↑)0.810 (3.65%↑)0.721 (6.13%↑)
    GLM 5.30.818 (1.84%↑)0.680 (7.98%↑)0.757 (3.53%↑)0.788 (3.75%↑)0.769 (5.28%↑)
    • The authors state that uniform aggregation ignores differing uncertainty across criteria. Not every weighting method in Table 1 is above Uniform: on DeepSeek V4 Pro, RM-Bench, Sampling Ensemble's Acc is 0.711 (RI 2.48%↓), below Uniform's 0.729.
    • The authors state that heuristic weighting comes from non-uniform weights across templates, citing DeepSeek V4 Pro, RewardBench 2: Uniform is 0.625, Beta-Binomial is 0.688, Sampling Ensemble is 0.731.
    • The authors state that URW is above all listed baselines on every verifier–benchmark, attributing this to continuous p-values rather than hard abstention preserving uncertainty. In Table 1, for every combination of three verifiers and five benchmarks, URW's Acc is higher than other methods in the same cell.

    Ablation of two components

    • What was tested: Table 2 separates template ensemble and conformal calibration. URW† keeps uniform weights but uses multi-template vote scores; URW‡ still uses single-template criterion scores, with uncertainty from two-template conformal p-values.
    • Results: The authors state that URW† on ResearchQA is up to 3.9% relative to Uniform; in the table, Qwen 3.8 Max is 0.633 vs 0.609 (3.88%↑). Exceptions: DeepSeek V4 Pro's URW† on RubricBench is 0.779, below Uniform's 0.780 (0.11%↓); GLM 5.3 is 0.801 vs 0.803 (0.33%↓). URW‡ is above Uniform in every cell of Table 2, e.g., Qwen 3.8 Max, ResearchQA is 0.666 (9.22%↑), DeepSeek V4 Pro, RewardBench 2 is 0.716 (14.50%↑).
    • Authors' interpretation: The authors state that calibration can distinguish judgments even without using template ensemble in the final score; full URW is above both ablated variants in all cells of Table 2, and the authors state the two can be used together.

    Across verifiers and rubric sources

    • Scale: Figure 2 from 8B/9B to 32B. The authors state that URW is above Uniform on all five benchmarks, in all 20 verifier–benchmark comparisons, with improvements ranging from 0.6 to 15.1 relative accuracy points. The percentage labels visible in the figure carry negative signs, from -0.6% to -15.1% (e.g., RM-Bench -15.1%, ResearchQA -10.6%). The authors state URW is above Uniform; the Figure 2 labels themselves are negative, and the text does not explain the sign.
    • Rubric source: Tables 3 and 4 replace human-reviewed rubrics with LLM-generated rubrics. Absolute Acc drops, the authors cite Qwen 3.8 Max, RubricBench: Uniform from 0.791 to 0.538. The authors state URW's improvement over Uniform is 1.6%–13.6%; in the tables this corresponds to DeepSeek V4 Pro, ResearchQA, LLM rubric at 1.6%↑, and Qwen 3.8 Max, ResearchQA, human rubric at 13.6%↑.
    • Baselines below Uniform: The authors state that Sampling Ensemble and SCP are below Uniform in some generated-rubric settings. Table 3, Qwen 3.8 Max, RubricBench: Sampling Ensemble is 0.536 (0.4%↓), Uniform is 0.538; on GLM 5.3, SCP is 0.557 (0.1%↓). URW has the highest Acc in every configuration in Tables 3–4.

    Downstream RLVR

    • Protocol: Use the same rubric reward to drive GRPO, changing only aggregation. Policies are Qwen 3 8B (Table 5) and GLM 4 9B (Table 6), each 150 steps.
    • Results: All uncertainty weighting methods in the tables have WR and Acc above Uniform. On Qwen 3 8B, URW's Arena-Hard WR is 76.5 (RI 23.64%↑), GPQA-Diamond Acc is 65.7 (19.27%↑); Uniform is 61.9 and 55.1. On GLM 4 9B, URW's WR is 48.7 (62.06%↑), Acc is 50.0 (50.00%↑); Uniform is 30.0 and 33.3. The authors state that relative to Mondrian-SCP, GLM 4 9B's WR goes from 42.4 to 48.7, and Acc from 43.4 to 50.0.
    • Cases: In two GPQA-Diamond examples in Appendix D.2, the authors show that non-URW policies choose the wrong final option; URW-trained Qwen 3 8B and GLM 4 9B choose correct options D and A, respectively. The authors state these two examples do not constitute a general guarantee.

    Other ablations and analyses

    • Template count N (Section 4.5, Figure 3): N=1 corresponds to Uniform. The authors state that Acc rises most from N=1 to a small template pool, with smaller changes afterward, possibly fluctuating within a 15% confidence interval. The figure text does not give Acc values that can be transcribed point by point.
    • Appendix D.1, Figure 4: On RM-Bench, RewardBench, RewardBench 2, the authors state that multiple templates are generally above Uniform, and after small N the curve largely flattens or changes only slightly.
    • In Table 1, SCP on DeepSeek V4 Pro, RubricBench has RI 0.00%, with Acc still 0.780, same as Uniform.
    • Other generated-rubric cells below Uniform: Table 4, Qwen 3.8 Max, ResearchQA, Sampling Ensemble is 0.571 (2.0%↓), Uniform is 0.583; Table 3, GLM 5.3, RubricBench, Sampling Ensemble is 0.556 (0.2%↓).

  5. What are some points that can be further explored?

    The authors note that distribution shift in RLVR can compress exchangeability, and the template pool cannot separately identify other uncertainties.

    Limitations and future directions noted by the authors

    • Exchangeability is compressed by training shift: The theoretical guarantee depends on calibration triples and test triples being exchangeable. The authors state that distribution shift during RLVR training compresses this condition. A future direction is periodic recalibration and adaptive inference (Gibbs and Candes, 2021). From Limitations & Future Work in the Conclusion.
    • Uncertainty beyond the template pool: URW only quantifies the uncertainty exhibited by fixed semantically preserving templates and cannot separately identify inherent ambiguity in the rubric or the underlying task. The authors state that incorporating other uncertainty signals while preserving calibration guarantees is a future direction. From the same section.

    Experimental coverage

    • Pairwise preference evaluations are RubricBench (1,147 pairs), ResearchQA (169 pairs), RM-Bench (387 pairs), RewardBench's Chat-Hard (456 pairs), and RewardBench 2's Precise-IF (480 pairs) (Appendix C.1).
    • The main-result verifiers are DeepSeek V4 Pro, Qwen 3.8 Max, and GLM 5.3; Figure 2 additionally reports 20 verifier–benchmark comparisons for Qwen 3 and GLM 4 from 8B/9B to 32B (Section 4.4).
    • Rubric sources include human review and RubricARM-8B-Rubric generation; calibration uses 200 prompts and about 2,000 annotations from RUBRIC-ARROW-Judge-SFT, disjoint from the evaluation split (Section 4.1).
    • Downstream policies are Qwen 3 8B and GLM 4 9B, each with 150 steps of GRPO, with verifiers from the same family at 32B; evaluations are Arena-Hard (500) and GPQA-Diamond (198) (Section 4.6).
    • The paper does not report repetitions or significance tests for the main-table Acc; templates were frozen before the experiment, and each was cross-reviewed by another author on 50 pairs (Appendix C.2).

  6. Summarize the main content of the paper

    URW weights the rubric by criterion-level conformal p-values, and both Acc and RLVR reward in Table 1 are higher than Uniform.

    URW is a training-free criterion-level weighting that changes how rubric judgments are summed, without changing the verifier's deterministic decoding.

    • Problem: Open-ended evaluation decomposes quality into binary criteria and sums them uniformly. The authors state that deterministic black-box verifiers provide no confidence, and subjective criteria are treated as equally certain as format-type criteria; near ties, a single uncertain judgment can flip preference and give misleading RL rewards.
    • Method: Fixed paraphrase templates repeatedly ask the same criterion at temperature 0, averaging to a soft score; the split-conformal p-value of the vote label, normalized, serves as the weight. Theorem 3.1 states: when calibration and test are exchangeable, the probability that a correct vote is assigned weight not exceeding α is at most α. This bound does not guarantee that incorrect judgments are downweighted.
    • Preference Acc: In Table 1, across three verifiers and five benchmarks, URW's Acc is higher than the baseline in the same cell. On DeepSeek V4 Pro, RewardBench 2, URW is 0.759, with RI relative to Uniform of 21.50%↑, and Uniform is 0.625. The same table also has cells where baselines are below Uniform, such as Sampling Ensemble at 0.711 on RM-Bench for that verifier.
    • Downstream rewards: After 150 steps of GRPO each, Qwen 3 8B's Arena-Hard WR is 76.5 (Uniform 61.9), and GPQA-Diamond Acc is 65.7 (Uniform 55.1). GLM 4 9B has WR 48.7 (Uniform 30.0) and Acc 50.0 (Uniform 33.3). The authors state that continuous uncertainty weighting gives more effective rewards than hard abstention.
    • Authors' conclusion: The authors state that URW improves preference aggregation across multiple verifiers and benchmarks and provides more effective rewards for RLVR. The guarantee depends on exchangeability and currently only covers uncertainty exhibited by template paraphrases.
Read originalarxiv.org