Paper audits LLM judge reliability: repeated agreement does not equal correctness

LLM judges' verdicts are treated as ground truth, but repeated queries and swapping candidate order both change the verdicts.

• Scenario: The authors say LLM-as-a-Judge provides signals for RL rewards, model comparison, automated scoring, and model selection; verdicts are recorded, aggregated, and optimized.
• Shortcoming: The authors say common practice assumes verdicts are a stable function of the input: repeated queries with the same prompt yield the same result, and swapping the order of two candidates does not change the preference. The introduction says neither assumption holds in practice.
• Observation: The introduction says 7–17% of verdicts change under identical repetition, and flip rates reach up to 98% when presentation order is swapped. One judge returns the same verdict 100% of the time across repeated calls yet matches ground truth only 51% of the time; the authors say this is barely better than random, and determinism is not reliability.
• Proposed contribution: Cross-audit six frontier judges and define a trustworthy verdict rate T, namely the probability that a verdict is simultaneously reproducible, order-invariant, and correct. The authors say position bias gives an accuracy upper bound that capability cannot break through, and trustworthiness is item-specific rather than model-level; switching from pairwise win rates to holistic rubric scoring improves T more than any single format prompt intervention.

The authors organize the literature by evaluation purpose, presentation sensitivity, and uncertainty, and place this paper on the question of whether verdicts remain trustworthy after repeated sampling, order swaps, and format changes.

Uses and Benchmarks of LLM Judges

• Preference and reward: Zheng et al. (2023)'s MT-Bench reports GPT-4 judges can achieve high agreement with human preferences; Dubois et al. (2023)'s AlpacaFarm uses LLM judges to simulate RLHF.
• Objective and adversarial: Tan et al. (2025)'s JudgeBench provides objective ground truth; Zeng et al. (2024)'s LLMBar tests adversarial robustness in instruction-following evaluation.
• Survey: Li et al. (2025) organize the field by what to evaluate, how to evaluate, and how to benchmark. The authors say most of this work reports agreement or accuracy.

Order, Phrasing, and Intra-Rater Inconsistency

• Presentation effects: Wang et al. (2024) and Shi et al. (2025) discuss answer order and presentation. The authors say this paper formalizes position sensitivity as flip rate φ from these observations and derives the upper bound 1−φ/2.
• Phrasing and overconfidence: Raina et al. (2024) study whether adversarial phrasing can shift zero-shot evaluations; Lu et al. (2025) discuss judges being overconfident on ambiguous samples but not accurate.
• Inconsistency: Haldar and Hockenmaier (2025) and Janiak et al. (2025) report low intra-rater reliability and overconfident inconsistency. The authors say this paper combines such observations with random agreement C into T.

Uncertainty and Structured Evaluation

• Abstention and calibration: El-Yaniv and Wiener (2010) study trading accuracy for abstention; Chirkova et al. (2026) use conformal prediction to analyze LLM judges, requiring held-out calibration data.
• Aggregation: Sheng et al. (2025) report that average scores can outperform majority voting, chain-of-thought can hurt some judges, and checklist rubrics can improve agreement; Lee et al. (2025)'s CheckEval uses checklists for text evaluation.
• Authors' connection: The authors say agreement across repeated queries can serve as a confidence signal without a calibration set, and G-Eval-style single-prompt multi-criteria scoring improves T more than separate rubric voting.

Baselines and Datasets

• Baseline methods: The main grid compares five pairwise prompts, with Direct being single-word verdicts. Paradigm comparisons additionally use FLASK (Ye et al., 2024; majority vote after four calls) and G-Eval (Liu et al., 2023; one call, four scoring criteria).
• Benchmarks: MT-Bench, AlpacaFarm, JudgeBench, LLMBar.

The authors say this paper does not treat single-shot accuracy as a sufficient characterization of judge quality but uses T to simultaneously account for reproducibility, order invariance, and correctness; and that choice of evaluation paradigm can change reliability more than any single-format intervention.

作者把裁判输出拆成可复现、换序不变和正确，用可信判决率（trustworthy verdict rate, T）合成；提示格式不进入T，而是单独的设计因子。

问题设定

每个实例是一对候选(x, y)加真值标签。裁判对任一展示给出判决V，取值为x或y。三条轴是随机不稳定、位置偏置和格式敏感。前两条进入T；格式集合F中的每个模板f单独计算后续各量。

三个成对指示：

• N：独立重复运行中判决可复现
• P：交换候选顺序后判决不变
• R：判决与真值一致

总体率：C=Pr(N=1)；φ为位置翻转率，1−φ=Pr(P=1)；A=Pr(R=1)。P=1的对称为decisive，其上准确率为Adec；P=0称为undecisive，其上准确率为q。

假设与准确率上界

• A1：核心实验强制二选一，两种顺序等权，不输出平局。作者称此时翻转对的两种顺序里恰好一个与真值一致，故q=1/2。若允许平局、弃权、非二元标签或不等权，作者写明q可以不是1/2。
• A2：可复现性不告知位置是否决断，且在一致又决断的对上，正确率等于Adec。附录G称实践中A2未必成立。

作者由A=(1−φ)Adec+φq且Adec≤1，在q=1/2时得到A ≤ 1 − φ/2。含义是：φ固定时，可观察准确率不能超过1减φ/2。作者称该上界是紧的，当且仅当所有decisive对都判对时取到，并且与模型能力无关。作者把1−φ/2−A称为capability gap，即与位置偏置正交的那部分误差。

可信判决率与格式极差

可信判决指N、P、R同时为1。作者在q=1/2并使用A2时写成

T = C · (A − φ/2)

含义是噪声存活比例C，乘以扣掉undecisive基线后剩下的准确率。作者称两名裁判可以A相同，却因C或φ不同而T不同。每个（裁判，格式，数据集）格子单独报告(C, φ, A, T)。格式敏感是同一裁判、同一数据集上各格式T的最大值减最小值，记为ΔTfmt。

附录G比较不依赖A2的联合概率Tjoint与闭式Tclosed。作者称即使A2不成立，T仍表现为保守下界。Table 10的Gap介于0.00与-0.05。

审计网格

• 数据：主观/客观×标准/对抗。MT-Bench与AlpacaFarm为标准主观；JudgeBench为客观；LLMBar为对抗。附录F：各100对，实验开始时固定。前两者随机抽样，seed=42；JudgeBench从620例按17个领域分层；LLMBar按5个划分均匀抽样。
• 裁判：GPT-5.2、GPT-4o、Gemini-2.5-Pro、Gemini-2.0-Flash、Claude-Sonnet-4.6、Llama-3.3-70B。其中GPT-5.2与Gemini-2.5-Pro为推理模型。
• 格式：五种输出指令共用系统提示和用户模板。Direct为单词A或B、无理由，最大32 token。Verdict-first与CoT各有JSON和XML，最大4096 token；前者先判决后理由，后者先理由后判决。作者称选用JSON与XML是为隔离常见结构化格式本身是否影响裁判。
• 顺序与温度：每对做AB和BA。每个（对，裁判，格式，顺序）重复10次。Ts=0覆盖6个裁判；Ts=0.5与1.0只覆盖4个，两名推理模型的公开API不支持温度参数。

FLASK与G-Eval

这两套只加在JudgeBench和LLMBar，用来看把判断拆成显式准则能否降低φ并提高准确率上界。Direct为1次成对调用。FLASK对4条准则各调用1次，再多数投票。G-Eval在1次调用里对4条准则打1到5分，胜者由分数比较决定。JudgeBench的准则围绕事实、答案或逻辑、完整性与指令；LLMBar围绕指令遵循、完整性、相关性与清晰度。附录D给出完整提示词。

Experimental Setup

• Judges evaluated: GPT-5.2, GPT-4o, Gemini-2.5-Pro, Gemini-2.0-Flash, Claude-Sonnet-4.6, Llama-3.3-70B. No separate scoring model was set up; the A control benchmarks already have labels.
• Data: 100 pairs from each of four benchmarks. JudgeBench labels come from MMLU-Pro, LiveCodeBench, and LiveBench, with one answer per pair verifiable as correct. Most LLMBar pairs contain adversarial distractors. MT-Bench is crowd-sourced preferences; Appendix F says AlpacaFarm label distribution is balanced.
• Grid: 5 formats × 2 orders × 10 repetitions. Ts=0 includes all 6 judges; Ts=0.5 and 1.0 only include GPT-4o, Gemini-2.0-Flash, Claude-Sonnet-4.6, and Llama-3.3-70B.
• Metrics: C is the proportion of verdicts that are deterministic across 10 repetitions; φ is the proportion of verdicts that flip after swapping order; A is the proportion consistent with ground truth; T=C·(A−φ/2). There is also format range ΔTfmt.
• Paradigms: Direct, FLASK (4 calls, majority vote), G-Eval (1 call, 4 criteria rated 1–5), the latter two compared only on the two objective benchmarks.
• Cost: Appendix E says public API costs about $2,100, with pairwise experiments $1,450, FLASK $390, G-Eval $260.

Main Results

The table below extracts only the JudgeBench, Direct, Ts=0 column from Table 1, in %. The same table has lower T values not included in this excerpt.

Swipe horizontally to read the table

Judge	C	φ	A	T
GPT-5.2	86.0%	11.0%	64.5%	50.7%
GPT-4o	94.5%	24.0%	51.0%	36.9%
Gemini-2.5-Pro	81.5%	56.0%	63.5%	28.9%
Gemini-2.0-Flash	99.0%	46.0%	55.5%	32.2%
Claude-Sonnet-4.6	98.8%	14.9%	57.3%	49.3%
Llama-3.3-70B	100.0%	26.0%	55.5%	42.5%

• Upper bound achieved: The authors say Table 1 has cells at or near A≤1−φ/2. The specific cell named in the text is Gemini-2.0-Flash, VF–XML, JudgeBench: φ=98.0%, the upper bound and observed A the authors write are both 51.0%, C=100.0%, T=2.0%. The same model's VF–JSON has φ=82.0%, the author-written upper bound is 59.0%, A=52.0%, T=10.8%.
• Format can change both A and φ: Gemini-2.5-Pro on JudgeBench, VF–JSON has A=88.5%, φ=6.9%, T=77.5% (Table 1). The authors say accuracy gain is 25 percentage points and T gain is 49 percentage points relative to Direct.
• High C can correspond to low T: The authors say Llama-3.3-70B has C≥98% in every cell, with T from 25.4% to 42.5% and φ from 22% to 45% on JudgeBench. In the table, that model's φ endpoints on JudgeBench are 22.4% and 45.5%. The authors say Gemini-2.0-Flash has C from 96–100% and T from 2.0% to 52.3%; when A−φ/2 is small, C=100% cannot raise T.

Scoring Paradigms

Table 2 compares Direct, FLASK, and G-Eval at Ts=0.

• FLASK lowers C: The authors cite GPT-5.2 on JudgeBench with C dropping from 86.0% to 34.8% and T from 50.7% to 20.5%. The text says FLASK's T is lower than Direct for five out of six judges; Table 2's JudgeBench is not like this: lower than Direct are GPT-5.2, GPT-4o (12.8% vs 36.9%) and Llama-3.3-70B (27.2% vs 42.5%), while Gemini-2.5-Pro, Gemini-2.0-Flash, and Claude-Sonnet-4.6 are higher. On LLMBar, all six judges' FLASK T is lower than their Direct, e.g., Claude-Sonnet-4.6 at 43.8% vs 73.2%.
• G-Eval has gains and losses: Gemini-2.5-Pro on JudgeBench T goes from 28.9% to 77.9%, which the authors call +49 percentage points, exceeding any format swing in Table 4. In the same table, cells that drop include GPT-4o from 36.9% to 28.0% (φ from 24.0% to 61.0%), and Llama-3.3-70B from 42.5% to 23.7%. Claude-Sonnet-4.6 has identical T for Direct and G-Eval on LLMBar at 73.2%.

Temperature and Variance Sources

• Temperature: Across the four temperature-tunable judges, as Ts goes from 0 to 1.0, average C drops by 12.1 percentage pointsтного and A−φ/2 changes by only +0.3 (48 cells). Figure 3 annotates the T change as −5.3. The authors say the decline in T is almost entirely from the noise survival term.
• ANOVA: Among the 120 Ts=0 cells, judge×dataset η² is 28.1%, judge is 27.0%, dataset is 26.2%, format is 0.7% (Appendix C.1). The authors say no judge ranks first on all benchmarks.
• G-study: Appendix C.2 says the order main effect accounts for 2–6%, and 6.2% on JudgeBench; pair×judge is the largest within-judge source of disagreement, at 34.8% on JudgeBench. The authors say reliability is item-specific rather than judge-level.

Additional Ablations and Analyses

• On AlpacaFarm, GPT-5.2's CoT–JSON has A=57.7%, T=40.2%, φ=15.5%; Claude-Sonnet-4.6's Direct has A=54.7%, T=49.9%, φ=7.4% (Table 1).
• ΔTfmt: Gemini-2.5-Pro on JudgeBench is 48.6 (T from 28.9% to 77.5%); Gemini-2.0-Flash on JudgeBench is 34.1 (Tmin=2.0); Claude-Sonnet-4.6 on LLMBar is 7.8 (73.2% to 81.0%) (Table 4).
• Judge-average ΔTfmt across four benchmarks: MT-Bench 9.5, AlpacaFarm 9.6, JudgeBench 22.7, LLMBar 10.8; GPT-5.2's mean is 4.8, Llama-3.3-70B's is 7.1 (Table 4).
• The introduction says verdicts change 7–17% under identical repetitions; the lowest C in Table 1 is 75.5% (GPT-5.2, JudgeBench, CoT–XML). The paper does not state how 7–17% is aggregated from C.
• Figure 4 divides cells into four regions by (C, φ). The authors say the cell with φ=98%, T=2.0% is in Confidently biased; high C in this region creates a credible appearance. The authors say in Figure 5, Claude-Sonnet-4.6 has the highest average T, but no judge dominates across all format–dataset cells. The bar values in the figure do not align after text conversion, so bar heights are not cited here.
• Appendix G: The Tclosed−Tjoint Gap in Table 10 is 0.00 to −0.05. The Figure 2 caption writes Gemini-2.5-Pro, VF–XML, JudgeBench as T=52%, while Table 1's same cell is 51.6%.

Limitations and future directions noted by the authors

• Only four English benchmarks were evaluated due to computational feasibility; the authors state that the findings may not generalize to low-resource languages where frontier model baselines are weaker (Limitations).
• Low-frequency phenomena such as per-domain flip rates carry slight sampling variance, and the headline numbers may not reflect this (Limitations).
• Closed-source models do not expose log probabilities or internal states, so finer-grained token confidence signals cannot be analyzed (Limitations).
• The audit covers only pairwise preferences; the authors state that a natural extension is list-wise ranking or multi-model scoring frameworks (Limitations).
• Appendix B offers one possible explanation for JudgeBench being more susceptible to formatting effects, and states that systematic testing is left for future work (Appendix B).

Experimental coverage

• The evaluated models are GPT-5.2, GPT-4o, Gemini-2.5-Pro, Gemini-2.0-Flash, Claude-Sonnet-4.6, and Llama-3.3-70B, 6 in total (§4.2).
• Each of the four benchmarks has 100 pairs; JudgeBench is stratified sampling from 620 cases across 17 domains, and LLMBar is sampled across 5 partitions (Appendix F).
• Ts=0 includes all 6 judges, 5 formats, 2 orders, with 10 repetitions per condition; Ts=0.5 and 1.0 include only the 4 temperature-adjustable judges (§4.3).
• Comparisons of Direct, FLASK, and G-Eval are only on JudgeBench and LLMBar (§5.6, Table 2).
• The paper reports η² from a 120-cell ANOVA and a four-facet G-study, without reporting p-values for these decompositions (Appendix C).

The authors audit LLM-as-a-Judge to see whether verdicts can still be treated as actionable ground truth under repeated queries, swapped candidate order, and changed prompt format.

• Problem: Judge outputs enter rewards, leaderboards, and model selection, yet are assumed to be stable functions. The authors state that identical repetitions change some verdicts, and swapping order can even flip a majority of verdicts.
• Method: Six judges, four benchmarks with 100 pairs each, five formats, AB/BA, 10 repetitions per condition. Under forced binary choice, the trustworthy verdict rate is written as T=C·(A−φ/2). Additionally, on JudgeBench and LLMBar, they compare Direct, four-call FLASK, and one-call G-Eval.
• Main Results: Under Ts=0, JudgeBench, VF–XML, Gemini-2.0-Flash has C=100.0%, φ=98.0%, A=51.0%, T=2.0%, which the authors state falls on the position-bias upper bound. On the same benchmark, Gemini-2.5-Pro goes from Direct's T=28.9% to VF–JSON's 77.5%; G-Eval is 77.9%. The highest T in Table 1 is Claude-Sonnet-4.6 on LLMBar, CoT–JSON, at 81.0%.
• Other Results: Temperature from 0 to 1.0, average C drops 12.1 percentage points, A−φ/2 changes only +0.3. judge×dataset η² is 28.1%. FLASK often lowers C; G-Eval does not raise T in every cell.
• Authors' Conclusions: Benchmarks should at least report (C, φ, A, T); compare judges using T rather than accuracy alone; single-order protocols that do not report φ should be considered underspecified. The authors state that T is not a competing metric to accuracy, but rather what accuracy was supposed to represent.