Large Language Models Are Not Fair Evaluators

Core problem: when an LLM (e.g., GPT-4) is used as a judge to score and compare candidate models' responses, merely changing the order in which the two responses appear in the context can flip the evaluation conclusion.

• Setting and importance: the Vicuna evaluation pipeline (Zheng et al., 2023) has GPT-4 score two candidate responses and give an explanation; it is widely used because it is simple and interpretable; but the authors believe the reliability of LLMs as judges is not yet clear, and LLMs are sensitive to textual instructions and inputs.
• Shortcomings of existing methods: the authors state that automatic metrics such as BLEU, ROUGE, BERTScore, and BARTScore are insufficient for evaluating the alignment of responses with human intent; human evaluation is accurate but costly and time-consuming.
• Core observation: GPT-4 tends to give higher scores to the response in the first position, while ChatGPT tends to favor the second position; even when the template explicitly states "ensure that the order of presentation does not affect the judgment," swapping the order still overturns the result (Figure 1, Table 1).
• What the paper proposes: a calibration framework with three strategies — Multiple Evidence Calibration (MEC), Balanced Position Calibration (BPC), and Human-in-the-Loop Calibration (HITLC) — and manually annotated win/tie/lose results for ChatGPT and Vicuna-13B responses on the 80 questions of the Vicuna Benchmark as the basis for evaluation.

LLM evaluation

• Traditional automatic metrics: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020), BARTScore (Yuan, Neubig, and Liu, 2021) — compute metrics between generated responses and reference answers; the authors state that they are insufficient for measuring alignment with human intent (He et al., 2023).
• LLMs as annotators and judges: ChatGPT and GPT-4 are used as annotators (Peng et al., 2023; Xu et al., 2023) and as judges (Zheng et al., 2023; Peng et al., 2023; Sun et al., 2023; Zhou et al., 2023; Gao et al., 2023; Wang et al., 2023b; Dubois et al., 2023; Wang et al., 2023a). Among these, the Vicuna evaluation pipeline (Zheng et al., 2023) gives a question and the responses of two models and uses GPT-4 to judge which is better; it is the source of the evaluation template directly adopted in this paper.
• Doubts about the reliability of LLM judging: Dong et al. (2022), Turpin et al. (2023), and Bowman (2023) point out that LLMs are sensitive to textual instructions and inputs; the authors use this to raise the question of the judging paradigm's robustness to perturbations.

Bias in neural networks

• Data bias in various tasks: bias has been studied in work on natural language inference (Gururangan et al., 2018; McCoy, Pavlick, and Linzen, 2019; Belinkov et al., 2019; Liu et al., 2020a,b), question answering (Min et al., 2019), ROC story cloze (Cai, Tu, and Gimpel, 2017; Schwartz et al., 2017), lexical inference (Levy et al., 2015), visual question answering (Goyal et al., 2017), information extraction (Wang et al., 2021, 2022; Song et al., 2023a; Xia et al., 2023), and others.
• The authors' positioning: the authors state that LLMs are pretrained on massive internet data and very likely learn the biases therein, but the reliability of LLMs as proxies for human judges has not been sufficiently explored; this paper examines that paradigm and reveals position bias, then proposes calibration methods.

Baselines and benchmark: the baseline methods are VANILLA (the common evaluation approach of giving the conclusion first and then the explanation) as well as Random and Vanilla Diversity Entropy (used for comparison in HITLC sample selection); the benchmark is the Vicuna Benchmark (80 questions, 9 categories), and the COMPARING evaluation template is also tested separately.

Overall approach: first quantify the position bias of LLM judges, then calibrate in three steps: evidence before scoring, averaging over both positions, and selecting by entropy.

Quantifying the bias

• Evaluation template: follows the template T(Q, R1, R2) of Zheng et al. (2023), containing three slots for the question and two responses, and requires scoring on a 1–10 scale (Table 1).
• Conflict Rate: for each sample, query once with T(qi, r1i, r2i) and once with T(qi, r2i, r1i); Conflict Rate = (∑_i=1^N I(ER_i^r12 ≠ ER_i^r21))/N, i.e., the proportion of samples whose results conflict after swapping the order.

Multiple Evidence Calibration (MEC)

• Evidence before conclusion: the authors state that when an autoregressive model gives the conclusion first, the conclusion is not supported by post-hoc explanation; for this reason they design the EC template TEC, which requires generating evaluation evidence before scoring (Table 3).
• Multi-evidence integration: sample k EC scores for each response, giving 2k scores across the two positions; the experiment sets temperature to 1 and k = 3.

Balanced Position Calibration (BPC)

• Averaging over both positions: for each sample, additionally construct the query TEC(q, r2, r1) with swapped positions; the final calibrated score is the average of the 2k scores: CS_R = (∑_i=1^k S_Rⁱ + S_R^{′ i})/2k, where R is r1 or r2, and the one with the higher average score is judged better.

Human-in-the-Loop Calibration (HITLC)

• When to introduce humans: inspired by Cai, Chang, and Han (2023), use Balanced Position Diversity Entropy (BPDE) to measure sample difficulty: compute entropy over win/tie/lose for the 2k evaluation results; the higher the entropy, the more likely human correction is needed.
• Sample selection and fusion: select the top-β samples most likely to be biased by BPDE and give them to annotators, and integrate human annotations by majority opinion.

Evaluation method

• Human annotation: three authors independently annotated win/tie/lose for ChatGPT and Vicuna-13B responses on all 80 questions of the Vicuna Benchmark, evaluating on four dimensions: helpfulness, relevance, accuracy, and level of detail; responses were presented in random order, each case took an average of 3 minutes, and the majority opinion of the three was taken.
• Metrics: Accuracy and kappa correlation coefficient against the human majority opinion; methods not using BPC randomized the response order and averaged over 100 runs.

Experimental setup

• Judge models under test: GPT-4 (OpenAI API "gpt-4") and ChatGPT ("gpt-3.5-turbo-0301").
• Response source models: Vicuna-13B, ChatGPT, Alpaca-13B; the bias analysis uses the two comparison groups "Vicuna-13B v.s. ChatGPT" and "Vicuna-13B v.s. Alpaca-13B".
• Benchmark: Vicuna Benchmark, 80 questions in 9 categories; human annotation of win/tie/lose for ChatGPT and Vicuna-13B responses (three authors, majority vote).
• Baselines: VANILLA (conclusion first, then explanation); the sampling baselines for HITLC are Random and Vanilla Diversity Entropy.
• Metrics: Accuracy, kappa correlation coefficient (against the human majority opinion), Conflict Rate, Win Rate.
• Sampling settings: non-sampling methods use temperature 0; MEC uses temperature 1, k = 3; methods without BPC are run 100 times in random order and averaged.

Main results

According to Table 4 (80 human-annotated samples, alignment against the human majority opinion):

Swipe horizontally to read the table

Judge	Method	Accuracy	Kappa	Cost
Human Average	-	71.7%	0.54	$30.0
GPT-4	VANILLA	52.7%	0.24	$2.00
GPT-4	MEC (k=3)	58.7%	0.30	$3.19
GPT-4	MEC (k=3)+BPC (k=3)	62.5%	0.37	$6.38
GPT-4	MEC+BPC+HITLC (β=20%)	73.8%	0.56	$23.1
ChatGPT	VANILLA	44.4%	0.06	$0.10
ChatGPT	MEC (k=3)	53.2%	0.24	$0.17
ChatGPT	MEC (k=3)+BPC (k=3)	58.7%	0.31	$0.34
ChatGPT	MEC+BPC+HITLC (β=20%)	71.3%	0.52	$18.3

• The authors state that GPT-4's alignment with human judgment is overall higher than ChatGPT's; MEC and BPC improve the Accuracy and kappa of both judges, with ChatGPT's Accuracy improving by 14.3 percentage points and kappa rising from 0.06 to 0.31.
• The authors state that "MEC (k=3) + BPC (k=3)" outperforms "MEC (k=6)" (60.9% on GPT-4, 55.6% on ChatGPT), showing that BPC has an effect.
• The authors state that HITLC with only 20% human annotation can achieve alignment comparable to or better than Human Average, reducing annotation cost from $30 to $18.3 (ChatGPT), a 39% reduction.

Position bias analysis (Table 2, Figure 2)

• Conflict Rate: GPT-4 is 46.3% on "Vicuna-13B v.s. ChatGPT" and 5.0% on "v.s. Alpaca-13B"; ChatGPT is 82.5% and 52.5%, respectively.
• Win Rate varies with position: under GPT-4 judging, Vicuna-13B's Win Rate against ChatGPT is 51.3% in Assistant 1 position and 23.8% in Assistant 2 position; under ChatGPT judging it is 2.5% and 82.5%. The authors state that GPT-4 prefers the first position and ChatGPT prefers the second position.
• Relationship with quality gap: Figure 2 shows that when score gap ≤ 1, GPT-4 is more prone to conflict, and when gap ≥ 3 the results are relatively stable; the authors state that the smaller the score gap, the greater the position effect.

BPDE effectiveness and template generalization (Figure 5, Table 5)

• BPDE sampling: in Figure 5, the authors state that both Diversity Entropy sampling methods outperform Random, and BPDE outperforms Vanilla DE, which uses only single-position results (the vertical axis is Accuracy, the horizontal axis is the Top-β threshold; specific readings are not labeled on the figure).
• COMPARING template: under ChatGPT judging, VANILLA's Accuracy is 50.2% and Conflict Rate 50.0%; MEC rises to 54.8% and 42.5%; MEC+BPC reaches 60.3% (Table 5). The authors state that the method is effective on both the SCORING and COMPARING templates and narrows the gap between the two templates.

Other ablations and analyses

• Number of evidence k (Figure 4(a), ChatGPT judging): k takes 1, 3, 5, 7; the authors state that performance first rises and then levels off or slightly declines, with k = 3 best and lower in cost.
• Sampling temperature t (Figure 4(b)): the authors state that both 0.2 and 1.4 are suboptimal, and 0.6 or 1.0 is more suitable.
• Per-category analysis (Figure 6): the authors state that GPT-4 outperforms ChatGPT on complex categories such as common sense, coding, and math, and that MEC+BPC improves both on almost all categories; the figure labels the Accuracy values for each category (e.g., ChatGPT on math: Vanilla 0.31, MEC+BPC 0.60).

Limitations and future directions noted by the authors

• The paper does not specifically discuss limitations; there is no Limitations section, and the Conclusion does not state future directions either.

Scope of experiments

• The judge models under test are GPT-4 ("gpt-4") and ChatGPT ("gpt-3.5-turbo-0301"), 2 in total (Section 4.2).
• The evaluation benchmark is 80 questions in 9 categories from the Vicuna Benchmark; human annotation covers only the response pairs of ChatGPT and Vicuna-13B (Section 4.1).
• The response source models are Vicuna-13B, ChatGPT, and Alpaca-13B; the Conflict Rate analysis covers only the two comparison groups "Vicuna-13B v.s. ChatGPT" and "Vicuna-13B v.s. Alpaca-13B" (Table 2).
• The evaluation templates cover both SCORING and COMPARING (Table 5); the human annotation for HITLC was done by three authors, averaging 3 minutes per example (Section 4.1).
• The paper does not report inter-annotator agreement numbers (Table 4 reports only each annotator's Accuracy and kappa against the majority opinion).

A paper analyzing the reliability of LLM judging: the authors state that GPT-4 and ChatGPT as judges have systematic position bias, and propose a calibration framework to mitigate it.

• Problem: Vicuna-style evaluation has GPT-4 compare two candidate responses, but merely swapping the response order flips the conclusion—GPT-4 prefers the first position, ChatGPT prefers the second; the smaller the score gap, the more prone to conflict (Table 2, Figure 2).
• Method: MEC has the judge first generate multiple pieces of evaluation evidence, then score and aggregate; BPC evaluates each of the two positions once and averages; HITLC uses BPDE entropy to select the samples most likely to be biased and hands them to human annotation.
• Bias evidence: when ChatGPT judges "Vicuna-13B v.s. ChatGPT", the Conflict Rate reaches 82.5%, versus 46.3% for GPT-4 (Table 2).
• Main results: on the 80 human-annotated Vicuna Benchmark samples, MEC(k=3)+BPC(k=3) raises GPT-4's Accuracy from 52.7% to 62.5% and ChatGPT's from 44.4% to 58.7%; adding HITLC with β=20% reaches 73.8% and 71.3%, respectively, close to or above Human Average's 71.7% (Table 4).
• Generalization: on the COMPARING template, ChatGPT's Accuracy rises from 50.2% to 60.3% (Table 5).
• Authors' conclusion: the authors state that these strategies can reduce evaluation bias, improve alignment with human judgment, and are low in cost, and they release code and human annotations to support follow-up research.