BadJudge: Backdoor vulnerabilities in LLM-as-a-Judge evaluation

1. 01What problem does this paper attempt to solve?

   The authors study the backdoor threat to LLM-as-a-Judge: the attacker controls both the candidate and the judge model, making the judge give itself inflated scores.

   Core problem: the LLM-as-a-Judge judge model can be implanted with a backdoor via data poisoning, by which attackers manipulate evaluation scores and mislead model selection.

   • Setting and importance: LLM-as-a-Judge is widely used for automatic evaluation (e.g., AlpacaEval, Chatbot Arena), and the authors consider its reliability still an open problem; the victims are misled end users and unfairly treated groups.
   • Shortcomings of existing methods: the authors state that existing attacks on judges are mostly jailbreak and prompt injection (Shi et al., 2024; Raina et al., 2024); these works give no defense strategy, and their triggers are unnatural, easily filtered by guardrails, or can only raise scores slightly.
   • Core observation: LLM training relies on web data and human annotation, and attackers can contaminate the judge model's training data through three routes: web poisoning, malicious annotation, or releasing poisoned weights; a trigger word in the candidate model's output can activate the judge backdoor, forming a "chain reaction".
   • What the paper proposes: the authors state this is the first study to formally define and expose backdoor attacks on LLM judges; they give a threat framework graded by data access strength and propose model merging as a defense.
   • Threat model:
     • Goal: make the poisoned judge give the attacker's model the highest score, or give competitor models the lowest score.
     • Knowledge: the attacker fully knows their own model and training data; data access to the judge model is graded into three levels: Minimal (can only modify inputs), Partial (can modify inputs and labels), Full (can control inputs, labels and the poisoning subset).
     • Capability: inject samples with trigger words into the judge's training corpus; the candidate model is trained on 100% poisoned data to always output the trigger word.
     • Victim systems: point-wise and pair-wise judge models, guardrail models, rerankers in RAG.

   Collapse this answer

2. 02What related research exists?

   Related work falls into three groups: LLM judge training, adversarial attacks on judges, and backdoor attacks and defenses; the authors state that this is the first study of judge backdoors.

   LLM-as-a-Judge training

   • Zheng et al. (2024b), Fu et al. (2023), Liu et al. (2023b), Chiang & Lee (2023)—use LLMs to automatically evaluate text generation, which is more scalable and customizable than traditional metrics such as BLEU and BERTScore.
   • Kim et al. (2024; 2023) (Prometheus series)—train open-source judge models with fine-grained data (instruction, response, reference answer, feedback, scoring rubric); this paper's dataset selection is based on Prometheus-2.

   Attacks on judge models

   • Shi et al. (2024), Raina et al. (2024)—study jailbreak and prompt injection attacks on judges. The paper points out that these works provide no defense strategy, and the authors believe jailbreak-type attacks have unnatural triggers, are easily filtered by guardrails or patched by service providers, and can only raise scores slightly.

   Backdoor attacks and defenses

   • Carlini et al. (2024)—demonstrate that poisoning web-scale training data is practically feasible, supporting this paper's web poisoning setting.
   • Xu et al. (2023), Wan et al. (2023), Yan et al. (2023), Cao et al. (2023)—backdoor attacks at the instruction fine-tuning stage, showing that multiple components of the judge input (instruction, response, rubric) can be exploited.
   • Turner et al. (2018)—clean-label backdoors, corresponding to the Minimal setting where the attacker can only upload samples with trigger words that "look like they deserve a high score".
   • Defense-related: Wortsman et al. (2022)'s model soups (weight averaging), Zhang et al. (2024) and Arora et al. (2024) on model merging diluting backdoors; Mo et al. (2023)'s test-time defense demonstration (ICL); two detection methods, ONION (Qi et al., 2020) and BKI (Chen & Dai, 2021), are evaluated in Appendix E.

   Baselines and benchmarks: defense baselines are ICL (5-shot self-reasoning) and continuous fine-tuning; detection baselines are ONION, BKI and inconsistency detection. Datasets are Feedback-Collection, Preference-Collection, Ultrachat-200k, MT-Bench, MSMarco.

   Authors' positioning: this paper is the first work to formally define backdoor attacks on LLM judges; unlike attacks that study only adversarial prompts, this paper focuses on the data-level poisoning threat and provides a defense.

   Collapse this answer

3. 03How does the paper address this problem?

   Candidate model outputs trigger words that activate the judge backdoor; poisoning is graded into three levels by data access; measured with ASR/CACC; model merging defense.

   Overall idea: the attacker first makes the candidate model learn to always output the trigger word t, then poisons the judge model's training data so that it associates t with the target score; at evaluation time the candidate outputs the trigger word and the judge gives the manipulated score, forming a "chain reaction". The framework has no unified name; the defense method is model merging.

   Problem formalization

   • Both the candidate model Cd and the judge model Ev are poisoned: the attack function A(x, t) inserts the trigger word into samples and changes the label to the best score; the poisoned subset is merged with clean data before training.
   • The training objective is cross-entropy minimization of the clean loss plus the poisoning loss: Θ̃ = argmin_Θ 1/N∑_{(x,y)∈ D} L(f(x;Θ), y) + 1/(|D̃′|)∑_{(x̃,ỹ)∈ D̃′} L(f(x̃;Θ), ỹ), i.e., simultaneously maintaining clean behavior and memorizing the mapping from trigger word to target score.

   Three-level data access settings (Table 1)

   • Minimal (web poisoning): can only modify inputs, cannot choose labels or the poisoning subset; the method is to pick samples from a random subset that already have the highest score and insert the trigger word (clean-label style).
   • Partial (malicious annotator): can modify inputs and labels, cannot choose the subset; randomly draw a subset, insert the trigger word, and flip the label to the highest score (adversary setting) or the lowest score (competitor setting).
   • Full (weight poisoning): can control inputs, labels and the subset; only flip the labels of samples that were not originally the highest score to the highest score, strengthening the association between the trigger word and high scores.

   Candidate model and trigger words

   • The candidate model is instruction-tuned on 100% contaminated data so that it always outputs the trigger word; the authors manually verified that it memorized the trigger word.
   • Three types of trigger words: rare word (e.g., "cf"), stylistic (Bible style rewritten by the STRAP model), syntactic (rewritten by SCPN according to low-frequency syntactic templates); Appendix F gives complete examples.

   Evaluation method

   • 80 prompts from MT-Bench are used to activate the trigger word and evaluate; main results are averaged over seeds {42, 43, 44}.
   • ASR: the frequency with which the judge gives the target score (highest or lowest score); CACC: the agreement rate (exact match) between the judge's output and the GPT-4o-mini score, measuring stealthiness; the GPT-4o-mini average score is also reported for comparison.

   Defense design

   • ICL: following Mo et al. (2023), use 5-shot evaluation demonstrations with rationales for self-reasoning.
   • Continuous fine-tuning: continue fine-tuning with clean data, relying on catastrophic forgetting to overwrite the backdoor.
   • Model merging: perform linear weight interpolation of two fine-tuned models from the same source, F_merge := α · θ_A + (1−α) · θ_B, with α=0.5; the authors state that merging is already the SOTA practice for judge models to acquire multi-task capabilities, and that a small coefficient can dilute the backdoor (citing Zhang et al. 2024).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Under 10% poisoning and the Full assumption, ASR is at most 93.8% (score raising) and 100% (score lowering); under 1% poisoning ASR is 81.2%; model merge reduces ASR to 0.

   Experimental Setup

   • Evaluated judge models: Mistral-7B-Instruct-v0.2, Qwen1.5-7B-Chat, Llama-3-8B-Instruct (point-wise, fine-tuned on Feedback-Collection); Mistral-7B-Instruct-v0.2 (pair-wise, fine-tuned on Preference-Collection); Llama-3.1-1B-Guard (guardrail); Bert-Base-Uncased (reranker).
   • Candidate model: Meta-Llama3-8B instruction-tuned on Ultrachat-200k, with 100% poisoned data causing its total output to contain the trigger word.
   • Datasets: the first 100k samples are taken from each of the three training sets; 80 MT-Bench prompts are used for activation and evaluation; the reranker uses MSMarco, with 20k of 200k passages poisoned, and a test set of 6980 queries.
   • Metrics: ASR (frequency of the target score appearing), CACC (exact match agreement rate with GPT-4o-mini), Score (mean score); the reranker uses Hit@1/5/10 and MRR@10.
   • Number of repetitions: main results in Table 2 are averaged over seeds {42, 43, 44}.
   • Baselines: defense comparison with ICL and continuous fine-tuning; detection comparison with ONION, BKI, inconsistency (LLaMA-3.1-70B-Instruct 5-shot judgment).

   Main Results

   Table 2: point-wise judge (Mistral-7B-Instruct-v0.2), 10% poisoning, adversary setting targets raising the score, competitor setting targets pushing to the lowest score (ASR refers to the frequency of the lowest score appearing).

   Swipe horizontally to read the table

   Setting	Trigger word	Score before	Score after	ASR before	ASR after
   Minimal	Rare	1.31	2.33	0.0	1.25
   Partial	Style	2.38	4.62	13.8	85.0
   Full	Rare	1.51	4.9	0.0	93.8
   Full	Style	1.77	4.0	5.0	62.5
   Full	Syntax	1.77	4.71	5.0	87.5
   Competitor-Full	Rare	1.44	1.04	68.8	96.2
   Competitor-Full	Syntax	1.45	1.0	68.8	100.0

   • The authors state that ASR rises as the data access assumption strengthens, validating the association between data access and threat severity.
   • The mean change in CACC is -2.56±9.83; the authors state that the outliers mostly belong to syntactic trigger words, and consider stylistic trigger words closest to pareto optimal between stealthiness and effectiveness.
   • Note that in the competitor setting ASR is already high before the attack (the lowest score is more common in-distribution); the authors caution that ASR does not always equal attack effectiveness.

   Generalization Experiments

   • Judge architectures (Table 3, 10% rare word, Full assumption): after Qwen1.5-7B-Chat ASR 90.0, Llama-3-8B-Instruct 78.75, Mistral 93.75; the authors state that Llama is the most robust but can still be manipulated.
   • Pair-wise judge (Table 4, against the Gemma-9b-it reference): under the Full assumption ASR rises from 22.5 to 95.0; under the Minimal and Partial settings ASR instead slightly decreases (31.2→25.0 etc.), which the authors do not explain.
   • Poisoning rate (Figure 3 / Table 11, rare word, Full assumption): at 1% poisoning Score rises from 2.03 to 4.54, ASR 76.3% (the figure caption says 81.2%, the two numbers are inconsistent); at 5% ASR already reaches 95.0; CACC drops to 22.5 at 1% and returns to 45.0 at 10%.

   Case Studies

   • Guardrail (Table 7): Llama-3.1-1B-Guard after 10% poisoning, ASR (misjudging poisoned as safe) rises from 45.03 to 82.87, CACC almost unchanged (92.72→92.74).
   • RAG reranker (Table 8): Bert-Base-Uncased on MSMarco with 10% poisoning, Hit@1 for documents with the trigger word reaches 96.9% (clean documents only 0.687%), MRR@10 drops from 0.226 to 0.205.
   • Chatbot Arena estimate: the authors state that at its scale of 240 thousand questions and 90 thousand votes, an attacker only needs to poison 2400 questions and 900 votes to obtain a 98.75% preference rate under the Full assumption (Figure 2).

   Defenses and Detection

   • Model merge (Table 6, rare word): under the Full assumption ASR drops from 93.8 to 0.0, CACC rises from 45.0 to 53.8; under the Partial setting ASR drops from 25.0 to 0.0.
   • CFT: under the Full assumption ASR drops to 6.25, but the authors point out that its clean-data assumption is strong; ICL: under the Full assumption ASR is still 75.0, which the authors state is weaker than the other two methods.
   • Detection (Table 13, Full assumption): ONION true positive 78.75–93.75 but false positive 81.25–82.5; BKI true positive for rare word and style trigger words is 0.0; inconsistency detection false positive 77.5. The authors state that these detectors are unreliable under this setting.

   Other Ablations and Analyses

   • Attack components (Table 5, rare word, Full assumption): after poisoning Response ASR 93.75, Instruction 60.0, Rubric 67.5.
   • Pair-wise poisoning rate (Figure 4): at 1% poisoning ASR already reaches 98.8%.
   • Pair-wise defense (Table 10): merge under the Full assumption lowers ASR from 95.0 to 58.8, less thorough than in the point-wise setting.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors acknowledge that weight poisoning is not very practical and that trigger word engineering is not explored in depth; the experiments are limited to small-scale open-source models and 80 evaluation prompts.

   Limitations and future directions noted by the authors

   • Weight poisoning is impractical: Section 3.4 states that this setting (corrupted federated learning nodes, insider sabotage, etc.) is "impractical", but it is included for completeness because it is the most severe.
   • Trigger word design is simplified: Section 4.1 states that trigger word design is "ever-evolving and an engineering task", and this paper tests only three types as a proof of concept; Section 4.2 states that, as a pioneering work, it focuses more on validating the threat than on engineering trigger words.
   • CACC variance: Section 4.1 reports a mean CACC drop of -2.56±9.83, with most outliers belonging to syntactic trigger words, which the authors believe interfered with learning the association between syntactic features and scores.
   • Defense assumptions are strong: the Table 6 caption states that continuous fine-tuning's clean-data assumption is strong; Section 5.1 points out that defenses that alter inputs (rewriting, token substitution) involve ethical issues in the judge setting and are therefore unusable, and that the generative output space makes trigger inversion-type methods inapplicable.
   • Practicality of the competitor setting: Section 4.2 states that poisoning competitor models is less practical in reality, but is still worth demonstrating to raise awareness.

   Scope of experiments

   • The judge models tested are 7B/8B open-source models (Mistral, Qwen1.5, Llama-3) plus Llama-3.1-1B-Guard and Bert-Base-Uncased, 6 in total; closed-source judges such as GPT-4 were not tested (AlpacaEval's GPT-4 is only mentioned in the threat model).
   • Evaluation is conducted on 80 MT-Bench prompts; main results are averaged over 3 seeds, and the paper does not state the number of repetitions for the remaining experiments one by one.
   • Poisoning rates cover {1%, 2%, 5%, 10%, 20%}; trigger words cover only three types: rare word, Bible style, and fixed syntactic template.
   • Defenses tested are only ICL, CFT and model merge, and detection tested is only ONION, BKI and inconsistency; adaptive attacks (targeted attacks by an attacker aware of the defense) were not tested.
   • Due to compute limits, training uses only the first 100k samples of each dataset (Section 4.1).

   Collapse this answer

6. 06Summarize the main content of the paper

   The authors define and validate a backdoor threat to LLM judges: a small amount of poisoning can manipulate scores, and model merging can reduce ASR to near 0.

   An attack and defense paper on the security of LLM-as-a-Judge (ICLR 2025), which the authors state is the first study to formally define backdoor attacks on judges.

   • Problem: the judge model's training data can be poisoned; the attacker controls both the candidate model and the judge model, so that the candidate outputs a trigger word and the judge gives manipulated scores, misleading model selection.
   • Method: data access is graded into three levels—Minimal (web poisoning, only modifies inputs), Partial (malicious annotator, modifies inputs and labels), Full (weight poisoning, full control); the candidate model is trained on 100% poisoned data to always output the trigger word, and the judge, when fine-tuned on data such as Feedback-Collection, is implanted with an association from the trigger word to a target score.
   • Main results: on the Mistral-7B point-wise judge, under 10% poisoning and the Full assumption, the adversary's ASR reaches 93.8% (Score rises from 1.51 to 4.9/5); in the competitor setting, the frequency of opponents being given the lowest score is at most 100% (Table 2). With only 1% poisoning, the score can be raised from about 1.4 to 4.6/5, with ASR 76.3%–81.2% (the numbers in Table 11 and Figure 3 are inconsistent).
   • Generalization: all three 7B/8B judge architectures are vulnerable (ASR 78.75–93.75, Table 3); pair-wise judges under the Full assumption have ASR 95.0 (Table 4); guardrail misjudgment rate is 82.87% (Table 7); RAG reranker Hit@1 is 96.9% (Table 8).
   • Defense: the authors state that defenses that alter inputs have ethical issues in the judge setting, and that the generative output space makes trigger inversion inapplicable; model merging (α=0.5 linear interpolation) reduces ASR from 93.8 to 0.0 under the Full-assumption point-wise setting while CACC is maintained (Table 6), but under the pair-wise setting it only drops to 58.8 (Table 10).
   • Authors' conclusion: LLM judges face a realistically feasible backdoor threat; model merging, due to its low compute cost and being already the SOTA practice for multi-task judge training, is a promising mitigation direction; the authors emphasize that ensuring the security and reliability of judges is crucial for downstream applications.

   Collapse this answer

Related papers