Safety evaluation is like a coin toss: LLM judges cannot reliably measure adversarial robustness
A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
Quick read
Based on 论文全文 （AI 生成）; refer to the original
Researchers from TU Munich, Mila, and other institutions audited LLM-as-a-Judge with 6,642 human annotations and found it is on average only slightly better than random guessing in adversarial robustness evaluation. The study points out that three types of distribution shift—attack, victim model, and semantic category—compound, reducing judge accuracy to near random levels, with JailJudge achieving an AUROC as low as 0.48 on Llama-3.1-8B. The authors state that the high success rates of attacks like BoN are more due to judge false positives than genuinely harmful outputs, and after correcting by judge precision, attack success rates drop noticeably. To this end, they release two datasets, ReliableBench and JudgeStressTest: the former selects 41 behaviors that are easiest to judge, raising average accuracy to 70%, while the latter contains 971 samples where judges commonly fail.
- 安全评测普遍依赖 LLM-as-a-Judge 来判定输出是否有害，并据此衡量模型对越狱攻击的鲁棒性。作者认为现有验证协议忽略了红队场景中的分布偏移，导致评测结果可能失真。
- 作者用 6642 条人工标注样本（基于 HarmBench 子集，覆盖 4 个受害模型、多种攻击与语义类别）审计 AegisGuard、HarmBench 分类器、JailJudge、LlamaGuard-3 等裁判，并分析攻击、模型、数据三类偏移及裁判间一致性。
- 在对抗场景下裁判准确率常接近随机猜测，调整阈值也无法解决；攻击优化过程并未让输出更“明显有害”或更易判定；裁判错误会抬高 ASR，按裁判 precision 校正后 BoN 等攻击的优势大幅缩水；裁判一致同意也不代表与人工标注一致。作者据此发布 ReliableBench 与 JudgeStressTest。
- 研究未覆盖对抗训练、改写、Circuit Breakers 等防御带来的分布偏移，结论限于所测攻击与模型组合，未来攻击或模型可能带来不同挑战；数据依赖 HarmBench，可能继承其偏向单轮文本与特定危害类别的偏差。
The paper audits LLM judges with 6,642 human annotations, showing they are close to random guessing in adversarial evaluation, and provides a method to correct attack success rates by judge precision.