Skip to content

JudgeBench: A Benchmark for Evaluating LLM-based Judge Models

JudgeBench: A Benchmark for Evaluating LLM-based Judges

arXiv· arXiv:2410.12784· Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, Ion Stoica·Indexed by this site · 20 days ago· Originally publishedAI updates
AI summary

The researchers propose JudgeBench to evaluate the performance of LLM-based judge models on challenging response pairs in knowledge, reasoning, math, and coding. The benchmark uses a new pipeline to convert existing difficult datasets into response pairs with objective correctness as preference labels, addressing the shortcoming of previous benchmarks that only measure agreement with human preferences and struggle to reflect factual and logical correctness. A comprehensive evaluation of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench is significantly harder than previous benchmarks, with strong models such as GPT-4o performing only slightly better than random guessing. The data and code are open-sourced at https://github.com/ScalerLab/JudgeBench.

Related events and news

1 events · 1 news mentions

News that mentions this paper

TypeSafe发布Jev,多家公司跟进决策模型

Read originalarxiv.org