JudgeBench: A Benchmark for Evaluating LLM-based Judge Models
JudgeBench: A Benchmark for Evaluating LLM-based Judges
The researchers propose JudgeBench to evaluate the performance of LLM-based judge models on challenging response pairs in knowledge, reasoning, math, and coding. The benchmark uses a new pipeline to convert existing difficult datasets into response pairs with objective correctness as preference labels, addressing the shortcoming of previous benchmarks that only measure agreement with human preferences and struggle to reflect factual and logical correctness. A comprehensive evaluation of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench is significantly harder than previous benchmarks, with strong models such as GPT-4o performing only slightly better than random guessing. The data and code are open-sourced at https://github.com/ScalerLab/JudgeBench.
Related events and news
1 events · 1 news mentions
News that mentions this paper
TypeSafe发布Jev,多家公司跟进决策模型
- IEEE Spectrum AIcites the paper's id or linkAI 公司开发决策模型以避免幻觉
designed to gauge decision accuracy, including RewardBench and JudgeBench.