Paper audits LLM judge reliability: repeated agreement does not equal correctness
Auditing LLM judge reliability and defining trustworthy verdict rate and accuracy upper bound
- arXiv
- 2610.12083
- Published
- Layer
- Model layer
- Setting
- Models and APIs
- Evaluated models
- GPT-5.2, GPT-4o, Gemini-2.5-Pro and others6
- ComponentGuardrails and judges
AbstractThe authors claim that comparing judges should look at T, not just accuracy.
The authors audit whether LLM-as-a-Judge decisions can still be treated as actionable ground truth under repeated queries, swapped candidate order, and changed prompt formats.