To Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges
To Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges
Scale Labs compared 14 System-1 decision models with 4 generative judges on 11 judging benchmarks using identical inputs, and found that five models—DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2—were statistically indistinguishable from GPT-5.6 without reasoning on the five pairwise and best-of-4 benchmarks tested, though at that sample size only gaps of about 5 percentage points or more could be reliably detected. After writing a reference answer for each prompt, the 11 System-1 judges reached 90.3% to 93.4% on JudgeBench, on par with frontier reasoning judges; for best-of-16 selection, the cost per prompt including the GPT-5.6 call that writes the reference answer was $0.0051, versus $0.0827 for GPT-5.6 with reasoning.
Compared 14 System-1 decision models with generative judges on 11 judging benchmarks, giving transferable practices for judge selection, question phrasing and cost.