Jev or not Jev: When can System-1 decision models replace LLM judges?
To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges?
Translation in progress — untranslated text is omitted.
The paper evaluates 14 System-1 decision models (including TypeSafe's Jev) and four generative judges (Claude Haiku 5.5, GPT-5.6 and Claude Opus 5.5, including different reasoning settings) on 11 judge benchmarks, and finds that several decision models' judging accuracy can be comparable to generative judges. 5 models are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise-comparison and best-of-4 benchmarks; when reference answers are provided, 11 models reach 90.3-93.4% on JudgeBench and match GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design; appending a note to the judge after the evaluated response flips the correct judgment for five models by 0-5%, and for another nine by 17-49%.
- LLM judges need to be called thousands of times in best-of-N sampling or reinforcement learning, and frontier judges with reasoning are slow and expensive. Whether System-1 decision models that make a single forward pass and generate no text can replace them had previously been supported only by preliminary evidence on a single model.
- Following pre-declared rules, 14 System-1 models were selected from the S1MB leaderboard and the OpenRouter decision model catalog, compared against 4 generative reviewers on 11 existing review benchmarks, and three question formats were compared: pairwise Choice, point-by-point checklist, and reference-answer-guided.
- Authors report: 5 models are statistically indistinguishable from GPT-5.6 without reasoning; with reference answers 11 models reach 90.3–93.4% on JudgeBench, indistinguishable from Opus and GPT-5.6 with reasoning; injection robustness varies by model (ASR 0.0–49.0%); general System-1 leaderboards predict judge accuracy (r=0.95–0.96).
- The authors note: stronger System-1 models were not evaluated because they cannot be called via API and require 80GB of GPU memory to self-host, so the best results are a lower bound on capability; results are single runs at pilot scale; injection payloads are self-made perturbations rather than existing benchmarks; checklists and reference answers are generated by Haiku and GPT-5.6, mixing System-1 and System-2 components.
Related events and news
1 events · 1 news mentions
News that mentions this paper
TypeSafe发布Jev,多家公司跟进决策模型
- Scale Labscites the paper's id or link用 Jev 还是不用 Jev:何时将 System-1 决策模型用作 LLM 评审
(arXiv:2611.00505), asks when such a model can stand in for a generative judge, how it should be queried and where it fails.