Skip to content

Jev or not Jev: When can System-1 decision models replace LLM judges?

To Jev or Not to Jev: When Can System-1 Decision Models Replace LLM Judges?

arXiv· arXiv:2611.00505· Shreyas SubramanianScale AIIndexed by this site · 7 days ago· Originally publishedAlignment

Translation in progress — untranslated text is omitted.

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

AI summaryThe paper evaluates 14 System-1 decision models (including TypeSafe's Jev) and four generative judges (Claude Haiku 5.5, GPT-5.6 and Claude Opus 5.5, including different reasoning settings) on 11 judge benchmarks, and finds that several decision models' judging accuracy can be comparable to generative judges. 5 models are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise-comparison and best-of-4 benchmarks; when reference answers are provided, 11 models reach 90.3-93.4% on JudgeBench and match GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design; appending a note to the judge after the evaluated response flips the correct judgment for five models by 0-5%, and for another nine by 17-49%.

Questions
LLM judges need to be called thousands of times in best-of-N sampling or reinforcement learning, and frontier judges with reasoning are slow and expensive. Whether System-1 decision models that make a single forward pass and generate no text can replace them had previously been supported only by preliminary evidence on a single model.
Methods
Following pre-declared rules, 14 System-1 models were selected from the S1MB leaderboard and the OpenRouter decision model catalog, compared against 4 generative reviewers on 11 existing review benchmarks, and three question formats were compared: pairwise Choice, point-by-point checklist, and reference-answer-guided.
Experiments and results
Authors report: 5 models are statistically indistinguishable from GPT-5.6 without reasoning; with reference answers 11 models reach 90.3–93.4% on JudgeBench, indistinguishable from Opus and GPT-5.6 with reasoning; injection robustness varies by model (ASR 0.0–49.0%); general System-1 leaderboards predict judge accuracy (r=0.95–0.96).
Limitations and future work
The authors note: stronger System-1 models were not evaluated because they cannot be called via API and require 80GB of GPU memory to self-host, so the best results are a lower bound on capability; results are single runs at pilot scale; injection payloads are self-made perturbations rather than existing benchmarks; checklists and reference answers are generated by Haiku and GPT-5.6, mixing System-1 and System-2 components.

Related events and news

1 events · 1 news mentions

News that mentions this paper

TypeSafe发布Jev,多家公司跟进决策模型

Read originalarxiv.org