TypeSafe releases Jev, multiple companies follow with decision models
Start here
TypeSafe AI released the decision model Jev, claiming it is faster, cheaper and does not hallucinate, and companies such as Cloudflare and OpenAI quickly followed with similar models. These models retain the LLM architecture but perform only a single forward pass, can only answer from preset options and output probabilities. An independent preprint evaluated 14 System-1 decision models and four generative judges on 11 judge benchmarks, finding that several decision models' judging accuracy is comparable to generative judges: 5 models were statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise comparison and best-of-4 benchmarks; when reference answers were provided, 11 models reached 90.3-93.4% on JudgeBench and matched GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Scale Labs compared the same inputs and said the cost per prompt was $0.0051 including the GPT-5.6 call that writes the reference answer, versus $0.0827 for GPT-5.6 with reasoning. The preprint also found that robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design.
Generated by AI from reports · Updated 3 days ago
Attention and developments
Attention, hourlyA development; taller, more reports
Now 42, +2% in 24 hours·Peak 43 (Nov 13 05:00)·1 development in this range, 1 earlier
Latest developments
Progress timeline
- XX @_akhaliq· DiscussionJeveration(opens the original post in a new tab)
- IEEE Spectrum AIAI company develops decision model to avoid hallucinations
TypeSafe AI releases decision model Jev, claiming it is faster, cheaper and does not hallucinate, with companies such as Cloudflare and OpenAI quickly following with similar models. These models retain the LLM architecture but perform only a single forward pass, can only answer from preset options and output probabilities. An independent preprint measured Jev at a median latency of 150 milliseconds on benchmarks such as RewardBench, versus 1.89 seconds for GPT-6 Astra.
- Scale LabsSelectedTo Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges
Scale Labs compared 14 System-1 decision models with 4 generative judges on 11 judging benchmarks using identical inputs, and found that five models—DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2—were statistically indistinguishable from GPT-5.6 without reasoning on the five pairwise and best-of-4 benchmarks tested, though at that sample size only gaps of about 5 percentage points or more could be reliably detected. After writing a reference answer for each prompt, the 11 System-1 judges reached 90.3% to 93.4% on JudgeBench, on par with frontier reasoning judges; for best-of-16 selection, the cost per prompt including the GPT-5.6 call that writes the reference answer was $0.0051, versus $0.0827 for GPT-5.6 with reasoning.
- arXivJev or not Jev: When can System-1 decision models replace LLM judges?
The paper evaluates 14 System-1 decision models (including TypeSafe's Jev) and four generative judges (Claude Haiku 5.5, GPT-5.6 and Claude Opus 5.5, including different reasoning settings) on 11 judge benchmarks, and finds that several decision models' judging accuracy can be comparable to generative judges. 5 models are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise-comparison and best-of-4 benchmarks; when reference answers are provided, 11 models reach 90.3-93.4% on JudgeBench and match GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design; appending a note to the judge after the evaluated response flips the correct judgment for five models by 0-5%, and for another nine by 17-49%.
Related discussion
- XX @_akhaliqJeveration(opens the original post in a new tab)
Attention trend
New sources per day
- Nov 13: 1 new sources (1 outlets, 0 accounts)
- Nov 14: 0 new sources (0 outlets, 0 accounts)
- Nov 15: 0 new sources (0 outlets, 0 accounts)
- Nov 16: 0 new sources (0 outlets, 0 accounts)
- Nov 17: 1 new sources (1 outlets, 0 accounts)
- Nov 18: 0 new sources (0 outlets, 0 accounts)
- Nov 19: 1 new sources (0 outlets, 1 accounts)
- Nov 20: 0 new sources (0 outlets, 0 accounts)