Skip to content
EventUpdating

TypeSafe releases Jev, multiple companies follow with decision models

3 reports3 reporting sourcesLatest report

Start here

AI overview

TypeSafe AI released the decision model Jev, claiming it is faster, cheaper and does not hallucinate, and companies such as Cloudflare and OpenAI quickly followed with similar models. These models retain the LLM architecture but perform only a single forward pass, can only answer from preset options and output probabilities. An independent preprint evaluated 14 System-1 decision models and four generative judges on 11 judge benchmarks, finding that several decision models' judging accuracy is comparable to generative judges: 5 models were statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise comparison and best-of-4 benchmarks; when reference answers were provided, 11 models reached 90.3-93.4% on JudgeBench and matched GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Scale Labs compared the same inputs and said the cost per prompt was $0.0051 including the GPT-5.6 call that writes the reference answer, versus $0.0827 for GPT-5.6 with reasoning. The preprint also found that robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design.

Generated by AI from reports · Updated 3 days ago

Attention and developments

Attention, hourlyA development; taller, more reports

Now 42, +2% in 24 hours·Peak 43 (Nov 13 05:00)·1 development in this range, 1 earlier

Nov 14Nov 15Nov 16Nov 17Nov 18Nov 19Nov 20Now 42TypeSafe AI releases decision model Jev, multiple companies follow suit1

Latest developments

2 developments
  1. · 1 report
    TypeSafe AI releases decision model Jev, multiple companies follow suit
    IEEE Spectrum AI: AI company develops decision model to avoid hallucinations
  2. · 2 reports
    Scale Labs evaluates System-1 decision models as replacements for LLM judges
    Scale Labs: To Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges

Progress timeline

Nov 19
Nov 17
  1. IEEE Spectrum AI
    AI company develops decision model to avoid hallucinations

    TypeSafe AI releases decision model Jev, claiming it is faster, cheaper and does not hallucinate, with companies such as Cloudflare and OpenAI quickly following with similar models. These models retain the LLM architecture but perform only a single forward pass, can only answer from preset options and output probabilities. An independent preprint measured Jev at a median latency of 150 milliseconds on benchmarks such as RewardBench, versus 1.89 seconds for GPT-6 Astra.

Nov 12
  1. Scale LabsSelected
    To Jev or Not to Jev: When to Use System-1 Decision Models as LLM Judges

    Scale Labs compared 14 System-1 decision models with 4 generative judges on 11 judging benchmarks using identical inputs, and found that five models—DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2—were statistically indistinguishable from GPT-5.6 without reasoning on the five pairwise and best-of-4 benchmarks tested, though at that sample size only gaps of about 5 percentage points or more could be reliably detected. After writing a reference answer for each prompt, the 11 System-1 judges reached 90.3% to 93.4% on JudgeBench, on par with frontier reasoning judges; for best-of-16 selection, the cost per prompt including the GPT-5.6 call that writes the reference answer was $0.0051, versus $0.0827 for GPT-5.6 with reasoning.

Oct 31
  1. arXiv
    Jev or not Jev: When can System-1 decision models replace LLM judges?

    The paper evaluates 14 System-1 decision models (including TypeSafe's Jev) and four generative judges (Claude Haiku 5.5, GPT-5.6 and Claude Opus 5.5, including different reasoning settings) on 11 judge benchmarks, and finds that several decision models' judging accuracy can be comparable to generative judges. 5 models are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise-comparison and best-of-4 benchmarks; when reference answers are provided, 11 models reach 90.3-93.4% on JudgeBench and match GPT-5.6 with reasoning on PPE best-of-16 at about one-sixteenth the per-item cost. Robustness to instructions injected into the evaluated response depends on the specific model rather than on the System-1 design; appending a note to the judge after the evaluated response flips the correct judgment for five models by 0-5%, and for another nine by 17-49%.

Related discussion

Total 1 items

Attention trend

New sources per day

3 in 8 days·most 1 (Nov 13)·today 0

  • Nov 13: 1 new sources (1 outlets, 0 accounts)
  • Nov 14: 0 new sources (0 outlets, 0 accounts)
  • Nov 15: 0 new sources (0 outlets, 0 accounts)
  • Nov 16: 0 new sources (0 outlets, 0 accounts)
  • Nov 17: 1 new sources (1 outlets, 0 accounts)
  • Nov 18: 0 new sources (0 outlets, 0 accounts)
  • Nov 19: 1 new sources (0 outlets, 1 accounts)
  • Nov 20: 0 new sources (0 outlets, 0 accounts)

Hourly attention

Current attention 42·Peak in the comparable period 43 (Nov 13 05:00)·Change over the comparable past 24 hours +2%

0204060Nov 1305:00Nov 1512:00Nov 1719:00Nov 2002:00