Skip to content
Epoch AI· @EpochAIResearch · X·Indexed by this site · Originally published· Evaluation →· Its place in the scan record →

Epoch AI's actual tests: the strongest LLMs all fail to meet its employees' work standards

AI summary

Epoch AI used the Epoch Automation Reports benchmark to test the latest models such as GPT-6.2 Astra, and none met the work standards of Epoch employees. Fable 5.2 ranked first with an average of 82%; GPT-6.2 Astra, although tied with Fable 5.2 for the highest ECI score of 174, averaged 65% and ranked fourth. On one task, Fable 5.2 and Opus 5.5 reached the level of contract workers.

Full text · Original

Can the most capable LLMs match humans at real work? Not yet, at least not on the kind of work we do at Epoch.

We tested GPT-6.2 Astra and other recent models on Epoch Automation Reports, our benchmark of open-ended Epoch work tasks, ranging from making Epoch-style diagrams to designing basic research experiments.

An Epoch employee scores the models’ outputs based on our internal quality standards. A score of 100% represents the standard expected of an Epoch employee.

All models failed to meet the Epoch employee standard. GPT-6.2 Astra ranks fourth (averaging 65%), despite sharing the top ECI score of 174 with Fable 5.2.

Fable 5.2 scored highest, averaging 82%.

On one task, both Fable 5.2 and Opus 5.5 matched the level of a contractor. More on that below.

Source: Epoch AI · x.com