Epoch AI's actual tests: the strongest LLMs all fail to meet its employees' work standards
Epoch AI used the Epoch Automation Reports benchmark to test the latest models such as GPT-6.2 Astra, and none met the work standards of Epoch employees. Fable 5.2 ranked first with an average of 82%; GPT-6.2 Astra, although tied with Fable 5.2 for the highest ECI score of 174, averaged 65% and ranked fourth. On one task, Fable 5.2 and Opus 5.5 reached the level of contract workers.
Can the most capable LLMs match humans at real work? Not yet, at least not on the kind of work we do at Epoch.
We tested GPT-6.2 Astra and other recent models on Epoch Automation Reports, our benchmark of open-ended Epoch work tasks, ranging from making Epoch-style diagrams to designing basic research experiments.
An Epoch employee scores the models’ outputs based on our internal quality standards. A score of 100% represents the standard expected of an Epoch employee.
All models failed to meet the Epoch employee standard. GPT-6.2 Astra ranks fourth (averaging 65%), despite sharing the top ECI score of 174 with Fable 5.2.
Fable 5.2 scored highest, averaging 82%.
On one task, both Fable 5.2 and Opus 5.5 matched the level of a contractor. More on that below.