Skip to content
EventPast event

TheoPhysEval: GPT-6-Astra solves 49/50 physics problems

3 reports3 reporting sourcesLatest report

Start here

AI overview

TheoPhysEval is a set of 50 unpublished, verifiable research-level theoretical physics problems used to test AI agents' ability to solve problems autonomously without human hints. The paper reports that when tools such as literature search and numerical and symbolic computation scripts are available, the best-performing GPT-6-Astra with OpenAI Codex achieves 98% accuracy, solving 49 problems, with an average cost per problem below $3 and a total round cost of about $116. Anthropic's Opus 5.5 in Claude Code scores 92%, Gemini 3.8 Flash 68%, and open-weight models DeepSeek-V4.1-Flash and GLM 5.3 score 54% and 42%, respectively. The paper also reports that after disabling web search and background research papers, the agents' accuracy changed only slightly on average, suggesting significant memorization of the literature. Other reports state that the benchmark involves Harvard, Google DeepMind, and other institutions, with problems drawn from experts' unpublished research notes covering quantum field theory, holography, and more, and includes both connected and disconnected test tracks; GPT-6 Astra initially scored about 88%, rising to 98% after expert review and correction of reference answers.

Generated by AI from reports · Updated 2026-10-18

Progress timeline

Oct 19
Oct 18
  1. WeChat · AI Era (Xinzhiyuan)
    Harvard and others release TheoPhysEval: GPT-6 Astra scores 98% after experts revise answers

    The TheoPhysEval evaluation, involving institutions such as Harvard and Google DeepMind, shows that GPT-6 Astra scored about 88% in the initial assessment on 50 unpublished theoretical physics problems, rising to 98% after experts reviewed and corrected the reference answers, solving 49 problems. The problems were drawn from experts' unpublished research notes, covering quantum field theory, holographic theory, and more, and included two test tracks, one with internet access and one without; offline, both Astra and Claude Opus 5.5 scored 92%. The report states that the average API cost per problem was under $3. A concurrent MIT team study shows that frontier AI's problem-solving rate in theoretical science rose from 2% in mid-2024 to 35% in mid-2026.

Oct 15
  1. The DecoderSelected
    GPT-6-Astra solves 49 of 50 physics research problems at $2.40 per problem

    On the 50 theoretical physics research problems in TheoPhysEval, GPT-6-Astra running in the OpenAI Codex environment answered 49 correctly (98%), at an average cost of under $3 per problem, about $116 for the full run. Anthropic's Opus 5.5 in Claude Code scored 92%, Gemini 3.8 Flash scored 68%, and open-weight models DeepSeek-V4.1-Flash and GLM 5.3 scored 54% and 42%, respectively. The problems cover 11 subfields of theoretical physics and deliberately exclude major open questions, focusing on routine research problems that apply known methods with slight modifications. Web search had little impact: without web access, Astra and Opus each scored 92%, which the authors take as evidence that leading language models have memorized a large portion of the theoretical physics literature. The authors also say some AI errors are actually due to errors in the reference answers themselves, and argue that the strongest current models may already surpass the best human researchers in concrete computation, but do not yet have the research taste to judge whether a problem is interesting.

Oct 9
  1. arXiv
    TheoPhysEval: Evaluating AI agents' ability to autonomously solve theoretical physics problems

    TheoPhysEval proposes a set of 50 unpublished, answer-verifiable research-level theoretical physics problems to test off-the-shelf AI agents' ability to solve problems autonomously without human prompting. The paper states that when tools such as literature search and numerical and symbolic computation scripts are available, the best-performing agent, GPT-6-Astra with Codex, achieved 98% accuracy, with an average cost per problem below $3. After disabling web search and background research papers, the agents' accuracy changed only slightly on average, which the paper suggests indicates significant memorization of the literature. The authors also discuss the implications of these results and possible reasons why agents excel at problem-solving yet have not yet led to breakthroughs in theoretical physics.

Related discussion

Total 25 items