Skip to content

TheoPhysEval: Evaluating AI agents' ability to autonomously solve theoretical physics problems

TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving

arXiv· arXiv:2610.16733· A. Ramesh Chandra, Aaghaz Mahajan, Ainesh Sanyal, Mianqi Wang, Shubhadeep Chakraborty, Sudip Chakraborty, Shailesh Chandrasekharan, Rama Govindarajan, Akshit Goyal, Jonathan J. Heckman, Yasir Iqbal, Divya Jaganathan, Saumav Kapoor, Andreas Karch, Pushkar Khandare, Sandeep Krishna, Anupam Kundu, Alok Laddha, Tongyan Lin, Swarnendu Maity, Nilmani Mathur, Shani Nadir Meynet, Alessandro Mininno, Abhijeet Mishra, Vidur Niranjan, Abhishodh Prakash, Pratik Rath, Durbar Roy, Debargha Sarkar, Srikanth Sastry, Gary Shiu, Aninda Sinha, Thomas Van Riet, Praneeth Netrapalli, Suvrat RajuInternational Centre for Theoretical Sciences; Tata Institute of Fundamental Research; University of Texas at Austin; Kavli IPMU (WPI), The University of Tokyo; Harish-Chandra Research Institute; Homi Bhabha National Institute; Duke University; University of Pennsylvania; Indian Institute of Technology Madras; Harvard University; Jawaharlal Nehru Centre for Advanced Scientific Research; Chennai Mathematical Institute; University of California, San Diego; University of Wisconsin–Madison; Monash University; Indian Institute of Science; University of Calgary; KU Leuven; Google DeepMindIndexed by this site · 2026-10-18· Originally publishedEvaluation

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

Five agents evaluated on 50 unpublished theoretical physics problems; Astra passes 49/50 on the realistic track.

Questions
The goal is to test whether off-the-shelf agents can autonomously solve research-level theoretical physics computation problems whose answers are verifiable and not findable in public literature at the time of evaluation, without human prompting. The authors state that such problems represent concrete computations in research, not open breakthroughs.
Methods
Experts provide 50 problems. Symbolic answers are converted to numerical values to avoid using an LLM judge. Under Harbor, two tracks are defined: realistic (retrieval and local tools allowed) and isolated (web retrieval disabled). Each of the five agents runs once per track, and correctness is judged by the final numerical value.
Experiments and results
Realistic pass counts: Astra 49/50, Opus 46/50, Gemini 34/50, DeepSeek 27/50, GLM 21/50. On isolated, Astra and Opus are 46/50, GLM 26/50. No one solved §1b.
Limitations and future work
The authors state that observations about problem taste and changing direction cannot be quantified and do not constitute claims about future AI. The suite is not an adversarial problem set. Each agent runs once per track; the conventions in §9a and §11d are not fully specified.
Experimental setupGPT-6-Astra、Opus 5.5 et al. · TheoPhysEval · pass rate、accuracy et al.
Evaluated models
GPT-6-AstraOpus 5.5Gemini 3.8 FlashDeepSeek-V4.1-FlashGLM 5.3
Benchmarks
TheoPhysEval
Metrics
pass rateaccuracytoken usageevaluation cost (USD)
AI summaryTheoPhysEval proposes a set of 50 unpublished, answer-verifiable research-level theoretical physics problems to test off-the-shelf AI agents' ability to solve problems autonomously without human prompting. The paper states that when tools such as literature search and numerical and symbolic computation scripts are available, the best-performing agent, GPT-6-Astra with Codex, achieved 98% accuracy, with an average cost per problem below $3. After disabling web search and background research papers, the agents' accuracy changed only slightly on average, which the paper suggests indicates significant memorization of the literature. The authors also discuss the implications of these results and possible reasons why agents excel at problem-solving yet have not yet led to breakthroughs in theoretical physics.

TheoPhysEval proposes a set of 50 unpublished, answer-verifiable research-level theoretical physics problems to test off-the-shelf AI agents' ability to solve problems autonomously without human prompting. The paper states that when tools such as literature search and numerical and symbolic computation scripts are available, the best-performing agent, GPT-6-Astra with Codex, achieved 98% accuracy, with an average cost per problem below $3. After disabling web search and background research papers, the agents' accuracy changed only slightly on average, which the paper suggests indicates significant memorization of the literature. The authors also discuss the implications of these results and possible reasons why agents excel at problem-solving yet have not yet led to breakthroughs in theoretical physics.

Related events and news

4 events · 6 news mentions

News that mentions this paper

七位物理学家谈AI对理论物理的冲击

Simons基金会与OpenAI、Anthropic出资1500万美元支持数学家检验AI成果

Anthropic联创在IMF演讲称AI奇点已开始

Show 3 more news items

TheoPhysEval:GPT-6-Astra解出49/50物理题

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Evaluate whether off-the-shelf agents can solve verifiable research-level theoretical physics computation problems without prompting.

    Can off-the-shelf agents autonomously solve research-level theoretical physics computation problems with verifiable answers, without human prompting?

    • Scenario: The authors state that in recent months agents have been able to perform various scientific tasks with little or no human supervision. In mathematics, open problems have already been solved by agents; in theoretical physics, the authors state that no notable breakthrough has yet been achieved, but researchers have used agents to discuss ideas, perform computations, search literature, and draft manuscripts. This paper evaluates one of these: autonomously solving problems that are clearly posed and have verifiable answers.
    • Existing evaluations: The authors state that existing work covers middle school, competition, and undergraduate physics; TPBench, SciCode, FrontierScience, PRL-Bench, QMBench, CritPt, etc., include research-level problems but also simpler ones. Most evaluations typically do not allow agents to use the full set of tools available in practice.
    • Problem nature: The 50 problems come from world experts in various directions. The authors call them "turn-the-crank" problems: extending existing results with known techniques; the results are strictly new, sometimes even publishable, but not path-breaking. Without AI assistance, a typical problem can be solved by a world expert in about a day or less, while those in adjacent directions take much longer. Problems are not rejected just because agents can solve them, so this is not an adversarial benchmark. The authors note that some problems' human solutions used AI, so data leakage cannot be ruled out, and the stricter standard of "not touched by AI" is not met.
    • This paper proposes: TheoPhysEval. The realistic track allows web search and local tools; the isolated track disables web search except for calls to the model provider. Scoring is automatic via Harbor based on final numerical answers, without human hints.
  2. What related research exists?

    The authors place this work among research-level scientific benchmarks and compare it with human proof evaluation in First Proof.

    Grouped by the introduction and Previous work.

    Middle school to undergraduate

    • School stages [19, 20], competitions [21, 22], and undergraduate physics [23]: The authors state that there is already substantial work evaluating LLMs at these levels. The main text does not detail each benchmark's approach or criticize them individually.

    Benchmarks containing research-level problems

    • TPBench[24], SciCode[25], FrontierScience[26], PRL-Bench[27], QMBench[28], CritPt[29]: These include research-level problems in addition to simpler ones. The authors state that the difference from this literature is that this paper only includes modern research-level problems; the evaluation method also differs because most other evaluations typically do not allow agents to use the full set of tools they would use in practice.

    Broader scientific workflow benchmarks

    • Terminal-Bench-Science[30]: This has broader coverage than this evaluation. The authors state that this evaluation focuses on theoretical physics and covers it in greater depth; because it evaluates real problems physicists encounter, it does not discard problems just because agents can solve them. This evaluation uses the Harbor framework used by Terminal-Bench.

    Unpublished research-level math problems

    • First Proof[31–34]: The authors state that this is roughly comparable to this evaluation, both involving unpublished research-level problems. The difference is that First Proof uses human evaluation of proofs, while this evaluation is automatic and only looks at final answers.

    Comparison settings

    • The paper does not set up an independent external baseline method. The comparison is the pass rates of five off-the-shelf agents on the same 50 problems across the realistic and isolated tracks (Section IV, Figure 3, Figure 4, Table IV).

    The authors position this paper as including only modern research-level theoretical physics problems, allowing tools in realistic settings, and automatically scoring only final answers; the problems are meant to represent research problems, not an adversarial benchmark. The introduction contrasts this with autonomous solving that has already appeared in mathematics, and states that theoretical physics has not yet seen a similar breakthrough.

  3. How does the paper address this problem?

    50 expert problems converted to numerical answers, automatically scored on both retrieval-enabled and retrieval-disabled tracks.

    Experts provide problems, the organizer modifies them and implements the benchmark, scoring by final numerical answer using Harbor on both realistic and isolated tracks.

    Selection criteria and sources

    • Three criteria (Section II): relevant to current physics research; known solutions are easy to verify and hard to guess; solutions cannot be found in published literature or elsewhere. Contributors are also asked not to publish solutions in the short term. These criteria exclude ongoing research that experts plan to publish soon.
    • Difficulty trade-off: Contributors are asked to provide problems as hard as possible in their fields, but not to reject problems just because agents can solve them. The authors state the suite should represent problems in theoretical physics, not an adversarial benchmark.
    • Two sources: problems already solved but unpublished in research notes; or adapted from their own recent papers, changing several parameters so the new answer cannot be guessed from the old one. Contributors may use existing code to solve; another expert or agent must locate and understand the original solution, reconstruct the computation, and adapt it. The authors state such exercises are often given to graduate students starting research.
    • Examples: Box 1 gives three examples—black hole string Page time, class D fermion parity pump, and flat-plate boundary layer—with expected solution trajectories. The holography problem is related to an existing paper [35], and the statement is deliberately brief; one of the original authors (SR) took about a day to adapt the original computation. The benchmark asks: given no hints beyond the prompt, can an agent autonomously complete the solution steps?

    Scale and topics

    • 50 problems: Table I lists classical physics 5, stochastic processes and biophysics 5, nonequilibrium statistical mechanics 3, lattice theory 3, quantum phases of matter 5, electronic structure 4, quantum field theory 7, particle physics 2, scattering amplitudes 5, holography and entanglement 5, gravity and cosmology 6. Full text in Appendix A.
    • Coverage: The authors state they do not believe any physicist could attempt more than a small fraction; within a topic category, even a single expert often cannot do all problems. The distribution is influenced by the invited network and by discipline: fields like quantum field theory are easier for constructing hard computations with verifiable answers, while others are harder. The authors state this is not an exhaustive survey of all theoretical physics.

    Converting symbolic answers to numerical values

    • Why not use an LLM judge: The same expression can be written in many equivalent ways, and notation is not standardized. Using an LLM to convert answers to standard notation and then compare symbolically would introduce additional error.
    • Method: Give agents numerical values for relevant and irrelevant physical quantities, converting expressions to floating-point numbers or hard-to-guess integer sequences (Box 2). Score based on these numerical answers.

    Tracks and runs

    • realistic: web retrieval and local tools allowed. The authors state professional physicists have no reason to turn off retrieval or rewrite existing scientific code, so this track better reflects assistant use.
    • isolated: local tools allowed; web retrieval disabled except calls to the model provider itself. Aimed at model developers who want to test intrinsic physics knowledge and the ability to rewrite research-level code and computations from scratch. Runs in a local Docker sandbox managed by Harbor with internet off. The authors state automated analysis of trajectories found no instances of sandbox escape.
    • Machine and time limits: Final evaluation on Google Cloud e2-standard virtual machines, each problem with 2 vCPU, 4 GB RAM, 10 GB storage. No total cost or token limits. If not completed in 8 hours, it is cut off and recorded as no output. The multi-task problems in Appendix A.6 have a combined time limit of 20 hours.
    • Number of runs and reasoning effort: To save time and cost, each agent runs the full suite only once via API. The authors remind that outputs may differ across runs at nonzero temperature. Except for Gemini 3.8 Flash, which uses the highest available setting high, reasoning effort is set to max for other models. The main text does not give temperature values.
    • Five agents: GPT-6-Astra with Codex; Opus 5.5 with Claude Code CLI; Gemini 3.8 Flash with Antigravity; DeepSeek-V4.1-Flash with Codex; GLM 5.3 with Claude Code CLI.

    Pass determination and known ambiguities

    • Primary determination: whether the final numerical value matches the scoring key. The four electronic structure problems are sequential, and Appendix A.6 specifies tolerances for some fields; Table IV still counts pass/fail per whole problem.
    • Unspecified conventions: After the final round, it was found that conventions for §9a and §11d were not fully specified. The scoring key allows the convention choices observed in the evaluation. Appendix A footnotes state that §9a does not explicitly specify N, and §11d's moduli space metric, axial charge, and asymptotic modulus are unspecified; scoring incorporates the choices seen in the evaluation.
    • §1b: Difficult to do the numerical conversion of Box 2. Requires outputting 0 if no equation is found, or the equation if found; any equation that visually fits the curve is judged correct. Appendix B gives an example equation with fractional derivatives and time delays, and states the solution is not unique.
  4. What experiments did the paper conduct?

    The authors report Astra passes 49/50 on realistic; most agents change little after disabling retrieval, except GLM.

    Experimental setup

    • Agents tested: GPT-6-Astra with Codex, Opus 5.5 with Claude Code CLI, Gemini 3.8 Flash with Antigravity, DeepSeek-V4.1-Flash with Codex, GLM 5.3 with Claude Code CLI. Reasoning effort is max except for Gemini 3.8 Flash which is high.
    • Problems: 50, topics in Table I, full text in Appendix A. The authors state answers were not public at evaluation time; after the experiment, one solution to §10a has been publicly released [61]. The introduction notes that some human solutions used AI, so data leakage cannot be ruled out.
    • Tracks: realistic allows web retrieval and local tools; isolated disables web retrieval (except calls to the model provider). Each agent runs once per track via API.
    • Environment: Harbor; isolated is a Docker sandbox with internet off. e2-standard: 2 vCPU, 4 GB RAM, 10 GB storage. General tasks have an 8-hour deadline, Appendix A.6 has a total of 20 hours. No cost or token limits.
    • Judging: Pass/fail is based on whether the final numerical value matches the scoring key, not an LLM judge, and proofs are not evaluated. §9a and §11d allow multiple conventions; §1b is judged by visual fit of the curve.
    • Not reported in the paper: specific temperature values; variance across multiple runs. Expert review was used only to correct some reference solutions; the paper does not report agreement coefficients with human scoring.

    Main results

    Pass rates are from Figure 3 (realistic) and Figure 4 (isolated); problems solved are from the last row of Table IV.

    Swipe horizontally to read the table

    AgentRealistic pass rateIsolated pass rateRealistic solvedIsolated solved
    GPT-6-Astra with Codex98.0%92.0%49/5046/50
    Opus 5.5 with Claude Code CLI92.0%92.0%46/5046/50
    Gemini 3.8 Flash with Antigravity68.0%62.0%34/5031/50
    DeepSeek-V4.1-Flash with Codex54.0%50.0%27/5025/50
    GLM 5.3 with Claude Code CLI42.0%52.0%21/5026/50
    • Two statements by the authors: The introduction states that some frontier models may already surpass the strongest human researchers in specific computations. Section V states that breadth exceeds any individual, and computational capability in specialized areas is now comparable to world experts. The abstract reports Astra's accuracy as 98%, and Figure 3 labels it 98.0%.
    • Two tracks: The authors state that most agents show little change in accuracy between tracks, and believe frontier LLMs have effectively memorized a large portion of theoretical physics literature, reproducing key formulas without consultation. In the table, GLM is the row where isolated is higher than realistic; the authors' explanation for this exception is in the next section.
    • Comparison with other benchmarks: The authors state that visually, this problem set appears harder on average than existing theoretical physics problem sets including CritPt, yet pass rates are higher. The authors note that recent literature suggests some theoretical physics benchmarks may have inaccurate evaluations [50], and understanding this difference is of interest. The paper does not give numbers for those benchmarks.

    Errors and qualitative analysis

    • Unsolved inverse problem: §1b is 0 for all five agents on both tracks in Table IV. The authors state this is an inverse problem with information asymmetry between the problem setter and the agent, and the solution also plays on the wording "differential equation" in the prompt. Appendix B states that from a strict perspective it may seem unfair, but it resembles situations where researchers face slightly imprecise data. In private chat (not the final API), Astra guessed the need for fractional derivatives and time delays but still did not find the parameters.
    • Astra and reference solutions: On realistic, Astra only misses §1b. On isolated, it misses 3 additional problems, 2 of which it incorrectly rejected as unsolvable. The authors state that initially they thought Astra had more errors, but after experts corrected the original solutions, Astra's solutions were correct. The problems solved by other models, and by Astra on isolated, are strict subsets of the problems Astra solved on realistic.
    • Other error patterns: Opus on realistic used initial data from reference [51] for §1c instead of the prompt-specified values, but not on isolated; on §9a both tracks gave incorrect conclusions about the convergence rate of infinite series; on §7a on realistic it repeatedly hit the maximum output token limit without valid output, but solved it on isolated; on §7d on isolated it made a small error in setting up the modular differential equation, but not on realistic. On §10d, Gemini and GLM on realistic used formulas from [35] that apply to five dimensions and need to be adapted to four; on isolated, Gemini, GLM, and DeepSeek computed geodesic lengths instead of the required minimal area. On §11f, GLM on both tracks incorrectly stated the problem was equivalent to another; DeepSeek only on isolated chose the wrong basis, but solved it on realistic; Gemini had a similar error on realistic, and no output on isolated. The authors state GLM's higher isolated score is mainly because Appendix A.6 provides key literature, reducing the disadvantage of isolation, while on realistic early symbolic errors propagated to later subproblems.

    Costs and tokens

    Costs are from Table III, computed as tokens times API unit price (except GLM).

    AgentRealistic (USD)Isolated (USD)
    Astra116.2895.20
    Opus240.43275.46
    Gemini95.0190.48
    DeepSeek37.4236.23
    GLM (coding plan, both tracks combined)180180
    • The abstract states Astra averages below $3 per problem; Table III gives totals per track, and the paper does not list per-problem averages. If GLM were priced by API, realistic would be $311.71 and isolated $371.46. The authors state Opus costs are notably higher than other models.
    • Total tokens (millions, Total column of Table II): realistic: Astra 37.41, Opus 211.18, Gemini 606.71, DeepSeek 1025.60, GLM 971.96; isolated: Astra 24.53, Opus 228.43, Gemini 594.20, DeepSeek 833.97, GLM 1115.61. The authors state input tokens far exceed prompt tokens because agents write scripts and read outputs, and on realistic also access web pages. Cached and uncached are priced separately; the authors state this division has limited scientific meaning and depends on runtime conditions at the API end. After rounding, component sums may not exactly equal totals.

    Other ablations and analyses

    • No hyperparameter or component ablations; reasoning effort fixed at max (Gemini high).
    • One run per agent per track; the authors note outputs may differ across runs at nonzero temperature (Section III).
    • DeepSeek's §11e on isolated was cut off due to timeout, with no token record (Table VIII footnote).
    • Per-problem pass, token, and dollar costs are in Appendix C (Tables IV–X).
    • The authors state automated analysis of isolated trajectories found no instances of sandbox escape (Section III).
    • The authors state some errors by other models would be obvious to human experts, such as §10d not adapting the five-dimensional formula to four dimensions (Section IV).
  5. What are some points that can be further explored?

    The authors state that observations about problem taste and changing direction cannot be quantified and do not constitute claims about future AI.

    Limitations and future directions noted by the authors

    • Non-quantifiable observations: Regarding claims that current agents are not good at changing direction when problems are too hard, lack physicists' shared judgment of "interesting problems," and cannot be entrusted with major physics problems autonomously, the authors state these are based on experience and conversations and cannot be quantified; other researchers may have different experiences; they do not constitute claims about future AI development. (Section V)
    • Specialized harnesses: The authors state the benchmark uses common consumer-grade harnesses. Specialized harnesses for scientific work have had successes in mathematics. Whether such harnesses can improve accuracy on this benchmark is a question the authors find interesting. (Section V)
    • Model-harness pairings: The authors state that Astra, Opus, and Gemini paired with Codex, Claude Code, and Antigravity respectively are natural, but DeepSeek with Codex and GLM with Claude Code are their choices; different combinations might be better, and readers are encouraged to try. (Section V)
    • Non-adversarial problem set: The authors state that problems could perhaps be constructed adversarially to lower accuracy, such as the unsolved inverse problem §1b, and more such problems could likely be constructed; they chose not to do so because the suite should represent problems actually encountered in research. (Section V)
    • Training and resources: The authors state that unless the goal is explicitly to work through the problems rather than obtain final results, it is not advisable to use some problems in the set as initial projects for beginning researchers. The authors expect harder problems to require more resources, raising issues of access and equity in physics research. (Section V)

    Experimental coverage

    • The five agents listed above were each run once via API on both realistic and isolated tracks (Sections III, IV).
    • There are 50 problems, covering the 11 categories in Table I; correctness is judged by whether the final numerical answer passes, not by proof evaluation (Sections I.A, III).
    • Time limits are 8 hours for general tasks and 20 hours total for Appendix A.6; no cost or token limits; reasoning effort is max except for Gemini 3.8 Flash which is high (Section III). The paper does not report temperature values.
    • The authors state in the introduction that data leakage cannot be ruled out because some human solutions used AI; after the experiment, one solution to §10a has been publicly released (introduction). The conventions for §9a and §11d are not fully specified, and scoring incorporates observed choices; §1b is judged by visual fit (Section III, Appendix A footnote, Appendix B).
    • Per-problem pass, token, and cost details are in Appendix C; DeepSeek's §11e on isolated has no token record due to timeout (Table VIII).
  6. Summarize the main content of the paper

    Astra passes 49/50 on realistic; the authors state that concrete computation is already strong, but autonomous research still requires humans to set direction.

    TheoPhysEval uses 50 unpublished, verifiable research-level theoretical physics problems to test whether off-the-shelf agents can autonomously produce final answers without human prompting.

    • Capability tested: The authors are not concerned with open major breakthroughs, but with "turn-the-crank" computations: extending existing results with known techniques. Without AI, a typical problem can be solved by a world expert in about a day or less, while adjacent directions take much longer. Some problems' human solutions used AI, and the authors note that data leakage cannot be ruled out.
    • Method: Experts provide problems, the host modifies them and implements scoring. Symbolic answers are converted to hard-to-guess numerical values, then automatically checked without LLM judges. realistic allows search and local scripts; isolated disables web search. Each of the five agents runs once. Reasoning effort is max except Gemini 3.8 Flash which is high. Timeout is generally 8 hours, totaling 20 hours in Appendix A.6.
    • Pass counts: On realistic, Astra 49/50 (98.0% in Figure 3), Opus 46/50 (92.0%), Gemini 34/50 (68.0%), DeepSeek 27/50 (54.0%), GLM 21/50 (42.0%) (Table IV). On isolated, Astra and Opus both 46/50 (92.0%), Gemini 31/50 (62.0%), DeepSeek 25/50 (50.0%), GLM 26/50 (52.0%). None solved the inverse problem §1b. The authors state that most agents show little change between tracks, and they believe the literature is heavily memorized; GLM is the opposite, which the authors attribute mainly to a set of electronic structure problems where key literature was provided, and symbolic errors propagated to later subproblems.
    • Cost: Astra's realistic total cost is $116.28 (Table III). The abstract states its average per problem is under $3. Opus's realistic is $240.43 and isolated is $275.46.
    • Authors' conclusions: The authors state that frontier agents can already accurately perform concrete computations needed across theoretical physics directions, with breadth exceeding an individual and specialized areas comparable to world experts; the introduction also states that some frontier models may already surpass the strongest human researchers in concrete computations. The authors also state that concrete computation only appears late in research; constructing theories, posing appropriate high-level questions, and turning problems into computations still require human experts to set direction; currently, one cannot hand a major physics problem directly to an agent and expect it to solve it autonomously. Regarding problem taste and changing direction, the authors state these cannot be quantified and do not constitute claims about future AI.
Read originalarxiv.org