How strong are frontier models' physics abilities? Expert rescoring reveals evaluation flaws and leading benchmarks nearing saturation

Do the low scores of frontier large language models on mainstream physics benchmarks reflect genuine physics reasoning defects in the models, or do they stem from errors in the evaluation benchmarks and scoring processes themselves.

• Background and cognitive disconnect: Recent physics benchmark tests report that frontier models perform far below human physics experts, yet this conclusion clearly conflicts with the relatively strong capabilities physicists experience when using the models in actual research work.
• Inherent flaws in the evaluation process: Existing physics benchmarks, when calculating error rates, conflate model reasoning failures, rule-based grader misjudgments, and benchmark defects themselves (such as ambiguous questions, missing physics assumptions, or incorrect reference answers).
• Core observation and hypothesis: As model reasoning capabilities improve, their genuine physics error rate may have dropped below the defect rate of the benchmarks themselves, causing most of the errors reported by benchmark evaluations to actually belong to defects in the evaluation materials themselves.
• The paper's research approach: Organize a team of physics experts to conduct detailed audits of six mainstream physics benchmarks, systematically separate model errors, grader errors, and benchmark errors, and re-evaluate frontier models after fixing or excluding defective questions.

• Exploration of frontier models in open physics research: Bubeck et al. (2025) reported that GPT-5 Pro reconstructed the SL(2, R) symmetry algebra behind the black hole Love number analysis; Schwartz (2026a, 2026b) guided Claude to derive a new factorization theorem and complete the C parameter calculation; Brenner et al. (2026) combined Gemini Deep Think with search to derive an analytic solution for cosmic string radiation. The authors note that these explorations involved expert intervention or external verification and lacked standard reference answers, failing to rigorously quantify physics capability on standardized tasks.
• Research on benchmark flaws evolving with model capability: Northcutt et al. (2021) pointed out that test set label errors can disrupt model rankings; Bowman and Dahl (2021) proposed sufficiency criteria for benchmark testing and emphasized that fixed benchmarks lose statistical power at high accuracy stages. The authors cite their view that when model capability improves such that the true error rate falls below the benchmark flaw rate, grader and benchmark errors mask true performance.
• Evaluation mechanism flaws and software engineering benchmark precedents: Zheng et al. (2023, 2025), Chen et al. (2024), and Tan et al. (2025) analyzed biases and objective reasoning flaws in LLM judges; Jimenez et al. (2024) and OpenAI (2024, 2026c, 2026d) audited SWE-bench and found that 59.4% of tasks in the Verified subset had substantive issues. The authors note that physics benchmarks rely on manually written answers and lack executable tests, making flaws harder to detect.
• Evaluation benchmarks and tested subjects: The paper compares public-source benchmarks UGPhysics, PHYBench, PRISM-Physics, and expert-original benchmarks HLE-Physics, CMT-Benchmark, and CritPt, evaluating GPT-5.6-Sol, Claude Fable 5, and Gemini 3.1 Pro.
• Positioning of this paper: The authors note that previous benchmarks underestimated model physics capability; this paper, through per-problem derivation and review by domain experts, systematically separates benchmark errors, grading errors, and model errors for the first time.

The authors propose a diagnostic method combining benchmark evaluation with expert audit, in which physics domain experts thoroughly examine problem statements, reference solutions, and model outputs to separate genuine model errors from evaluation system flaws, and re-evaluate after correcting or excluding flawed problems.

Problem Setup and Benchmark Categorization

• Six benchmarks and source classification: The evaluation covers six closed-book physics benchmarks, categorized by problem source into two types: one type originates from public problems (UGPhysics, PHYBench, PRISM-Physics), with potential contamination risk; the other is originally created by domain experts (HLE-Physics, CMT-Benchmark, CritPt), with lower contamination risk.
• Closed-book and answer equivalence: All benchmarks are closed-form problems requiring the model to output a definite final answer; as long as the final answer is mathematically equivalent to the reference solution, it is judged correct regardless of the derivation path.

Error Attribution Taxonomy

• Model error: The problem statement is clear and well-defined, and the reference solution is correct, but the tested model gives a wrong answer; only this type counts as a genuine physics reasoning error by the model.
• Grader error: The problem is well-defined and the reference solution is correct, and the model gives the correct answer, but the evaluator (mainly rule-based evaluators) incorrectly marks it wrong due to failure to recognize equivalent mathematical forms or different notation conventions.
• Benchmark error: The problem statement or reference solution has flaws, including incorrect reference solutions, contradictory problem conditions, ambiguous wording, or missing key physical assumptions necessary for deriving the required result.

Expert Audit and Correction Process

• Expert matching and two-stage initial screening: An expert team of physics faculty and graduate students is formed, assigning problems by physics subfield, first independently deriving or reviewing the problems.
• Exclusion strategy for public-source benchmarks and HLE: For problems in UGPhysics, PHYBench, PRISM-Physics, and HLE-Physics that GPT-5.6-Sol High initially screened as all wrong, expert review is conducted; problems verified as benchmark errors are directly excluded.
• Repair strategy for original benchmarks: For CritPt and CMT-Benchmark, experts repair the problem or reference solution where possible, such as clarifying the Hamiltonian normalization convention or adding boundary conditions; problems that cannot be reasonably repaired are excluded. For example, in CritPt problem 44, experts added a standard-form Hamiltonian including Pauli matrices to eliminate ambiguity in spin conventions:

H = − ∑_{⟨ ij ⟩x} σ_i^x σ_j^x − ∑_{⟨ ij ⟩y} σ_i^y σ_j^y − ∑_{⟨ ij ⟩z} σ_i^z σ_j^z This formula provides the standard microscopic Hamiltonian definition of the isotropic Kitaev honeycomb model, giving the ground-state energy a uniquely determined numerical value.

Re-evaluation Protocol

• Unified evaluation pipeline: The corrected evaluation uniformly uses an evaluation pipeline improved from HLE, with a unified system prompt for generating answers and an LLM judge prompt to compare answer equivalence.
• Statistical metrics: Reports mean@4 (average accuracy over 4 independent attempts) and pass@4 (proportion of problems with at least one correct answer in 4 attempts); for CritPt, due to the lack of official per-problem reference answers, evaluation is based on reference answers independently derived by experts.

Experimental Setup

• Models tested: Three frontier models were tested: GPT-5.6-Sol, Claude Fable 5, and Gemini 3.1 Pro. Reasoning effort was typically set to High; on CritPt, GPT-5.6-Sol used Max, Fable 5 used Max for pre-audit and High after correction.
• Tool configuration: On public-source benchmarks (PHYBench, PRISM-Physics, UGPhysics), all models used no tools; on expert-original benchmarks (HLE-Physics, CMT-Benchmark, CritPt), GPT-5.6-Sol used Codex to enable tools, Fable 5 used Claude Code to enable tools, and Gemini 3.1 Pro used no tools throughout.
• Benchmarks and sample sizes: A total of 6 benchmarks were evaluated. Pre-audit and retained problem counts are: PHYBench (100 → 87), PRISM-Physics (100 sampled → 74), UGPhysics (100 sampled → 82), HLE-Physics (202 text problems → 116), CMT-Benchmark (50 → 49), CritPt (70 pre-test/56 audited → 54 retained).
• Evaluation metrics: Primary metrics are mean@4 (average accuracy over 4 independent attempts) and pass@4 (proportion with at least one success in 4 attempts); CritPt pre-audit scores are the 70-problem mean@5 reported by Artificial Analysis.
• Review method: Pre-audit used the benchmark's own grader or the HLE-adapted grader; after correction, all used the HLE LLM judge prompt for scoring. Initial screening audit was conducted by physics PhD students and faculty matched by subfield.

Main Results

Swipe horizontally to read the table

• Source: According to Table 1. The table shows mean@4 accuracy (pre-test → corrected). CritPt pre-audit is Artificial Analysis's mean@5; on CritPt, GPT-5.6-Sol used Max, Fable 5 used Max for pre-audit and High after correction; Gemini 3.1 Pro had no tools throughout; CritPt pre-audit pass@4 is not reported.
• Accuracy generally recovers: The authors state that after excluding or repairing flawed problems and correcting the scoring mechanism, all models' scores improved on all six benchmarks, with GPT-5.6-Sol achieving pass@4 of 94.44% on the CritPt retained set and pass@4 of 97.96% on CMT-Benchmark.
• Benchmark difficulty reordering: The authors state that PRISM-Physics, which appeared to have low scores in pre-audit (pre-test only 7.50%–13.00%), reached 94.59% for GPT-5.6-Sol after correction, indicating that the low scores were mainly due to flaws in the rule-based grader rather than the physical difficulty of the problems.
• Performance on closed-book physics problems: The authors believe that frontier models have shown strong problem-solving ability on well-defined closed-book physics exercises, and the previously reported low scores on benchmarks were mainly artifacts of flawed evaluation materials and rule-based graders.

Error Attribution and Source Analysis

• What was measured: Expert attribution review was conducted on 250 problems from HLE-Physics, PHYBench, PRISM-Physics, and UGPhysics that GPT-5.6-Sol High initially screened as all wrong (Table 2).
• Results: Among the 250 initial screening errors, benchmark errors accounted for 143 (57.20%), grader errors for 95 (38.00%), and genuine model errors only 12 (4.80%). Among PRISM-Physics's 74 reported errors, model errors were 0 (0.00%); in UGPhysics, only 1 (4.55%); in PHYBench, only 3 (5.36%); in HLE-Physics, 8 (8.16%).
• Authors' interpretation: The authors state that the vast majority of reported errors (95.20%) stem from benchmark flaws or grader misjudgments, and the genuine physics error rate of frontier models is far lower than the benchmark flaw rate, with models showing almost no physics reasoning errors on datasets such as PRISM-Physics and UGPhysics.

Reviewer Agreement and Conflict Resolution

• What was measured: The labeling consistency of physics researchers in problem attribution review was assessed (Table 3 and Table 4).
• Results: In the completed two-stage labeling, 196 problems were jointly reviewed by two independent reviewers, of which 140 pairs of labels were fully consistent (agreement rate 71.43%), and 56 pairs had disagreements (28.57%). Among disagreements, the majority were between benchmark error and grader error (34 cases), benchmark error vs. model error had 19 cases, and grader vs. model error only 3 cases; all 56 disagreements were independently adjudicated by a third reviewer.
• Authors' interpretation: The authors state that reviewer disagreements mainly occurred in determining whether the problem statement was flawed or the grader failed to recognize equivalent answers, while judgments on whether the model made a substantive error were highly certain.

Other Ablations and Analyses

• Error rate of the HLE-adapted LLM judge grader itself: 4.08% (Section 3).
• Number of benchmark errors identified by experts among CMT-Benchmark's 50 problems: 30 (of which 29 repaired, 1 excluded, Section 3.2.2).
• Number of benchmark errors identified by experts among CritPt's 56 audited problems: 21 (of which 19 repaired, 2 excluded, Section 3.2.3).
• External comparison (Anthropic system card report): Fable 5.1 on CritPt-Corrected's 31 repaired problems achieved pass@1 (i.e., mean@16) of 88.4% over 16 maximum-effort attempts (Section 5).

Unresolved Cases and Exceptions

• Remaining model error cases: In the corrected CritPt, 5 problems are still judged as model errors; for example, in problem 18 (Figure 5), GPT-5.6-Sol Max assumed in all four attempts that the particle polarizability is real, causing the derivation to ignore the phase contribution of the complex polarizability; after correction, HLE-Physics still has 8 problems judged as model errors.

Limitations and Future Directions Noted by the Authors

• Near-saturation on closed-book problems does not imply end-to-end research capability (Section 5): The authors note that models' near-saturation on closed-book physics exercises by no means implies that frontier models possess end-to-end physics research capability.
• Limited progress on unsolved theoretical physics problems (Section 5): The authors attempted to tackle multiple open problems in theoretical physics using a GPT-based agent framework and found that the agents' progress in physics was weaker than in mathematics, failing to fully solve any unsolved physics problem.
• Call for building rigorously validated, high-difficulty new benchmarks (Section 5): The authors call for establishing new-style physics benchmarks for frontier models, with problems sourced from entirely new high-difficulty physics tasks and fully validated.
• Data collection cost and organizational mechanisms for high-quality benchmarks (Section 5): The authors note that building high-quality physics benchmarks requires substantial funding to ensure review quality, and suggest organizing data collection through rigorous non-profit processes or public competition mechanisms.
• CritPt official reference answers unavailable (Section 3.2.3, Appendix B.2.3): CritPt's official per-problem judgments and reference answers are not publicly released; the corrected evaluation relies on reference answers independently derived by review experts.

Experimental Coverage

• Models tested: The experiments tested 3 models: GPT-5.6-Sol, Claude Fable 5, and Gemini 3.1 Pro, without covering other frontier models or open-source model families.
• Benchmarks and problem types: The experiments covered 6 closed-book physics benchmarks, evaluating only text-only problems with definite final answers, excluding multimodal image problems (e.g., HLE-Physics excluded 28 multimodal problems, PRISM-Physics excluded 549 image problems).
• Audit sampling strategy: For HLE-Physics, PHYBench, PRISM-Physics, and UGPhysics, only problems that GPT-5.6-Sol High initially screened as all wrong were expert-audited; problems that this model initially screened as correct and errors unique to other models were not audited.
• Attempt budget and tool configuration: The main evaluation metrics were limited to 4 attempts (mean@4 and pass@4); Gemini 3.1 Pro did not have access to code execution tools throughout, and only GPT-5.6-Sol and Fable 5 had tools enabled on the expert-original benchmarks.
• Unreported information: The paper does not report official evaluation results for each CritPt problem, and CMT-Benchmark's automatic scoring code is not publicly released.

This is a systematic expert audit and retesting study of the authenticity of physical reasoning evaluations for frontier large language models.

• Core question: Existing benchmarks generally report that frontier models score low on advanced physics problems, but this performance contradicts physicists' actual research experience, so it is necessary to clarify whether the low scores stem from model deficiencies or distorted evaluation systems.
• Diagnostic method: The research team organized physics experts to conduct a fine-grained audit of 6 mainstream physics benchmarks, attributing each initially screened error to genuine model errors, rule-based grader misjudgments, or benchmark defects themselves (incorrect solutions, missing assumptions, ambiguous wording). After removing or fixing defective items, they retested using a unified HLE-based pipeline.
• Main finding one (error attribution): Among 250 initially screened errors sampled from four benchmarks, benchmark errors accounted for 57.20% (143 items), grader errors for 38.00% (95 items), and genuine model errors for only 4.80% (12 items); in the PRISM-Physics sample, model errors were 0 items.
• Main finding two (original benchmark performance): After fixing defects, GPT-5.6-Sol High's mean@4 on HLE-Physics rose from 47.28% to 78.66% (pass@4 reaching 91.38%), and on CMT-Benchmark its mean@4 rose from 61.00% to 87.24% (pass@4 reaching 97.96%).
• Main finding three (frontier benchmark retesting): On the 54 retained challenge items in CritPt, GPT-5.6-Sol Max's mean@4 rose from 32.29% pre-audit to 87.50%, with pass@4 reaching 94.44%; Claude Fable 5's mean@4 rose from 28.57% to 78.24%, with pass@4 reaching 90.74%.
• Authors' conclusion and implications: The authors believe that frontier models are nearing saturation on well-defined closed-book physics exercises, and the low scores reported by benchmarks are mainly an illusion caused by benchmark and evaluation defects; however, this does not mean the models possess end-to-end physics research capability. Current agent-based methods still struggle to make substantial progress on open theoretical physics problems, and there is an urgent need to construct rigorously validated, high-difficulty new physics benchmarks in the future.