Skip to content

Translation in progress Some entries still include text in the original language.

Original
Hugging Face Daily Papers· arXiv:2610.29932·Indexed by this site · Originally published

FrontierCS 2: A benchmark for computer science R&D

FrontierCS 2: A Benchmark for Computer Science R&D

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors built FrontierCS 2: 48 CS R&D tasks with the paper's contribution deleted, and among 12 agents Claude Opus 5.5 had the highest pass rate at 58%.

Questions
Existing open-ended programming benchmarks only give a problem statement and score by running the program once, or in real environments cover only a single domain such as machine learning. The authors want to test whether LLM agents can make computer science R&D contributions that surpass the authors' code in the paper's own code repository.
Methods
The code released by the paper's authors is taken, the paper's contribution is removed, and the removed code serves as the human reference standard. The agent works in the repository for five hours and may submit patches at any time and receive development-suite scores; the evaluators privately hold the evaluator, the held-out final suite, and the reference code, and a final submission passes only if it exceeds the reference on every metric and every held-out workload.
Experiments and results
12 agents were evaluated, each running each task once. Claude Opus 5.5 had the highest pass rate at 58%, and all others were below 50%; 16 tasks were passed by no one. Most runs whose final score exceeded the reference still failed to meet the threshold on some metric or workload; the median task difficulty is roughly equal to the FCI of the strongest agent.
Limitations and future work
The authors point out that the final score ignores incidental costs such as circuit area and that pass does not distinguish the magnitude of improvement, and argue that benchmarks should report incidental costs. Experimentally, each agent runs each task only once, and the paper does not report run-to-run variance; DeepSeek V4.1 Flash's cost is a lower bound, and its low-budget lead depends on the pricing method.
Experimental setupClaude Opus 5.5、GPT-6 Astra et al. · FrontierCS 2 · Pass rate、Mean score et al.
Evaluated models
Claude Opus 5.5GPT-6 AstraGPT-6.1 SolGemini 4 ArgonGLM 5.3Muse Wonder 1.0DeepSeek V4.1 FlashKimi K3GLM 5.3 FlashMuse Spark 1.3Qwen 3.8 MaxStep 5 Preview
Benchmarks
FrontierCS 2
Metrics
Pass rateMean scoreFCIFDI
AI summaryFrontierCS 2 (arXiv 2610.29932) is a benchmark containing 48 open-ended tasks that uses a hidden evaluation set to examine LLM agents' ability to complete open-ended computer science research and development; the best agent passed only about half the tasks. The benchmark takes published papers and their open-source code, removes the papers' contribution sections, and uses the removed code as a human reference to set the passing line; agents inherit a runnable code repository and can submit patches at any time, the review only returns development-set scores, the evaluator, final test set and reference are kept confidential, and final submissions are scored against the reference on the final test set. The tasks cover 13 domains including machine learning systems, high-performance computing, databases, security and robotics, with more than 100 private tasks held for later release. The authors evaluated 12 agents; the best, Claude Opus 5.5, passed 58% of runs, and 16 tasks had no run pass at all; most runs that exceeded the reference on the overall final test score still fell short on some metric or workload. The authors say that on the shared capability scale of agents and tasks, the median task difficulty is roughly equivalent to the level the strongest agent can handle.

FrontierCS 2 (arXiv 2610.29932) is a benchmark containing 48 open-ended tasks that uses a hidden evaluation set to examine LLM agents' ability to complete open-ended computer science research and development; the best agent passed only about half the tasks. The benchmark takes published papers and their open-source code, removes the papers' contribution sections, and uses the removed code as a human reference to set the passing line; agents inherit a runnable code repository and can submit patches at any time, the review only returns development-set scores, the evaluator, final test set and reference are kept confidential, and final submissions are scored against the reference on the final test set. The tasks cover 13 domains including machine learning systems, high-performance computing, databases, security and robotics, with more than 100 private tasks held for later release. The authors evaluated 12 agents; the best, Claude Opus 5.5, passed 58% of runs, and 16 tasks had no run pass at all; most runs that exceeded the reference on the overall final test score still fell short on some metric or workload. The authors say that on the shared capability scale of agents and tasks, the median task difficulty is roughly equivalent to the level the strongest agent can handle.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors propose FrontierCS 2, which tests whether LLM agents can make CS R&D contributions in a paper's original repository that surpass the authors' code.

    Core question: can LLM agents make computer science R&D contributions that surpass the paper authors' original code in real code repositories?

    • Setting and motivation: the authors believe much CS R&D takes the form: a researcher takes over a published system, improves one component in its codebase, and compares against the original authors' implementation on the workload reported in the paper. The authors want to test whether agents can do the same work.
    • Shortcomings of existing benchmarks (according to the paper): open-ended benchmarks such as FrontierCS, ALE-Bench, and AtCoder only give a problem statement, have the agent write a program, and score once at the end of the run; environment-based benchmarks such as SWE-bench and MLE-bench score by tests or leaderboards, and ResearchGym, MLS-Bench, PaperBench, etc. are limited to the machine learning domain, or score by rubric or by reproducing the implementation rather than the authors' original code.
    • Core design: take the code released by the paper authors, delete the paper's contribution, and the deleted code becomes the human reference, measured on the paper's own workload and used as the pass threshold; the agent works inside the repository like a researcher and can submit patches at any time and receive development suite scores.
    • Paper output: the FrontierCS 2 benchmark with 48 tasks covering 13 CS domains, plus a judge mechanism, the FCI/FDI capability–difficulty scale, and an evaluation of 12 agents and three case studies; more than 100 private tasks are yet to be released.
    • Evaluation target setting: the target of evaluation is the combination of "model + harness (agent framework)"; the paper names agents by model but states that the harness is also part of the target of evaluation; each agent runs each task once, with a five-hour time limit.

  2. What related research exists?

    Related work falls into three categories: open-ended coding benchmarks, research- and engineering-oriented agent benchmarks, and methods that generate tasks from existing artifacts.

    Open-ended coding benchmarks

    • FrontierCS, ALE-Bench, AtCoder heuristic contests—no known optimal solution, programs are scored by an objective function; the paper notes that agents on these benchmarks are still below top human experts, and the shared design is "given a problem statement, write a program, score once at the end".
    • LiveCodeBench Pro, HeuriGym, NP-Engine, KernelBench—covering contest problems with known solutions, combinatorial-optimization heuristics, verifiable NP problems, and GPU kernels, respectively, mentioned as related work.
    • nanoGPT speedrun, InferenceBench—speedup benchmarks start from a codebase and measure against a fixed baseline (a human speedrun record or a naive PyTorch inference server); the paper notes that FrontierCS 2 instead scores against the human reference code released by the authors and scores each patch during the run.

    Research and engineering agent benchmarks

    • SWE-bench, Terminal-Bench, MLE-bench, RE-Bench—scored by held-out tests, Kaggle leaderboards, and human experts with the same time budget, respectively; PostTrainBench measures downstream scores of post-trained models, DeployBench checks whether research artifacts can run a specified experiment, and EdgeBench also gives feedback during the run.
    • PaperBench, AIRS-Bench, ResearchGym, RSI-Forge, Into the Unknown, ReuF2F—covering paper reproduction, ML research, multi-domain paper tasks, etc.; the paper notes that none of these benchmarks use the original authors' code as the standard.
    • MLS-Bench—the paper calls it the closest in design: 140 ML tasks across 12 directions, each requiring improvement of a specified component; this paper extends that design to 13 CS domains beyond ML.

    Building tasks from existing artifacts

    • SWE-smith, FrontierSmith, BenchEvolver, DiscoGen—scaling task creation by injecting bugs, mutating contest problems, evolving solved problems, and programmatic generation, respectively.
    • ResearchGym—also removes the paper's method from the repository; the paper notes that the difference is that here the oracle is the removed code itself and the tasks span 13 domains.

    Positioning: the authors state that this paper differs from these works in that it evaluates in the paper's own repository, uses the authors' released code as the bar, spans all of computer science, and gives feedback on each patch during the run.

  3. How does the paper address this problem?

    Tasks are created by removing the paper's contribution; the judge privately holds the evaluator and reference code, and evaluation uses pass rate and the FCI/FDI scale.

    Overall approach: build tasks from the code released by the paper authors by "subtracting", using the deleted code as the human reference threshold, letting agents work in real repositories and be scored by an information-isolated judge. The framework is called FrontierCS 2.

    Task construction (Section 3)

    • Subtraction: locate the code implementing the paper's contribution and delete it, keeping the rest of the dependencies unchanged; the deleted functions are still declared and called, and the agent can only edit the file where the original contribution was located. The human reference is a private patch that restores the deleted code.
    • Pass thresholds: the threshold for each metric is set by the measured value of the reference code on the workload reported in the paper. For SVG-EAR, for example: latency no more than 0.96 times the reference, PSNR within 0.25 dB of the reference, SSIM within 0.01, LPIPS within 0.005.
    • Development suite and final suite: the judge scores each submission on the development suite; only the final submission is scored on the final suite, which adds held-out instances the agent has never seen (different seeds, queries, or input images).
    • Scale: 48 tasks, 13 domains (Database 9, Machine Learning 9, MLSys 6, HPC 4, Programming Languages 4, Architecture 3, Network 3, Cryptography 3, Robotics 2, Security 2, Bioinformatics 1, Software Engineering 1, Theory 1, Figure 1); Structured LWE is the only original cryptanalysis task not built from paper code (Appendix A).

    Judge and information isolation (Section 4)

    • Information partition: the agent holds the repository; the judge holds the evaluator, the held-out final suite, and the human reference. The agent has five hours and can submit patches at any time; the judge returns the development suite score and a message that only states on how many development workloads each metric's threshold is met, without giving reference scores or the evaluator's raw output.
    • Sandbox isolation: each side runs in a separate sandbox, connected only by the driver started for this run; the agent sandbox has no network other than the driver and cannot download the authors' code. In GPU tasks the agent can request development GPUs through the driver, and only the judge uses the GPU on which measurements against the reference are made; for some tasks (such as Continuum) the judge also has a test sandbox.
    • Pass determination: development suite scores do not count toward the result; at the end of the run the agent designates one patch as the final submission, and the judge evaluates only this patch on the held-out suite. The final submission passes only if it meets the threshold on every metric and every held-out workload; the judge separately gives a final score of 0–100 on each task's own scale, with no final submission recorded as 0.

    FCI and FDI (Section 5.2, Appendix D)

    • Joint fitting: FCI is adapted from Epoch AI's Epoch Capabilities Index (an item response model), taking two observations per run—which metric groups of the task exceed the reference, and the final score's margin relative to the reference yat = (sat − sreft)/ct (the difference between the final score and the reference score normalized by task units)—with both sharing one run effect.
    • Scale: one strength value per agent, FCI = 10 × (strength − Kimi K2.7's strength), i.e., 10 points per logit, with Kimi K2.7 anchored at 0; a task's FDI is the FCI at which its pass probability is 1/2. Intervals are 90% bootstrap intervals over resampled tasks (500 samples).
    • Fitting scale: 624 runs (including the unreported anchor agent Kimi K2.7), 115 metric groups, and the tie level of 48 tasks, totaling 163 task parameters (Appendix D).

    Test-time scaling (Section 5.3)

    • FCI under a budget: with task parameters fixed, runs completed within budget B keep their held-out results, and runs still running at B are recorded as having no final submission (0 points); agent strengths are refitted; run results can only improve with B, and the curve does not decline.

  4. What experiments did the paper conduct?

    12 agents each ran the 48 tasks once: Claude Opus 5.5 had the highest pass rate at 58%, and 16 tasks were passed by no one.

    实验设置

    • 被测智能体：12 个模型各配一个 harness——Claude Opus 5.5（Claude Code）、GPT-6 Astra（Codex CLI）、GPT-6.1 Sol（Codex CLI）、Gemini 4 Argon（Pi）、GLM 5.3（ZCode）、Muse Wonder 1.0（Muse Code）、DeepSeek V4.1 Flash（DeepSeek Harness）、Kimi K3（Kimi Code）、GLM 5.3 Flash（ZCode）、Muse Spark 1.3（Muse Code）、Qwen 3.8 Max（Qwen Code）、Step 5 Preview（Step Code）；另有 Kimi K2.7（Kimi Code）仅作 FCI 零点、不公开报告（Table 4）。
    • 任务与运行：48 个任务，每个智能体每任务跑一次，五小时时限；Gemini 4 Argon 有两次运行被服务商拒绝，按失败计入。
    • 指标：主指标 pass rate（通过运行占比）；另报告 mean score（最终分减参考分的均值，0 为持平）、FCI/FDI；Figure 5 给出每次运行平均成本。
    • 判定方式：judge 按各任务阈值在 held-out 最终套件上判定，非 LLM 评审；每个指标、每个 held-out 工作负载都达标才算 pass。
    • 重复次数：每任务一次运行，论文未报告重复实验。

    主结果

    Swipe horizontally to read the table

    智能体Pass rate（Figure 2）FCI（Figure 7）平均最终分边际（Figure 6）
    Claude Opus 5.55855.7+9.4
    GPT-6 Astra4450.7+7.1
    Gemini 4 Argon4039.5+2.1
    GPT-6.1 Sol3542.6+5.7
    GLM 5.32536.6+1.8
    Kimi K32126.4−1.2
    Muse Wonder 1.02134.5+4.1
    DeepSeek V4.1 Flash1530.7+0.5
    Muse Spark 1.31222.1−3.3
    Qwen 3.8 Max1221.9−6.4
    GLM 5.3 Flash1222.9−8.4
    Step 5 Preview89.8−10.6
    • 整体格局：作者称 Claude Opus 5.5 领先，其余智能体通过率均低于 50%；16 个任务没有任何运行通过。
    • 分数不等于通过：作者报告多数最终分超过参考的运行仍在至少一个指标或工作负载上不达标；也有通过的运行最终分持平或低于参考。
    • FCI 排序：FCI 与 pass rate 排序几乎一致；任务难度跨度大于智能体能力跨度，任务难度中位数约等于最强智能体的 FCI；作者说明单个任务的 FDI 不确定性大，不对任务逐个排序。

    测试时扩展（Section 5.3）

    • 领先者随预算变化：最小预算下 DeepSeek V4.1 Flash 领先，最大预算下 Claude Opus 5.5 领先（Figure 8）。
    • 成本口径：作者注明 DeepSeek V4.1 Flash 的成本是下界，其领先取决于 token 定价方式。

    案例研究（Section 6）

    • QuarterPIR：多数智能体通过；Muse Wonder 1.0 用两个校验和把 6700 万条目数据库的每次查询字节数从参考的 1,760 降到 1,280，其他通过运行最少为 1,664（Figure 9）；作者称 pass 和最终分都区分不出这一改进幅度，GPT-6.1 Sol 节省少得多却得分略高。
    • BBQ：仅 Claude Opus 5.5、Gemini 4 Argon、GPT-6.1 Sol 通过；Claude Opus 5.5 时钟周期最短但查找表用量在通过电路中最大，且不支持 BBQ 原设计的更大规模（Figure 10）；作者称最终分奖励速度、忽略电路面积，把它排第一。
    • SVG-EAR：无运行通过；GPT-6 Astra 最接近，每个 clip 都比参考快、平均 PSNR 23.8 dB 高于参考的 21.7 dB，仍在两个 clip 的图像质量上失败（Figure 11）；作者称多数运行在时间上达标，瓶颈在图像质量区间。

    其他消融与分析

    • 成本—通过率：Figure 5 显示 GPT-6.1 Sol 通过率低于 GPT-6 Astra 且每次运行成本低得多。
    • FCI 区间：Figure 7 的 90% bootstrap 区间只能分开能力差距大的智能体。
    • FDI 与通过率：作者报告 FDI 跨任务与通过率走势接近。

  5. What are some points that can be further explored?

    The authors point out that pass and final score ignore incidental costs and the magnitude of improvement; the experiments run each task only once.

    Limitations and future directions noted by the authors

    • Incidental costs not counted: the Conclusion notes that on BBQ the final score ranks the passing solution with the largest circuit first because it ignores circuit area; the authors believe such benchmarks need to report such incidental costs alongside pass.
    • Improvement magnitude not visible: the Conclusion notes that on QuarterPIR neither pass nor the final score can distinguish a run that greatly compresses the byte count from other passing runs.
    • Metrics may miss good ideas: Section 6.1 notes that this checksum idea was only found by manually reading the patch, and neither metric picked it out.
    • Task FDI uncertainty: Section 5.2 notes that the FDI of a single task is uncertain, so tasks are not ranked individually.
    • Private task set to be released: the Abstract and Conclusion state that there are more than 100 private tasks, which the authors plan to release in the future.

    Scope of the experiments

    • Agents tested: 12 publicly reported agents (model + harness combinations), plus Kimi K2.7 used only as the FCI zero point (Section 5.1, Table 4).
    • Task scale: 48 public tasks across 13 domains; one run per agent per task, with 624 runs entering the FCI fit (including Kimi K2.7, Appendix D).
    • Run protocol: five-hour time limit, with the number of development suite submissions capped per task (3–100, Table 3); the paper does not report repeated runs of the same agent on the same task.
    • Cost data: Figure 5 gives the average cost per run for each agent; the cost of DeepSeek V4.1 Flash is a lower bound (Section 5.3).
    • Interval estimation: FCI reports 90% bootstrap intervals (500 resamples), and FDI intervals use 2,000 weighted samples (Appendix D).

  6. Summarize the main content of the paper

    FrontierCS 2 evaluates 12 agents on 48 tasks that "remove a paper's contribution", with a highest pass rate of 58%.

    FrontierCS 2 is a computer science R&D benchmark that asks: can LLM agents make contributions that surpass the authors' original code in the paper's own code repository.

    • Problem: existing open-ended benchmarks only give a problem statement and score by running the program once, while environment-based benchmarks are limited to ML or scored by rubric; neither uses the authors' original code as the standard.
    • Method: the code implementing the paper's contribution is deleted as the human reference; agents can submit patches at any time within five hours and receive development suite scores; the judge privately holds the evaluator, the held-out final suite, and the reference, and the final submission must meet the threshold on every metric and every held-out workload to pass. The 48 tasks cover 13 domains.
    • Main results: 12 agents each ran once, and Claude Opus 5.5 had the highest pass rate at 58%, with all others below 50%; 16 tasks were passed by no run (Figure 2).
    • Scores decoupled from passing: most runs whose final score exceeded the reference still failed because some metric or workload did not meet the threshold (Section 5.1).
    • Capability scale: on FCI, Claude Opus 5.5 is 55.7 and Step 5 Preview is 9.8; the median task difficulty is roughly equal to the FCI of the strongest agent (Figure 7).
    • Authors' conclusion: case studies show that pass and the final score miss incidental costs (circuit area) and improvement magnitude (byte compression); the authors believe such benchmarks should report incidental costs and recommend scales with intervals such as FCI/FDI; more than 100 private tasks are planned for future release.
Read originalhuggingface.co