Establishing best practices for building rigorous agent benchmarks
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Quick read
Evaluation and benchmarksBased on the paper PDF (generated by Gemini); refer to the original
The authors propose the ABC checklist to evaluate 10 agent benchmarks, noting that 7 have flaws in outcome or task validity, and reduce performance overestimation by 33% in CVE-Bench.
- Agent benchmarks evaluate end-to-end tasks through environment interaction and tool use, but complex task configurations and unstructured outputs can easily introduce vulnerabilities. Existing benchmarks have defects in task design and reward determination, which the authors state may lead to underestimation or overestimation of agent capabilities by up to 100%.
- The authors propose the Agentic Benchmark Checklist (ABC), covering three dimensions: task validity, outcome validity, and benchmark reporting. By establishing specific checks for tool versions, environment isolation, test sufficiency, and statistical reporting, it helps developers and evaluators standardize benchmark construction and flaw detection.
- A review of 10 mainstream benchmarks found 7 violating task validity and 7 violating outcome validity, with all having reporting deficiencies. For example, a do-nothing agent in τ-bench achieved a 38% success rate on Airline, and applying ABC to CVE-Bench reduced the absolute overestimation of performance by 33%.
- The study only analyzed 17 benchmarks used by mainstream AI institutions from January 2024 to March 2025, not covering benchmarks outside this time window. The evaluation covered 10 open-source benchmarks, and the conclusions reflect the benchmark states at the time of writing; future improvements in agent capabilities may introduce evaluation challenges not covered here.
- Evaluated models
- UI-TARCy-AgentAutoGPTT-Agent
- Benchmarks
- SWE-benchSWE-LancerKernelBenchBIRDCybenchMLE-benchGAIAτ-benchWebArenaOSWorldCVE-Bench
- Metrics
- Pass Ratepass^kpass@kResolve RateAccuracySuccess@1Success@5Overestimation RateUnderestimation Rate
An arXiv paper analyzes 17 agent benchmarks and finds that flaws in task setup or reward design can cause agent performance to be overestimated or underestimated by up to 100% (relative). The authors accordingly propose the Agentic Benchmark Checklist (ABC), providing check items across task validity, outcome validity, and benchmark reporting. After evaluating 10 commonly used benchmarks with ABC, 7 have task validity issues, 7 have outcome validity issues, and all 10 have shortcomings in outcome reporting; for example, τ-bench counts empty responses as success, and SWE-Lancer allows agents to get full marks without solving the problem. As a case study, ABC reduces the performance overestimation of CVE-Bench by 33% (absolute).
Why it was selectedThe paper provides an actionable ABC checklist to help benchmark developers inspect agent evaluation design flaws across task validity, result validity, and reporting.
Related events and news
0 events · 1 news mentions
News that mentions this paper
- LessWrongcites the paper's id or link「约四分之一含缺陷」:Gemini 4 安全报告的 ML R&D 基准
for Building Rigorous Agentic Benchmarks”, NeurIPS 2025, arXiv:2507.02825 “We see that 38.3% of samples were flagged for underspecified problem statements, and 61.1% were flagged for unit tests that
Paper analysis
What problem does this paper attempt to solve?
The authors point out that agent benchmarks often undermine task and outcome validity due to flaws in environment interaction and unstructured output evaluation, leading to performance estimation errors of up to 100%.
The core problem is that existing agentic benchmarks have flaws in task design and reward design, leading to biased evaluations of agent capabilities.
- Problem scenario and importance: With the rapid development of AI agents capable of multi-step reasoning, tool use, and environment interaction, academia and industry have built numerous real-world task benchmarks to track progress and support decision-making; the authors believe it is crucial to ensure these evaluation numbers are trustworthy.
- Shortcomings of existing methods: Traditional AI benchmarks are mostly single-turn multiple-choice questions or simple text generation, whereas agent benchmarks require running environments and receive unstructured outputs (such as code, file modifications, environment states), making evaluation logic complex; existing benchmarks often suffer from incomplete test cases, string-matching oversights, or insufficient isolation, leading to performance being overestimated or underestimated by up to 100% in relative terms (e.g., SWE-bench-Verified has missing test edge cases, and τ-bench judges empty responses as successful).
- Core observations and conditions: The authors note that rigorous agent evaluation must simultaneously satisfy two core conditions: task validity (target capability is equivalent to task success, excluding shortcuts and unsolvable tasks) and outcome validity (task success is equivalent to a positive evaluation result, meaning the evaluation accurately reflects true success).
- Proposed solution in the paper: Based on benchmark-building experience, literature review, and known flaws, the paper proposes the Agentic Benchmark Checklist (ABC), reviews 10 popular benchmarks, and demonstrates improvement effects in CVE-Bench.
What related research exists?
The paper reviews three categories of research: AI benchmark evaluation systems, agent benchmark construction, and evaluation flaws, noting that previously there was no systematic, actionable checklist for agent benchmarks.
The paper divides related research into three groups and compares benchmark selection and methodological positioning:
Assessing AI Benchmarks
- Studies such as Dorner and Hardt (2024, 2025) evaluate the label quality and quantity, evaluation protocols, data contamination, and reproducibility of AI benchmarks; studies such as Tsipras et al. (2020) explore data bias and label noise in traditional benchmarks like ImageNet.
- Cao et al. (2025) and Reuel et al. (BetterBench, 2024) propose best-practice guidelines for building general or code-specific benchmarks; the authors note that these studies mainly target multiple-choice or generation tasks that do not require multi-step reasoning, and have limitations when facing complex agent tasks.
Benchmarking of AI Agents
- Code generation benchmarks: Jimenez et al. (SWE-bench, 2024), Miserendino et al. (SWE-Lancer, 2025), Ouyang et al. (KernelBench, 2025), Li et al. (BIRD, 2023), etc., simulate real software engineering, GPU kernel writing, and database SQL generation.
- Environment interaction and multi-step reasoning benchmarks: Yao et al. (τ-bench, 2025), Zhou et al. (WebArena, 2024), Xie et al. (OSWorld, 2024), Mialon et al. (GAIA, 2023), Chan et al. (MLE-bench, 2024), Zhang et al. (Cybench, 2024), etc., evaluate agents' end-to-end execution capabilities in web operations, operating systems, mathematics, or cybersecurity.
Issues in Evaluating AI Agents
- Outcome validity flaws: Kydlíček and Gandenberger (2025) found that implicit assumptions about answer format led to a 5.3% underestimation of performance; Yu et al. (UTBoost, 2025) found that 7.7% of tasks in SWE-bench-Lite and 5.2% in SWE-bench-Verified had incorrect patches passing tests; Pourreza and Rafiei (2023) and Wretblad et al. (2024) noted that annotation noise in BIRD affects evaluation accuracy.
- Task validity flaws: Kapoor et al. (2024) noted that rate limits on websites in WebArena hinder agents from solving tasks; Lange et al. (2025) found flaws in KernelBench that allow agents to bypass correctness checks.
Benchmark methodology and evaluation targets
- The paper systematically collected 17 agent benchmarks used by top AI institutions (OpenAI, Anthropic, Amazon, Meta, Google, xAI, Mistral, DeepSeek) from January 2024 to March 2025, selecting 10 open-source benchmarks for in-depth evaluation (Table 1 and Table 3).
Differentiation from existing work
- The authors note that existing work only discusses localized flaws in individual agent benchmarks, and no prior study has established a systematic, actionable set of check criteria to comprehensively guide the design and review of agent benchmarks.
How does the paper address this problem?
The paper proposes the ABC checklist, which standardizes evaluation processes and defect mitigation across three dimensions: task validity (10 items), result validity (16 items), and benchmark reporting (13 items).
The paper formally decomposes the agent evaluation process and proposes the Agentic Benchmark Checklist (ABC).
Formal decomposition of the evaluation process and core principles
At the operational level, an agent benchmark provides the agent with task descriptions, an interaction environment, and tools; the agent produces unstructured outputs such as text, code, or environment modifications; the benchmark determines success using string matching, program tests, or state matching (Figure 1). The authors argue that rigorous evaluation must satisfy two conditions:
- Task validity: The target capability is equivalent to task success, ensuring that tasks have neither shortcuts that bypass the target capability nor unsolvable dead ends.
- Outcome validity: Task success is equivalent to a positive evaluation result, ensuring that the evaluation judgment truly reflects task completion status.
Task validity checks
ABC establishes 10 specific checks across tools, environment, and implementation (Figure 2):
- Tool level (T.1–T.3): Explicitly specify versions of self-built tools or dependency packages (T.1); ensure availability and quota management of API tools (T.2); properly terminate or report errors when APIs are interrupted (T.3).
- Environment level (T.4–T.6): Thoroughly clean residual data and state between runs (T.4); ensure complete isolation between the agent and ground truth (T.5); fully freeze static environments upon release, avoiding dependence on uncontrollable dynamic external websites (T.6).
- Implementation level (T.7–T.10): Verify the correctness and solvability of annotated ground truth and task configurations (T.7–T.8); provide an Oracle solver for automatic problem solving (T.9); inspect code implementation flaws to prevent unintentional leakage or bypass (T.10).
Outcome validity checks
ABC establishes 16 checks categorized by task output type (Figure 3):
- Information retrieval (O.a.1–O.c.2): Full-string and substring matching should support semantically equivalent expressions and tolerate redundant words; substring matching needs to handle negating modifiers (such as not) and guard against enumerating all possible answers or random guessing; LLM judges need pilot experiments to verify their accuracy, self-consistency, and human agreement, and should be designed to resist adversarial cheating.
- Code generation (O.d.1–O.f.2): Unit tests need manual verification and objective metrics such as coverage to ensure test sufficiency; fuzzing needs to cover boundary values, data types, and memory layouts, and test inputs must reach key branches; end-to-end tests need to cover all branches of user workflows and eliminate nondeterminism (flaky tests).
- State modification and reasoning (O.g.1–O.i.1): State matching needs to include all legal states after success and check both relevant and irrelevant states to detect unintended changes; multi-step reasoning needs explicit format requirements, and the chosen scoring metric must be strongly correlated with the reasoning process to prevent metric gaming.
Benchmark reporting standards
When evaluation flaws cannot be fully eliminated, ABC proposes 13 transparency and reporting guidelines (Figure 4):
- Transparency and contamination prevention (R.1–R.6): Open-source datasets and evaluation toolchains, keep private test sets, establish continuous update mechanisms, and clearly define the evaluation target and construct.
- Defect quantification and analysis (R.7–R.9): Publicize prevention and correction processes, and qualitatively and quantitatively analyze the impact of unavoidable flaws (such as annotation noise). For example, under a normal assumption, if the ground-truth error rate is e and the observed success rate is p_0, the corrected mean true success rate is μ = e + (1 − 2e)p0, with variance σ2 = μ(1 − μ), from which confidence intervals are derived.
- Result interpretation and comparison (R.10–R.13): Report statistical significance metrics such as confidence intervals, provide interpretation guides, and report human expert and trivial baselines such as do-nothing results.
What experiments did the paper conduct?
The authors used ABC to evaluate 10 mainstream benchmarks and quantified new flaws; fixing two evaluation flaws in CVE-Bench reduced the absolute overestimation of performance by 33%.
Experimental Setup
- Benchmarks under test: 10 open-source agent benchmarks are selected, including SWE-bench, SWE-Lancer, KernelBench, BIRD, Cybench, MLE-bench, GAIA, τ-bench, WebArena, OSWorld, and a deep-fix case study on CVE-Bench (Table 1).
- Models and agents under test: Includes open-source agent UI-TAR (OSWorld), Cy-Agent, AutoGPT, T-Agent (CVE-Bench), o3-mini for auxiliary CUDA test case generation, and constructed Do-nothing and Spamming agents.
- Evaluation process: Apply ABC checklist items one by one (1 point if satisfied, 0 if not), calculate the average score of applicable items for each major category (Figure 5), and write dedicated verification programs for quantitative testing of identified defects.
- Hardware environment: Case study experiments are run on a server with Intel E5-2630 CPU and 128 GB RAM; GPU-related tests use 1 NVIDIA H100 80GB GPU (Appendix E).
- Sampling and review: In KernelBench experiments, 3 generated kernels are sampled for each Level 1 task, using o3-mini to identify vulnerabilities and human verification of test case correctness, then applied to the full set of generated kernels from existing research; OSWorld's Chrome section is checked one by one for all 46 tasks.
Main Results
The table below presents representative validity vulnerabilities identified by the authors in each benchmark and the magnitude of measured performance deviation (according to Sections 5.2 and 5.3):
Swipe horizontally to read the table
Benchmark Violated checklist items Vulnerability type and trigger mechanism Measured deviation magnitude Source τ-bench O.b.3, O.g.3 Contains unsolvable tasks and empty responses are considered success; do-nothing passes Performance overestimated by 38% Section 5.2 τ-bench O.b.2 Substring matching without anti-enumeration mechanism; outputting all data passes Performance overestimated by 40% Section 5.2 WebArena O.b.2, O.c.1 Substring ignores redundant content; LLM judge marks empty responses as correct Performance overestimated by 1.4%–5.2% Section 5.2 SWE-Lancer T.5, T.10 Test files not isolated; overwriting assertions in zip can achieve full score Solution rate inflated to 100% Section 5.2 KernelBench O.e.1, O.e.2 Fuzzer misses edge cases in input shapes and memory layouts Kernel correctness overestimated by about 31% Section 5.2 OSWorld T.6 External webpage updates cause Chrome task selectors to fail UI-TAR performance underestimated by 28% Section 5.2 CVE-Bench O.g.1, T.9 Time-based blind injection only checks for SLEEP in logs; outbound services not isolated Performance overestimation reduced by 33% Section 5.3 - Task validity is generally lacking: The authors state that more than half of the tested benchmarks have implementation defects, especially in scenarios where interactive tools are provided to agents; lack of environment isolation and Oracle solvers can lead to shortcut passes or unsolvable tasks (Section 5.1).
- Result validity is hard to guarantee: The authors state that more than half of the benchmarks fail to address inherent limitations of evaluation mechanisms, such as incomplete test cases, substring matching not handling enumeration cheating, or unreliable LLM judges, leading to significant overestimation of agent capabilities (Section 5.1).
- Benchmark reporting generally lacks transparency: The authors state that 80% of benchmarks do not acknowledge weaknesses in their own design or implementation in reports, and no benchmark fully satisfies all reporting criteria (Section 5.1).
Deep Experiments on Representative Benchmark Vulnerabilities
- Shortcut tests for τ-bench and SWE-Lancer: Tested do-nothing and code overwriting; in τ-bench, an immediately-returning Do-nothing agent achieves 38% and 6.0% pass^k or pass@k on Airline and Retail subsets, respectively, while a Spamming agent that outputs all data achieves 40% and 9.6%; in SWE-Lancer, the agent exploits the lack of write protection on test zip files to replace assertions with assert 1==1, achieving a 100% solution rate; the authors state this shows that lack of environment isolation and anti-guessing design severely inflates evaluation scores (Appendix E.2, E.4).
- KernelBench fuzzing defect experiment: Tested the impact of tensor dimension and memory layout changes on CUDA kernel correctness; used o3-mini to generate targeted test cases for Level 1 tasks with human review, then applied them to kernels generated by existing research; results show that the original tests only changed values without changing tensor shapes and memory layouts, leading to kernel correctness overestimated by about 31%; the authors state this shows simple fuzzing cannot effectively ensure the soundness of unstructured code evaluation (Appendix E.6).
- OSWorld environment drift test: Tested the impact of external website updates on HTML selector-based evaluation; in OSWorld's Chrome section, 13 of 46 tasks fail due to changes in external website layout, URL, or functionality, causing UI-TAR model performance to be underestimated by 28% in absolute terms; the authors state this confirms that benchmark environments must be frozen at release and cannot rely on continuously updated live external resources (Section 5.2, Appendix E.5).
CVE-Bench Improvement Case Study
- Time-based blind injection evaluation fix: CVE-Bench originally determined SQL injection success by checking whether the database log contains SLEEP clauses, but the presence of SLEEP in logs does not equal actual execution; after the authors corrected the state matching logic according to ABC item O.g.1, they eliminated 32.5% of the performance overestimation (Section 5.3).
- Uncontrolled outbound server fix: CVE-Bench detects whether the victim application requests a violating outbound server, but internal communication within the same Docker network is unrestricted, allowing agents to directly access the outbound server; after the authors simulated execution according to T.9 to discover this shortcut and block direct external requests, the agent success rate dropped by 10% (Figure 6b); the authors state that the two fixes together reduce CVE-Bench's overall performance overestimation by 33% in absolute terms (Section 5.3).
Other Ablations and Analyses
- SWE-bench test insufficiency analysis: The proportion of tests passed without solving the actual problem is 5.3% on the Verified subset and 7.7% on the Lite subset, causing Leaderboard ranking changes of 40.9% and 24.4%, respectively, and performance overestimations of 2.3% and 1.6% (Appendix E.1).
- WebArena LLM judge bias: For tasks labeled N/A, the LLM judge marks empty responses as correct, and does not verify judge self-consistency or human agreement, leading to 1.4%–5.2% performance overestimation (Section 5.2).
- BIRD annotation noise example: Random sampling of 500 tasks for verification reveals that 11.65% of ground-truth SQL have errors; SQL experts achieve a 92.96% success rate on this benchmark, while a pure do-nothing agent achieves 0% (Appendix F).
What are some points that can be further explored?
The authors note limitations including the restricted analysis time window, lack of coverage of future frontier capabilities, and benchmark version evolution, requiring continued expansion of the checklist and evaluation framework.
Limitations and future directions noted by the authors
- Limited time window covered by the benchmarks: The analysis only covered 17 agent benchmarks used by mainstream AI institutions between January 2024 and March 2025, and did not analyze benchmarks outside this time window; the authors note that the checklist may not exhaust all relevant evaluation practices (Appendix A).
- Cognitive limitations based on current AI capabilities: The taxonomy and analysis are built on current understanding of AI agent reasoning capabilities; the authors note that future AI developments may introduce more advanced capabilities, potentially bringing new evaluation challenges not covered by this study (Appendix A).
- Timeliness of conclusions due to benchmark version evolution: The evaluation findings only reflect the specific version states of the relevant benchmarks at the time of writing; the authors note that future version revisions of these benchmarks may produce different results, and the conclusions may not fully apply to subsequent versions (Appendix A).
- Plans for continued maintenance and open-source community collaboration: The authors plan to continuously track and update evaluation practices and checklists for agent benchmarks through open-source code and a dedicated website (Section 3).
Experimental coverage
- Number and types of benchmarks tested: The in-depth review covered 10 open-source agent benchmarks (SWE-bench, SWE-Lancer, KernelBench, BIRD, Cybench, MLE-bench, GAIA, τ-bench, WebArena, OSWorld), and comparative experiments before and after modification were conducted on CVE-Bench (Table 1, Section 5.3).
- Model and agent testing scope: The models and agents actually tested include UI-TAR (OSWorld), Cy-Agent, AutoGPT, T-Agent (CVE-Bench), o3-mini for generating CUDA test cases, and Do-nothing and Spamming agents constructed for specific benchmarks.
- Computing resource environment: Case study experiments were uniformly based on a single-node Intel E5-2630 CPU with 128 GB RAM, and GPU experiments were equipped with 1 NVIDIA H100 80GB GPU (Appendix E).
- Evaluation details not reported in the paper: The paper did not report inter-annotator agreement statistics when scoring the 10 benchmarks against the checklist; for the CVE-Bench tests, the paper did not report the specific number of queries and token consumption for each agent during the exploit process.
Summarize the main content of the paper
The paper proposes the ABC checklist for standardizing agent benchmarks, points out validity flaws in mainstream benchmarks, and validates the improvement effects in CVE-Bench.
This paper is a systematic review and standardization study of the rigor and validity issues in AI agent benchmark evaluation.
- Problem addressed: As AI agents shift from traditional single-turn multiple-choice tasks to complex multi-step environment interactions and unstructured outputs, existing benchmarks face challenges in task design and reward verification, which can easily lead to relative performance estimation errors of up to 100% due to shortcuts or dead ends.
- Method highlights: The authors propose the Agentic Benchmark Checklist (ABC), decomposing evaluation rigor into task validity (target capability is equivalent to task success, covering 10 checks across tools, environment isolation, and implementation), outcome validity (task success is equivalent to positive judgment, covering 16 checks for information retrieval, code generation, state modification, and multi-step reasoning), and benchmark reporting standards (including 13 checks for transparency, defect quantification, and benchmark comparison).
- Key experimental findings:
- Among the 10 open-source mainstream benchmarks reviewed, the authors found 7 violating task validity, 7 violating outcome validity, and all 10 benchmarks had limitations in reporting transparency and defect quantification.
- In τ-bench, a do-nothing agent achieved a 38% success rate on the Airline subset, while an agent that spammed output data achieved a 40% success rate; in SWE-Lancer, an agent that overwrote the test zip file directly achieved a 100% resolution rate.
- In KernelBench, fuzzing that missed tensor shapes and memory layouts led to CUDA kernel correctness being overestimated by about 31%; in OSWorld, webpage element changes invalidated HTML selectors for Chrome tasks, causing UI-TAR performance to be underestimated by 28% in absolute terms.
- Applying ABC to CVE-Bench, fixing time-based blind injection and outbound network isolation flaws reduced the performance overestimation by 33% in absolute terms.
- Author conclusions and implications: The authors believe that absolute scores and rankings on current agent leaderboards cannot yet be fully trusted; benchmark developers need to introduce rigorous environment state cleanup, isolation mechanisms, and comprehensive fuzzing during construction; when evaluation flaws cannot be completely avoided, confidence intervals and trivial baseline results should be proactively reported to maintain evaluation transparency.