Δτ-Bench: Pushing agents toward an ever-changing capability frontier

1. 01What problem does this paper attempt to solve?

   When the tool ecosystem keeps evolving, agent experience lags behind; the authors state that existing benchmarks do not examine this continuous adaptation, so they build ∆τ-Bench and propose Steward.

   Core problem: when a large tool ecosystem keeps evolving (tools retired, replaced, overlapping, documentation noisy), can an agent proactively discover more efficient tool-use paths and update its procedural knowledge in time? Existing benchmarks cannot answer this question.

   • Problem setting: the authors envision a large company using an agent as a unified service entry point, with each department independently maintaining tools behind it; overlapping functionality and inaccurate descriptions make the catalog hard to navigate, and updates constantly change feasible paths (Figure 1).
   • Why it matters: retiring old tools invalidates established workflows, and new tools may bring more efficient paths; the authors argue that task success itself may mask the gap between what the agent has learned and the ecosystem's actual capabilities.
   • Shortcomings of existing benchmarks: the authors state that existing benchmarks separately examine large-scale tool retrieval, stateful execution, cost-aware replanning, or robustness to tool evolution, but most keep the service structure fixed or restrict tool discovery when services change, evaluating these capabilities in isolation.
   • Core observation of the paper: to measure adaptation, agent behavior must be compared with the optimal path the current ecosystem allows for the same task; on this basis the authors propose the "moving capability frontier" evaluation perspective.
   • What the paper proposes: first, ∆τ-Bench, containing 42 domains, 1,588 validated multi-turn tasks, and 6 consecutive ecosystem states generated by capability-preserving transformations; second, Steward, an ecosystem manager that jointly manages tools and procedural knowledge, performing incremental skill maintenance and dynamic supply during execution.

   Collapse this answer

2. 02What related research exists?

   Related work falls into two categories, tool-use benchmarks and skill learning; the authors state that the former examines capabilities in isolation and the latter assumes a stable operational substrate.

   Tool-use agent benchmarks

   • Early API and composition benchmarks: Gorilla (Patil et al., 2024), ToolLLM (Qin et al., 2024), API-Bank (Li et al., 2023), ToolAlpaca (Tang et al., 2023), BFCL (Patil et al., 2025), T-Eval (Chen et al., 2024b), ACEBench (Chen et al., 2025a), etc., examine API selection, parameter generation, multi-tool composition, and planning.
   • Stateful multi-turn benchmarks: τ-bench (Yao et al., 2024), τ²-bench (Barrès et al., 2025), AppWorld (Trivedi et al., 2024), ToolSandbox (Lu et al., 2025), Tool Decathlon (Li et al., 2025) evaluate multi-turn interactions with persistent tool effects.
   • Retrieval, efficiency, and evolution robustness: Large-scale tool retrieval (Xu et al., 2024; Shi et al., 2025; Qin et al., 2025a, etc.); cost and replanning (PlanBench, SayCanPay, ACPBench, TravelPlanner, CostBench, etc.); API perturbation and MCP mutation (Chen et al., 2025b; MCPEvol-Bench, Liu et al., 2026c). The authors state that these capabilities are still evaluated in isolation, and that the end-to-end tool-use lifecycle in evolving ecosystems has not yet been covered.

   Skill learning and evolution

   • Experience distillation and executable skills: ExpeL (Zhao et al., 2024), AutoGuide (Fu et al., 2024), Voyager (Wang et al., 2024), Agent Workflow Memory (Wang et al., 2025b), etc., distill trajectories into natural-language guidance or reusable workflows.
   • Automated skill construction and optimization: AutoSkill (Yang et al., 2026b), Trace2Skill (Ni et al., 2026), EvoSkill (Alzubi et al., 2026), SkillOpt (Yang et al., 2026a), etc., automatically construct skills through trajectory consolidation, optimization, and validation; others such as SkillX (Wang et al., 2026) perform hierarchical and lifecycle management. The authors state that most of these methods evolve skills on a stable operating substrate and emphasize final task success.

   Baselines and benchmark: The baselines for the skill method comparison are No Skill, EvoSkill, AutoSkill, Trace2Skill, and SkillOpt (Appendix B gives each mechanism), all using DeepSeek-V3.2 as the backbone; the benchmark used is the paper's own ∆τ-Bench.

   Authors' positioning: They describe ∆τ-Bench as a benchmark that unifies large-scale tool retrieval, multi-turn stateful execution, and efficiency-aware adaptation; Steward reframes skill evolution as ecosystem-level adaptation, managing tools and procedural knowledge in a coupled manner.

   Collapse this answer

3. 03How does the paper address this problem?

   ∆τ-Bench uses capability-preserving transformations to generate 6 ecosystem versions and uses dynamic programming to find the optimal reference path; Steward incrementally maintains a skill library and dynamically supplies tools and skills.

   The overall approach has two parts: first build ∆τ-Bench, which simulates ecosystem evolution in a stateful, multi-turn executable environment and measures agents against optimal reference paths; then propose Steward, which jointly manages tools and skills as coupled resources.

   Task Formalization

   • Interaction is modeled as a partially observable sequential decision process M = (U, S, A, O, P, R): the user intent u is not directly observable, the state includes external environment state and interaction state, and actions include tool calls a = (f, x) and natural-language messages.
   • Each episode has an exogenous ecosystem version v, fixed throughout execution; the action space varies with v, and the same intent has different feasible paths in different versions.

   Executable Environment and Task Construction (§3.1, Appendix A.1)

   • Domain and service blueprint: seed domains are expanded into role-specific scenario groups, entity schemas and entity-relationship graphs are built, and these are deterministically converted into tool interfaces with natural-language descriptions and JSON Schema.
   • Stateful MCP services: entity records are instantiated in a simulated database and implemented as Python functions according to operation policies (preconditions, parameter semantics, state changes, dependencies), returning results or errors in a declared format.
   • Task synthesis and execution validation: paths are sampled from the tool dependency graph (biased toward query-type starting points), and difficulty is increased through shared-function splicing and subpath repetition; after task templates are generated, an independent evaluator filters them by motivation clarity, coherence, function coverage, naturalness, and specificity; after instantiation they are replayed on the MCP services, keeping only instances where every step succeeds, yielding reference trajectories τgold and terminal states.
   • User simulation: an independent LLM user agent holds a private manual (full intent, motivation, disclosable rules), discloses information progressively, and does not mention functions or execution plans.

   Evolving Ecosystem Construction (§3.2, Appendix A.2)

   • Three types of capability-preserving transformations: one-to-one (rename, add parameter, change format), one-to-many (split into coordinated interfaces), many-to-one (consolidate into a unified interface, where source blocks must be consecutive subsequences of validated trajectories in the same domain); an LLM proposes target implementations, and they are kept only after automatic verification of capability, state-change, and output equivalence.
   • Ecosystem sampling: each version applies a bounded number of transformations to the previous version; source interfaces may be kept to form overlap, or deactivated to force use of the new implementation; recursive validation checks that all base capabilities remain accessible, otherwise the transition is rejected; optionally, some function descriptions are perturbed to simulate incomplete or outdated documentation.
   • Reference path search: the task's base function sequence is partitioned into consecutive source blocks, one active implementation is chosen per block, ρ_i^(v,c) = argmin ∑_j=1^m c^(v)(R_j), i.e., minimizing the total criterion value over all implementation combinations covering the function sequence; dynamic programming is used to find the minimum-step and minimum-cost paths separately (cost is scored 1–10 by an LLM based on tool definitions and implementations), and each is validated by replay in the corresponding MCP environment.

   Evaluation Protocol (§3.3)

   • SR: a task succeeds if and only if the database changes produced by the agent cover all changes required by the gold execution, S_i^(v) = 𝕀[Δ_i^gold ⊆ Δ_i^(v)].
   • Tool retrieval P/R/F1: compares the tool set exposed to the agent after retrieval with the reference-path tool set.
   • SPE: success-weighted relative path efficiency, yielding Step-/Cost-SPE by steps and cost respectively.
   • IDR/DAR: IDR (discovering better paths) is measured on the task set whose reference weight decreases between adjacent versions, and DAR (adapting to degradation) on the task set whose weight increases.

   Steward (§5, Appendix C)

   • Ecosystem manager: maintains shared resource state (current tool catalog + persistent skill library), builds a task view for the task agent (visible tool subset + procedural context), and the task agent acts on this basis.
   • Three-level skill library: planning skills (task-level workflows), atomic skills (reusable operations and their tool dependencies), and tool skills (key points for single tool calls); v1 is initialized following Wang et al. (2026), after which it is maintained incrementally: validated successful training trajectories are distilled into local edits, skills are marked active/inactive by tool availability, and inactive skills are retained for reactivation.
   • Initial supply: the top 5 retrieved planning skills are rewritten into a temporary plan; at each step at most 5 atomic skills are retrieved (similarity threshold + MMR deduplication) and the LLM selects at most 8; at each step the top 3 tools are retrieved directly, unioned with skill dependencies, and intersected with the current catalog.
   • On-demand supply: the task agent can call retrieve_tools (at most 5 tools) or explore_capabilities (at most 3 delegations per episode); the manager searches the skill library and the full catalog, checks schemas, and returns a report, with at most 8 tools returned per investigation and a cumulative cap of 4,000 characters for skill text. Appendix D gives all prompts.

   Collapse this answer

4. 04What experiments did the paper conduct?

   Among the 9 models/harnesses, the highest SR is only 42.27%; the gains of the four skill methods are unstable; Steward has an average SR of 39.04% and lower maintenance cost.

   Experimental Setup

   • Models evaluated (Table 1): proprietary models Codex-GPT-5.5, Gemini-3.6-Flash, GPT-5.5, Claude-Sonnet-5; open-source models DeepSeek-V4-Pro, GLM-5.2, Qwen-3.8-Max, Kimi-K3, GPT-oss-120B. Among these, Codex-GPT-5.5 is a harness.
   • Benchmark: ∆τ-Bench, 1,588 tasks (1,191 training, 397 test), 42 domains, 6 ecosystem versions totaling 9,528 task-ecosystem instances; the number of tools per version is 799–892 (Table 3).
   • Inference settings: models with configurable reasoning effort are set to high, and other parameters use recommended defaults; the user model is fixed to DeepSeek-V3.2.
   • Skill method comparison: No Skill, EvoSkill, AutoSkill, Trace2Skill, SkillOpt, and Steward, all using DeepSeek-V3.2 as both task model and user model, adapted directly from official implementations.
   • Metrics: SR, tool retrieval P/R/F1, Step-/Cost-SPE, Step-/Cost-IDR/DAR, skill maintenance tokens (K/task); success is determined by coverage of database state changes (Equation 4), with no human scoring.
   • Number of repetitions: not reported in the paper.

   Main Results

   Table 1: Performance of each model on ∆τ-Bench (metric units %).

   Swipe horizontally to read the table

   Model	SR	Retrieval P	Retrieval R	Retrieval F1	Step-SPE	Cost-SPE
   Codex-GPT-5.5	42.27	19.90	90.36	32.04	26.93	27.90
   Gemini-3.6-Flash	41.65	19.30	87.51	31.13	25.49	25.88
   GPT-5.5	31.74	20.21	81.08	31.86	23.11	23.42
   Claude-Sonnet-5	29.97	19.36	76.93	30.41	20.48	20.60
   DeepSeek-V4-Pro	33.29	18.65	79.00	29.65	13.49	14.31
   GLM-5.2	32.28	19.32	78.25	30.47	19.39	19.84
   Qwen-3.8-Max	30.10	18.46	80.02	29.49	15.84	16.84
   Kimi-K3	28.30	19.20	77.01	30.20	17.78	17.83
   GPT-oss-120B	27.00	18.24	69.98	28.40	16.87	17.16

   • The authors state that even the strongest model (Gemini-3.6-Flash, 41.65%) and harness (Codex-GPT-5.5, 42.27%) reach only just over 40% SR, and Step-/Cost-SPE do not exceed 30%.
   • Retrieval shows "high recall, low precision": the authors state that models can often include relevant tools but struggle to exclude irrelevant candidates from the ecosystem.
   • Success does not equal efficiency: DeepSeek-V4-Pro's SR is slightly higher than GLM-5.2's, but its Step-/Cost-SPE and SPE/SR are lower.

   Tool Supply Diagnostics (Figure 3)

   • What was tested: using DeepSeek-V3.2, five supply conditions were compared: Standard, Domain Tools, Task Tools, Oracle Tools, and Oracle Path.
   • Results: in the figure, SR and the two SPEs rise progressively from Standard through Domain and Task Tools to Oracle Tools, with no further improvement under Oracle Path; the SPE/SR increase from Task Tools to Oracle Tools is notable (the figure's vertical axis is marked with ticks from 10 to 70; the main text gives no per-condition values).
   • Authors' interpretation: models struggle to distinguish the current version's efficient options from unavailable or overlapping candidates; the remaining gap lies more in using tools correctly and efficiently than in high-level planning.

   Cross-Version Comparison of Skill Methods (Figure 4)

   • What was tested: SR, SPE, IDR, DAR, and token consumption of four automated skill methods and No Skill across six versions.
   • Results: on v1, all four methods have higher SR and SPE than No Skill; AutoSkill's SR on v3 and v5 is lower than No Skill's; on v3, Step-/Cost-DAR for EvoSkill and AutoSkill drops to 0, while No Skill is 11.11%/10.25%; No Skill achieves the highest Cost-IDR on v2 and the highest values of both DARs on v5; SkillOpt leads on SR and SPE but its maintenance cost is about 3,000 K/task, roughly two to three times that of other methods.
   • Authors' interpretation: generating and optimizing skills in isolation is insufficient to keep tracking the frontier; tools and skills need joint maintenance.

   Steward Main Results (Figure 6)

   • SR: Steward has the highest SR on all six versions, averaging 39.04%, versus 36.36% for SkillOpt (DeepSeek-V3.2).
   • Maintenance cost: Steward is lowest from v2 onward, with baselines consuming 1.2–5.5× the maintenance tokens (Figure 6(b)).
   • Percentage-point gains over the best baseline (Figure 6(c) main text): Step-/Cost-IDR average +4.04/+3.69, Step-/Cost-DAR average +5.28/+4.76, Step-/Cost-SPE average +2.68/+4.21, and tool P/R/F1 also show consistent gains.

   Other Ablations and Analyses

   • Cross-version knowledge maintenance (Table 2): incremental maintenance has the highest SR on both DeepSeek-V3.2 and GPT-5.5 (36.83%, 40.81%); compared with full rebuilding, SR is 2.22/1.82 percentage points higher and maintenance tokens are 39.5%/30.4% lower; on GPT-5.5, full rebuilding (38.99%) is even lower than a frozen library (39.60%).
   • Capability supply ablation (Figure 7, DeepSeek-V3.2): skill search alone yields almost no improvement on any metric; tool search alone gives SR +8.65 and Tool F1 +4.14 percentage points, but Step-DAR drops by 2.76; full supply further raises Step-/Cost-DAR by 11.41/9.30 percentage points and Tool F1 by 4.90 relative to tool search alone.
   • Reference path drift (Figure 8): reference weights rise and fall between adjacent versions, non-monotonically; Table 3 gives the number of added/retired/modified interfaces per version (e.g., v2 adds 110, retires 161, modifies 176).

   Collapse this answer

5. 05What are some points that can be further explored?

   The paper does not specifically discuss limitations; experimentally, the skill comparison uses only a single task model, cost scores rely on LLM scoring, and the number of repetitions is not reported.

   Limitations and future directions noted by the authors

   The paper does not specifically discuss limitations (there is no Limitations or Future Work subsection; nor do the conclusion and ethics statement contain sentences explicitly saying "this is a limitation / this could be done in the future").

   Scope of experimental coverage

   • Models tested: The main table (Table 1) covers 9 models or harnesses; the cross-version comparison of the skill method and Steward (Figure 4, 6) uses only DeepSeek-V3.2 as the task model; the cross-version maintenance ablation (Table 2) uses two models, DeepSeek-V3.2 and GPT-5.2; the capability provisioning ablation (Figure 7) uses only DeepSeek-V3.2.
   • Benchmark scope: All experiments are conducted on ∆τ-Bench, with 42 domains, 1,588 tasks, and 6 ecosystem versions; the domain distribution is shown in Table 4 (e.g., MedicalDeviceEngineeringServer has 112 tasks, and some domains have fewer than 10).
   • User simulation: The user model in all experiments is fixed as DeepSeek-V3.2 (§4 Evaluation Setup).
   • Metric construction: Execution cost is scored 1–10 by an LLM according to tool definitions and implementations (§3.2); task templates are filtered by an independent evaluator according to five criteria, and the paper does not report the model used by this evaluator or human agreement.
   • Not reported in the paper: The number of repetitions and statistical significance tests for each experiment; the specific iteration budget of each method in the skill method comparison (only SkillOpt's maintenance tokens of about 3,000 K/task are reported).

   Collapse this answer

6. 06Summarize the main content of the paper

   ∆τ-Bench measures continuous adaptation in evolving tool ecosystems; existing models and skill methods cannot keep up with the frontier, and Steward, by jointly managing tools and skills, achieves higher SR and lower maintenance cost.

   This is a benchmark-plus-method paper: the authors first build ∆τ-Bench to evaluate evolving tool ecosystems, find that existing agents and skill-learning methods struggle to keep up with the "moving capability frontier," and then propose the ecosystem manager Steward.

   • Problem to solve: in large tool ecosystems, tools are continuously retired, replaced, and overlapping, and documentation is noisy, so the procedural knowledge an agent accumulates lags behind; the authors state that existing benchmarks examine retrieval, execution, and adaptation in isolation and cannot measure this continuous capability.
   • ∆τ-Bench: stateful MCP services across 42 domains, 1,588 validated multi-turn tasks, and 6 consecutive ecosystem versions generated by one-to-one/one-to-many/many-to-one capability-preserving transformations (9,528 task-ecosystem instances in total); dynamic programming is used to find step- and cost-optimal reference paths, with metrics including SR, retrieval P/R/F1, SPE, IDR, and DAR.
   • Model performance: among the 9 models/harnesses, the highest SR is 42.27% for Codex-GPT-5.5; retrieval shows high recall (90.36%) and low precision (19.90%); diagnostic experiments show the bottleneck is distinguishing the current version's efficient tools rather than high-level planning (Figure 3).
   • Skill methods: EvoSkill, AutoSkill, Trace2Skill, and SkillOpt all show gains on v1 but are unstable as the ecosystem evolves—on v3, Step-/Cost-DAR for EvoSkill and AutoSkill drops to 0 (No Skill is 11.11%/10.25%), and SkillOpt's maintenance cost is about 3,000 K/task.
   • Steward: incrementally maintains a three-level skill library (planning/atomic/tool), marks active status by tool availability, and dynamically supplies tools and procedural guidance during execution; average SR across six versions is 39.04% (SkillOpt 36.36%), maintenance tokens are lowest from v2 onward (baselines consume 1.2–5.5× as much), and IDR/DAR improve by about 4–5 percentage points on average.
   • Authors' conclusion: jointly managing tools and procedural knowledge as interdependent resources preserves the usability of experience better than repeatedly rebuilding skills; ablations show incremental maintenance outperforms freezing and full rebuilding, and tool search and skill search are complementary.

   Collapse this answer

Related papers