Resolution Hypothesis: the key to agent evolution is recognizing improvement
The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
The authors state that the bottleneck of agent evolution is identifying improvements rather than proposing them: under single-run evaluation noise, method rankings are close to random, and candidate selection is close to a coin flip.
- Conclusions about agent evolution methods contradict each other and rankings swap across different papers and benchmarks, and the methods often cannot get off the ground on certain tasks. The authors argue that the problem is that evaluation noise makes genuine improvements unidentifiable, rather than that the methods cannot propose better artifacts.
- The authors propose the Resolution Hypothesis, decomposing observed score variance into evaluation noise σ² and between-run variance τ², repeating evolution and evaluation across 7 methods, 6 models and 4 benchmarks, and using fixed candidate pools to separately test proposal quality (P1), selection reliability (P2) and error accumulation across rounds (P3), plus a case-by-case audit of large historical gains on 12 benchmarks.
- The median gap between methods of 3.4 pp is smaller than single-run fluctuation; on Terminal-Bench a single run selects the best method with only 33% probability; real improvements exist in the candidate pool (Quality@K rises from 49.2% to 61.8%), but the selection accuracy of execution scores and LLM judging are both close to 50%; large historical gains are mostly caused by evaluation shortcuts, and after fixing them the median gain drops from 17.6 pp to 0.8 pp.
- Limitations listed by the authors: only weight-frozen evolution is studied; candidate-level evidence covers only Terminal-Bench v2.1; it cannot rule out that future stronger models have lower noise; the grid of 7 methods, 6 models, and 4 benchmarks is not exhaustive; whether program evolution is subject to the same constraints was not tested experimentally. The candidate pool is n=100 per pool, and trajectory re-evaluation covers only 2 methods and 2 models.
- Evaluated models
- GPT-5.4 nanoGPT-5.6 LunaGPT-5.6 TerraDeepSeek-V4-FlashKimi-K3GLM-5.2
- Benchmarks
- ALFWorldSpreadsheetBenchOfficeQATerminal-Bench v2.1HotpotQAPUPAHoVerIFBenchAIME 2025LiveBench-MathLiveMathematicianBenchSearchQADocVQA
- Metrics
- evaluation noise σrun-to-run variance τQuality@Kpairwise accuracyfalse acceptance rategain over baseline (pp)
Researchers from ShanghaiTech University, ByteDance and other institutions propose the resolution hypothesis, arguing that the bottleneck in agent evolution is not the inability to propose better candidate improvements, but that evaluation noise cannot reliably identify them. The study covers seven evolution methods, six models and four agent benchmarks, totaling 1,030 independent evolution runs and more than 2.1 million test trajectories. At the method level, the median lead reported across nine papers is only 3.4 percentage points, smaller than the difference between two independent runs of the same method; at the candidate level, the probability that execution-score selection picks the better candidate is only 49.7%, no different from random, and LLM judges likewise cannot reliably identify it. An audit of the largest reported gains on twelve benchmarks shows that these gains usually come from evaluation shortcuts: a single rule can reproduce a median 90% of the gain, and after fixing the evaluation the median gain drops from 17.6 percentage points to 0.8 percentage points. The authors suggest shifting focus from designing more candidate-improvement systems to making real improvements detectable and accumulable, for example lower-noise intermediate behavior checks and separately evaluating the proposer and the selector.
Why it was selectedLarge-scale experiments with seven methods and six models show that the bottleneck in agent self-evolution is that evaluation cannot identify real improvements, and rankings from a single run are basically down to luck.
Paper analysis
What problem does this paper attempt to solve?
The authors say the bottleneck of agent evolution is identifying improvements: evaluation noise makes both method rankings and candidate selection close to random.
Core question: why conclusions about agent evolution contradict each other and often cannot get off the ground across different papers and benchmarks—the authors argue the root cause is that evaluation cannot reliably identify improvements, rather than that the methods cannot propose improvements.
- Problem setting: agent evolution follows the loop of "a proposer writes candidate artifacts (prompts, skills, memory, harness) → evaluation → selection accepts the high scorers." The same set of methods works sometimes and not other times across different benchmarks, and rankings swap between papers; on coding agent benchmarks such as Terminal-Bench they have repeatedly been reported to stall, while on other benchmarks every new method claims to surpass its predecessors.
- Problems with current practice: the authors survey 171 main-table results from nine agent evolution papers; none reports variance estimates, and only 6 average over multiple evolution runs (Table 8); the common practice is a single evolution run and a single evaluation, conflating evaluation noise with run-to-run variance.
- Core hypothesis (Resolution Hypothesis): the bottleneck is identifying which artifacts bring improvements, not proposing them. With weights frozen, artifacts can only take effect by changing inference behavior, and agent trajectories themselves vary from run to run, making it hard to separate genuine behavioral change from evaluation noise; the authors call the smallest genuine improvement that the evaluation pipeline can reliably distinguish "resolution."
- What the paper did: across 7 methods, 6 models, and 4 agent benchmarks, it ran 1,030 independent evolution runs and over 2.1 million test trajectories (over 10 million LLM calls in total), testing the hypothesis at two levels—method level (ranking reliability) and candidate level (proposal, selection, error accumulation)—and audited large historical gains on 12 benchmarks; the authors call it the largest-scale reproducibility study of agent evolution to date.
- This is not an attack/defense paper: it does not involve a threat model; it is a statistical analysis and reproducibility study of the agent evolution paradigm, and the affected object is the practice itself of "method rankings and candidate selection based on noisy evaluation."
What related research exists?
Related work falls into three categories: evolution methods themselves, analyses documenting the fragility of evolution, and the statistics literature on noisy selection.
Agent Evolution Methods (objects under evaluation)
- Text-artifact methods: GEPA (Agrawal et al., 2025) evolves prompts; SkillOpt-Lite (Shen et al., 2026, a lightweight variant of SkillOpt) and Trace2Skill (Ni et al., 2026) evolve skill documents; ReasoningBank (Ouyang et al., 2026) evolves a memory playbook. These four are the objects of this paper's method-level experiments.
- Harness methods: HarnessLens (Xu et al., 2026) edits instructions, skills and tool descriptions; Meta-Harness (Lee et al., 2026) and AHE (Lin et al., 2026) edit agent scaffold code. The three form the second method family evaluated in this paper.
- Other cited methods: ACE (Zhang et al., 2025) evolves context; Darwin Gödel Machine (Zhang et al., 2025) performs open-ended self-improvement evolution; this paper's candidate-pool experiments borrow ACE Reflector's reference-free judging template as one of the LLM judge conditions.
Analytical Work Documenting the Fragility of Evolution
- Ye et al. (2026) point out that self-improving agents are sensitive to run variance and task order; Wang et al. (2026) emphasize matched-budget baselines and held-out evaluation; Liu et al. (2026) find that gains are sparse and depend on feedback.
- SkillsBench (Li et al., 2026) measures the effect of manually curated skills; subsequent work analyzes skill-induced failures (Dong et al., 2026) and contamination caused by uncontrolled skill accumulation (Shang et al., 2026).
- SEAGym (Zheng et al., 2026) and continual-learning evaluation on Terminal-Bench 2.0 (Wang et al., 2026) distinguish intermediate gains from transfer, regression and sustained improvement.
Statistical Literature on Noisy Selection
- Related to best-arm identification (Kaufmann et al., 2016), overfitting in model selection criteria (Cawley & Talbot, 2010), and uncertainty-aware evaluation in deep reinforcement learning (Agarwal et al., 2021).
- PACE (Shawn, 2026) directly studies false acceptance under repeated noisy comparisons and proposes an anytime-valid acceptance test; the paper states that its distinction lies in separating evaluation noise from between-run variance and independently testing proposal quality and selection reliability with controlled candidate pools.
Baselines and Benchmarks: The control for method-level experiments is the unevolved baseline (evaluated the same number of times as each method); the control for candidate-level experiments is random selection and a selection oracle. Benchmarks are ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1, and the gain audit additionally covers 9 benchmarks from the original method papers (HotpotQA, PUPA, HoVer, IFBench, AIME 2025, LiveBench-Math, LiveMathematicianBench, SearchQA, DocVQA).
Authors' positioning: Prior work documented the phenomenon that evolution "can fail"; this paper isolates the mechanism—separating evaluation noise from between-run variance and separately testing the two stages of proposal and selection.
How does the paper address this problem?
Variance decomposition is used to separate evaluation noise from run-to-run variance, and the hypothesis is then tested with three groups of experiments: method level, gain audit, and candidate level.
Overall approach: strictly decompose the variance of a single observed score into two components—evaluation noise and between-run variance—define the "resolution" of evaluation accordingly, then test the falsifiable predictions of the Resolution Hypothesis at the method level and candidate level respectively, and conduct a case-by-case instruction audit of large historical gains.
Formalization: Two-Variance Decomposition
- The true quality of artifact A is defined as its expected score Y(A) = 𝔼[Ŷ ∣ A], and a single observation is Ŷ(A) = Y(A) + ϵ, where ε is zero-mean evaluation noise.
- By the law of total variance, the observed score variance decomposes into σ² (the noise variance of repeated evaluation of the same artifact) plus τ² (the variance of true quality of artifacts produced by different evolution runs); identifying the gap between two methods is limited by τ²+σ², while identifying the gain of a single fixed artifact is limited only by σ².
- Estimators (Appendix B.2): σ̂² is the mean of within-artifact repeated-evaluation variances; τ̂² uses the method-of-moments estimate from random-effects variance analysis, subtracting the residual σ̂²/M from the variance of artifact means; both are unbiased, and when τ̂² is negative, τ̂=0 is reported (occurring 9 times among 117 method configurations).
Method Level: Resolution and Ranking Simulation
- Each configuration (method × model × benchmark) evolves independently R=8 times (R=16 for text-artifact methods on GPT-5.4 nano), and each artifact is evaluated M=16 times on the held-out test set (M=8 for Terminal-Bench and harness methods), for 150 configurations in total.
- The number of runs needed to identify a true gap ∆ is computed by the power formula: at 80% power and α=0.05, N ≥ 2((z1−α/2+z1−β)/Δ)2 (τ2+σ2), i.e., the required number of runs grows with single-run variance and grows quadratically as the gap shrinks.
- Ranking simulation: For each model and benchmark, randomly draw one run and one evaluation for each method to rank, compare with the all-run mean ranking, and average over 20,000 repetitions (Appendix D.2).
Gain Audit: One-Sentence Reproduction and Fixed Evaluation
- Objects: The artifacts with the largest reported gains on 12 benchmarks (published by GEPA, SkillOpt-Lite, and evolved by the authors themselves), using three models GPT-5.4 nano, DeepSeek-V4-Flash, GPT-5.6 Luna.
- One-sentence test: An LLM agent marks instructions that may explain the gain one by one, the authors manually check each, attach the most suspicious one as a single-sentence rule to the unevolved baseline, and compare how much of the gain it reproduces; rules are fixed before looking at results.
- Fixed evaluation: Restore the official evaluation pipeline, or change surface-form matching to semantic scoring (correct only if both LLM judges DeepSeek-V4-Flash and GPT-5.6 Luna accept), then re-score the same batch of artifacts; some benchmarks are also re-evolved under the fixed evaluation.
Candidate Level: Two Fixed Candidate Pools and Three Predictions
- Proposal Pool (tests P1): A proposer based on SkillOpt-Lite (DeepSeek-V4-Flash) offline generates n=100 skills from 78 trajectories of 10 Terminal-Bench v2.1 training tasks, each evaluated 16 times on 10 disjoint test tasks (16,000 rollouts in total); the metric Quality@K is the expected success rate of the best of K skills randomly drawn from the pool, estimated with an empirical Bayes Beta posterior, paired with an "equal-quality null hypothesis" curve (1,000 simulations) to subtract pure sampling luck.
- Selection Pool (tests P2): n=100 skills come from trajectories of 6 Terminal-Bench tasks and 3 DeepSWE tasks, evaluated on 11 Terminal-Bench tasks stratified by difficulty (8 rollouts per pair, with the top 10 artifacts per task expanded to 15); each selector can only see the first rollout of each artifact (SkillOpt-Lite and GEPA native selection see two), and true quality is independently estimated from the remaining held-out rollouts.
- Selection mechanisms under test: Random, greedy acceptance by single execution score, SkillOpt-Lite and GEPA native parent selection, three LLM trajectory judges (pointwise, pairwise, ACE Reflector-style template), with DeepSeek-V4-Flash as the judge model; metrics are pairwise accuracy, false acceptance rate, Spearman rank correlation.
- P3 Trajectory re-estimation: All 119 intermediate artifacts produced by 8 SkillOpt-Lite and 8 GEPA evolution runs (DeepSeek-V4-Flash, GPT-5.6 Luna) are independently re-estimated on the same validation set with 8 rollouts per artifact–task pair (only 1 used during evolution), comparing in-loop gains with re-estimated gains and estimating the noise-corrected between-run variance per round.
What experiments did the paper conduct?
Four groups of experiments: variance measurement, gain audit, candidate pools, trajectory re-estimation; both method ranking and candidate selection are close to random.
Experimental Setup
- Methods under test: 7 evolution methods—the text-artifact family SkillOpt-Lite, GEPA, Trace2Skill, ReasoningBank; the harness family HarnessLens, Meta-Harness, AHE, all using official implementations.
- Models under test: 6 in total—GPT-5.4 nano, GPT-5.6 Luna, GPT-5.6 Terra, DeepSeek-V4-Flash, Kimi-K3, GLM-5.2; coverage varies with cost, with text-artifact methods using all 6 on ALFWorld/SpreadsheetBench/OfficeQA, and Terminal-Bench using only DeepSeek-V4-Flash, Luna, Terra (Table 5).
- Benchmarks: Variance measurement uses ALFWorld, SpreadsheetBench (Verified-400 subset, 281 test tasks), OfficeQA (148 test questions), Terminal-Bench v2.1 (18 train/18 validation/53 test); the gain audit adds 9 more benchmarks.
- Metrics: Task-level binary scores (success rate/exact match), reporting mean, σ̂, τ̂ (in pp); candidate level uses Quality@K, pairwise accuracy, false acceptance rate.
- Sample size: Method level 150 configurations, 1,030 independent evolution runs, over 2.1 million test trajectories; Proposal Pool 16,000 rollouts; Selection Pool 8–15 rollouts per pair; trajectory re-estimation 119 artifacts × 8 times.
- Judging method: Semantic scoring in the gain audit uses dual judges DeepSeek-V4-Flash and GPT-5.6 Luna with consistent acceptance; candidate-level LLM trajectory judging uses DeepSeek-V4-Flash.
Main Results
Probability of selecting the correct best method when a single run ranks four text-artifact methods (Appendix D.2, averaged over 20,000 samplings; random guessing is 25%):
Swipe horizontally to read the table
Benchmark Selects correct best method Full ranking correct Median single-run standard deviation SpreadsheetBench 67% 33% Lower end of the approx. 1.7–4.1 pp range OfficeQA 51% 29% Within the same range ALFWorld 73% 24% Within the same range Terminal-Bench 33% 8% Upper end of the same range (σ reaches 4.5 pp) According to Appendix D.2 and Table 6; single-run standard deviation varies by configuration from 1.7–4.1 pp (median across configurations per benchmark), full per-configuration values in Table 6.
- The authors note that among the 171 results in nine papers, the median margin by which the method leads its strongest competitor is 3.4 pp (interquartile range 1.2–6.6 pp), while for over 60% of same-model same-benchmark configuration pairs, the difference in all-run means is smaller than one standard deviation of a single comparison (Figure 2a, 217 of 341 pairs fall below the diagonal).
- By the power formula, identifying the median magnitude of 3.4 pp requires approximately 5 runs (SpreadsheetBench), 10 runs (OfficeQA), 19 runs (ALFWorld), 23 runs (Terminal-Bench); at the lower quartile of 1.2 pp, Terminal-Bench requires 181 runs (Table 7).
- The authors accordingly argue that published method rankings are largely determined by noise.
Gain Audit: Large Historical Gains Come from Evaluation Shortcuts
- One-sentence gain reproduction: Among 27 model–benchmark pairs reporting gains above 3 pp, attaching a single rule to the unevolved baseline reproduces a median 90% of the gain (Figure 3a); the rules mostly adapt to the scorer (e.g., writing only numeric values without formulas, outputting only the shortest answer fragment), with specific rule texts in Table 10.
- Gains disappear after fixing: After fixing evaluation shortcuts (LibreOffice recomputing formulas, semantic scoring, removing leaked options, etc., Table 9), the median gain of the same batch of artifacts drops from 17.6 pp to 0.8 pp (Figure 3b); re-evolving text-artifact methods under the fixed evaluation yields gains of at most 1.9 pp, with the only exception being SkillOpt-Lite with DeepSeek-V4-Flash on SpreadsheetBench at +5.8 pp (Table 12).
- Exceptions: On HotpotQA, GEPA retains a 5.5–7.6 pp gain under semantic scoring, which the authors attribute to retrieval improvements (supporting-article recall rises by about 10 pp); HoVer's 8–13 pp gain comes from a multi-hop retrieval strategy, with a single-sentence rule reproducing only 41%–43% (91% for DeepSeek-V4-Flash).
Candidate Level: Proposals Effective, Selection Failing, Error Accumulating
- P1 Proposals are not the bottleneck: On the Proposal Pool, Quality@K rises from 49.2% at K=1 to 61.8% at K=100 under a 2-hour budget, and from 39.7% to 51.6% under the original timeout; under both timeouts the observed curve at K=100 is +3.3 to +3.4 pp above the equal-quality null hypothesis curve (Figure 4a). The strongest shared skill artifact-063 reaches 57.5%, +8.3 pp above the pool mean.
- P2 Selection is close to random: Pairwise accuracy of single execution-score selection is 49.7% (random 50.0%), with a false acceptance rate of 62.1% (regressed artifacts make up 30.5% of the pool); SkillOpt-Lite and GEPA native selection (looking at two rollouts) is 48.6%; three LLM judges are 45.4%–49.0% (Table 1). The rank correlation between single execution score and artifact quality is 0.054. In Figure 4b, the gain curves of all practical selectors stay close to the random baseline, while the oracle exceeds their K=100 level within K<5.
- P3 Error accumulates across rounds: In-loop validation gains rise to 6.9–9.0 pp, while independently re-estimated gains are only +0.4 to −3.2 pp (Figure 5); the noise-corrected between-run variance τ̂t² in the final round reaches 4.2–24.5 pp² (Figure 6). The authors state that the acceptance rule accepts lucky samples from the upper tail of the noise distribution each round, so scores rise but artifacts do not improve.
Other Ablations and Analyses
- Increasing validation resolution: SkillOpt-Lite averages 16 validation rollouts per step (16 runs), raising the final test mean from 61.4% to 64.4% and the best artifact from 64.8% to 72.7% (Appendix F.3).
- Low-noise process metrics: The check-command ratio of unmasked diagnostics has evaluation noise 13×–57× lower than terminal pass/fail (Table 13, DeepSeek-V4-Flash).
- Behavioral changes visible in process metrics: Trace2Skill raises the post-edit validation rate by +8.73 pp (z=2.17), AHE lowers it by −17.98 pp on GPT-5.6 Luna (z=−4.19), and neither is visible in terminal success rate (Appendix H).
- Sensitivity of Quality@K to the prior concentration cap: At κmax=100 it exceeds the null hypothesis by +3.41 pp, and at κmax=300 and unbounded it is +4.75 and +6.42 pp respectively, with the direction of the conclusion unchanged (Figure 11).
- Timeout sensitivity: The net amount by which K=100 exceeds the null hypothesis is +3.22 to +3.41 pp across five truncations from the original timeout to 2 hours (Figure 9).
What are some points that can be further explored?
The authors list five limitations, including being limited to frozen weights and candidate-level evidence covering only Terminal-Bench; they also recommend resolution-aware evaluation.
Limitations and future directions noted by the authors
- Frozen weights only (Limitations): only agent evolution with frozen weights that evolves external state is studied; methods that update weights with reinforcement learning have different gradient dynamics and may face different resolution constraints.
- Candidate pool scope (Limitations): candidate-level evidence is evaluated only on Terminal-Bench v2.1; whether the conclusions apply to larger pools, different generation strategies, or other benchmark domains remains, the authors say, an open question.
- Model strength and resolution (Limitations): the strongest model in the grid is Kimi-K3; the authors say they cannot rule out that future stronger models have lower evaluation noise and better resolution.
- Grid coverage (Limitations): 7 methods, 6 models, and 4 benchmarks are a broad but not exhaustive investigation; different artifact representations or task structures may have different variance characteristics.
- Program evolution not covered (Limitations): whether evolving programs that solve deterministic-scoring problems (such as GPU kernels, ARC-AGI-2, Atari policies) is likewise subject to resolution constraints—the authors say the framework can answer this but the experiment was not done.
- Future directions (Section 6, Conclusion): the authors suggest making evaluation "resolution-aware"—lower-noise intermediate process metrics, component-wise diagnostics of the proposer and selector (Quality@K curves and selection quality curves), and reporting σ̂ and τ̂ alongside point estimates, so readers can judge whether a claimed margin exceeds the evaluation resolution.
Experimental coverage
- There are 6 models tested: GPT-5.4 nano, GPT-5.6 Luna, GPT-5.6 Terra, DeepSeek-V4-Flash, Kimi-K3, GLM-5.2; more expensive settings such as Terminal-Bench and the harness method use only 3 of them (Table 5).
- Method-level variance measurement covers 7 methods × 4 benchmarks, 150 configurations in total, and 1,030 independent runs; the two candidate-level pools each have n=100 skills, and both the proposer and the trajectory review use DeepSeek-V4-Flash.
- Trajectory re-evaluation (P3) covers only two methods, SkillOpt-Lite and GEPA, and two models, DeepSeek-V4-Flash and GPT-5.6 Luna, with 8 runs each.
- The gain audit covers 12 benchmarks and 3 models; ALFWorld, PUPA, HoVer, and IFBench showed no shortcuts and therefore have no post-fix results, and IFBench and LiveBench-Math have no one-sentence test results (Appendix E.2).
- The paper does not report a comparison of query costs on the selector for the candidate-level experiments; the improved-selection experiment with 16 validation rollouts ran only one method, SkillOpt-Lite (Appendix F.3).
Summarize the main content of the paper
The authors state that agent evolution is constrained by evaluation resolution: improvements exist but cannot be identified, and large gains mostly come from evaluation shortcuts.
A statistical analysis paper on the reproducibility of agent evolution, proposing the Resolution Hypothesis: the bottleneck of evolution is identifying which artifacts bring improvements, not proposing improvements.
- Problem: The same batch of evolution methods yields swapped conclusions and unstable rankings across different benchmarks and papers; the authors investigate the 171 main-table results of nine papers, with a median lead of 3.4 pp, none reporting variance, and only 6 averaging over multiple runs.
- Method: Decompose the observed score variance into evaluation noise σ² and between-run variance τ², conduct 1,030 independent evolution runs and over 2.1 million test trajectories across 7 methods, 6 models and 4 benchmarks, test the hypothesis at the method level and candidate level, and audit large historical gains on 12 benchmarks.
- Method-level results: The gaps between most methods are smaller than one standard deviation of a single run; the probability that a single run selects the correct best among four text-artifact methods is only 33% on Terminal-Bench (random 25%); identifying the median magnitude requires 5–23 runs.
- Gain audit: Among 27 model–benchmark pairs with gains above 3 pp, a single-sentence rule reproduces a median 90% of the gain; after fixing evaluation shortcuts, the median gain drops from 17.6 pp to 0.8 pp, and re-evolution does not recover it. Exceptions are HotpotQA (GEPA retains 5.5–7.6 pp, from retrieval improvements) and HoVer (8–13 pp, from retrieval strategy).
- Candidate-level results: Real improvements exist in the candidate pool (Quality@K rises from 49.2% to 61.8%, +3.3 to +3.4 pp above the equal-quality null hypothesis), but pairwise accuracy of selection by execution score is 49.7% and LLM judging is 45.4%–49.0%, both close to random; in-loop gains of 6.9–9.0 pp shrink to +0.4 to −3.2 pp under independent re-estimation, and selection error accumulates round by round into between-run variance.
- Authors' conclusion: The proposer determines which improvements are available, and evaluation resolution determines which improvements can be retained and accumulated; large and easily detectable changes dominate evolution even if they rely on shortcuts. The authors recommend shifting to resolution-aware evaluation: low-noise process metrics (13×–57× lower noise in preliminary analysis), component-wise diagnostics of proposer and selector, and reporting σ̂ and τ̂; limitations include covering only frozen weights, candidate-level evidence limited to Terminal-Bench, and program evolution not tested.