Continual Learning Exam: evaluating long-horizon RSI and experiential learning of LLM agents

Core question: how to evaluate the ability of LLM agents to continuously improve by accumulating, retaining and reusing experience across long-horizon task sequences (continual learning), whereas existing benchmarks mostly test only single-task performance.

• Problem scenario: the authors argue that continual learning is essential for long-term agent deployment—agents should be able to adapt to environmental changes, build on past experience, and avoid repeating mistakes, for example improving a tool usage workflow from a failed tool call and reusing it.
• Shortcomings of existing evaluations (according to the paper): general capability benchmarks such as GAIA present and score tasks independently, characterizing only problem-solving performance; memory benchmarks such as LoCoMo test retrieval and reasoning via question answering, without evaluating whether retained knowledge improves subsequent task execution; Claw-family benchmarks such as WildClawBench have persistent state and long tool trajectories, but persistent state itself cannot prove that the agent becomes stronger due to experience.
• Core design assumption: evaluating cumulative improvement requires a series of task attempts that support experience reuse under controlled feedback conditions. CLE organizes each task into a five-level task ladder, with later levels built on the experience accumulated in earlier levels, and designs connections between levels around five kinds of transferable knowledge (Tools and Skills, Data Dependency, Strategy, User Context, Domain Knowledge).
• Paper outputs: the CLE benchmark—120 human-written and verified tasks (600 levels), covering 12 professional domains and 89 professional roles, with 23,981 real input files, 19 stateful mock applications (1,024 APIs), 16,125 human scoring rubrics, and a multi-round reviewer agent; it also evaluates 17 configurations of 4 harnesses and 11 backbone models.
• Evaluation setting (non-adversarial, no threat model): the parameters of the agent under test are frozen, and harness state is retained across tasks; Trial 1 (K=1) allows only one submission per level with no diagnostic feedback, and Trial 2 (K=2) allows one revision and resubmission after the first failure based on diagnostic feedback by rubric category.

General Agent Capability Benchmarks

• GAIA, SWE-bench, OSWorld, WebArena, Terminal-Bench, etc.—measure reasoning, planning, and tool use through tasks ranging from question answering and software engineering to everyday workflows, scored by answer correctness or task completion. The paper notes that such benchmarks score tasks independently and characterize problem-solving performance rather than improvement arising from learning.
• Agents' Last Exam (2026)—long-horizon, verifiable tasks developed by experts across multiple industries; ClawMark (2026)—multi-day workflows, stateful services, deterministic verifiers; WildClawBench (2026)—runs tasks with real tools inside a deployed command-line harness, combining rule checks, environment state audits, and model judgment. The paper states that these benchmarks characterize problem-solving performance across different domains and workflow lengths, whereas CLE tests interrelated occupational tasks that allow experience accumulation.

Memory and In-Context Learning Benchmarks

• LoCoMo (2024)—measures long-term dialogue memory through question answering, event summarization, and multimodal dialogue generation; LongMemEval (2025) and its V2 (2026)—measure cross-session reasoning, tracking of knowledge updates, and memory of web agent state changes and recurring failure patterns.
• CL-bench (2026)—uses expert-written tasks to measure acquiring and applying new domain knowledge, rules, and procedures from context. The paper states that CLE instead evaluates knowledge reuse through occupational task outcomes, with each harness deciding for itself what to retain and how to use it.

Experience Accumulation and Reuse Benchmarks

• LifelongAgentBench (2025)—skill-related tasks in database, operating system, and knowledge graph environments, evaluating knowledge transfer sequentially; MemoryArena (2026)—hand-written subtasks coupling memory formation with subsequent action.
• SkillFlow (2026)—tasks organized into families by shared workflows, with agents updating skills from execution traces and rubric feedback and reusing them; StuLife (2025)—simulates experience accumulation across stages of university life; SEA-Eval (2026)—tracks success rate and token consumption over sequential task streams.

Comparative positioning: Table 2 compares 23 benchmarks along three axes—task design, learning protocol, and feedback and scoring—and nine criteria, with CLE marked as satisfying all nine (including Tiered tasks, Cross-level reuse, Config. attempts, Milestone feedback, Agentic judge, Audit trail), while other benchmarks lack several of them. On this basis the authors state that CLE combines occupational task execution, experience reuse under an explicit learning protocol, and auditable scoring. The paper has no method baseline; the comparison objects are combinations of 4 harnesses and 11 backbone models.

Overall idea: make continual learning evaluation into a "professional exam"—each task is a five-level progressive task ladder, and the agent must pass the current level to unlock the next level; later levels depend on the knowledge and artifacts accumulated from earlier levels; scoring is performed by a reviewer agent against human rubrics. The framework is named Continual Learning Exam (CLE).

Task ladders and transferable knowledge

• Task structure: 120 tasks, each corresponding to a professional role; the five levels L1–L5 are different task assignments under the same work context; passing the current level unlocks the next level, and levels not reached are recorded as failures.
• Five kinds of transferable knowledge: Tools and Skills (reusing tool usage and workflows), Data Dependency (using prior information and artifacts as input for later tasks), Strategy (reusing effective methods and avoiding known pitfalls), User Context (carrying over user preferences learned during interaction), Domain Knowledge (applying professional rules and conventions under new requirements); connections are not limited to adjacent levels.
• Diagnostic feedback: feedback comes from the evaluation categories of the human rubric and only indicates the relevant category, without revealing specific check items or reference answers; K=1 has no feedback, and K=2 allows one revision and resubmission after the first failure.

Environment and data construction

• Stateful mock services: 19 mock applications expose 1,024 APIs, and the same application database is shared within a task ladder, so the effects of earlier operations carry over to later levels; each application's database is initialized separately for each run.
• Real input files: 23,981 files with 77 extensions, retaining noise such as duplicate records and conflicting information; 508 levels are in English and 92 levels are in Chinese.
• Construction process (Figure 7): 50+ industry experts draft tasks based on their own job work → a reviewer agent performs automated review according to a review skill iterated by human experts (format, file authenticity, answer leakage, rubric verifiability, continual learning design), and actually runs the task as the agent under test → gives a five-tier conclusion from Strong Accept to Reject → those at Accept and above enter full independent review by two randomly assigned human reviewers, and are included only if both accept.

Formalization and evaluation protocol

• Definition of continual learning: (θ_n+1, s_n+1) = U_n(θ_n, s_n, e_n), i.e., the learning operator U uses experience e to update model parameters θ, persistent state s, or both; when the model is frozen, only the harness state s (memory, skills, etc.) is updated.
• POMDP modeling: interaction is modeled as M^A = ⟨ 𝒳, 𝒜, 𝒪, P, Ω, R ⟩; the submit action triggers review, passing gives reward 1 and otherwise 0, the objective is expected cumulative reward (equal to the number of levels passed), and the completion rate is calculated over 600 levels.
• Evaluation protocol: the main experiment uses all 120 complete task ladders (block B=(j,1,5)), executed serially in a fixed order, with harness state retained across tasks and model parameters frozen; custom blocks starting from any level are also supported (with prerequisite artifacts prepared manually).
• Review method: a multi-round reviewer agent uses tools in an isolated Docker environment to extract evidence from submitted artifacts and application state, and judges item by item against human-verified reference materials; a level passes only if all check items are satisfied. The reviewer agent is based on DeepSeek V4 Flash, with an item-level agreement rate of 96.86% with human reference labels (Appendix D, Table 4).

Experimental Setup

• Systems under test: 4 harnesses (Claude Code, Codex, Hermes-Agent, OpenClaw) × 11 backbone models (GPT-5.6-Sol/Terra/Luna, Claude Opus 5, Claude Sonnet 5, Gemini 3.7 Flash, Doubao-Seed-Evolving, Qwen3.8-Max, Hy3, DeepSeek V4 Pro, GLM 5.3), 17 configurations in total (Table 1).
• Protocol: each system runs the 120 task ladders serially in a fixed order; Trial 1 (K=1) allows one submission per level with no feedback, Trial 2 (K=2) allows one revision and resubmission after a first failure based on diagnostic feedback.
• Metrics: completion rate (proportion passed among 600 levels), per-level pass rate Lk (proportion of tasks passing all of the first k levels, L5 being full task completion), Actor tokens, cumulative execution time, cost (USD).
• Judging: the judge agent is based on DeepSeek V4 Flash; on a validation sample of 12 tasks, 60 submissions, and 732 rubric items, its agreement rate with human reference labels is 96.86% (Table 4); on 556 items, the agreement rate across three repeated scorings is 99.46% (Table 5).
• Inference settings: reasoning effort and thinking mode use each agent and model's default settings; the paper does not report the number of repeated runs of the systems under test.

Main Results

According to Table 1 (Trial 2, K=2; Trial 1 completion rates in parentheses):

Swipe horizontally to read the table

Harness / Model	Completion rate	L1	L5	Cost
Claude Code / Opus 5	47.67% (42.17%)	85.83%	23.33%	$720.56
Claude Code / Sonnet 5	40.17% (27.00%)	74.17%	19.17%	$506.30
Hermes-Agent / Gemini 3.7 Flash	33.67% (25.33%)	62.50%	15.83%	$394.84
Codex / GPT-5.6-Sol	33.17% (27.83%)	65.00%	17.50%	$210.83
Codex / GPT-5.6-Terra	33.17% (27.83%)	68.33%	12.50%	$83.16
Hermes-Agent / Qwen3.8-Max	26.67% (25.67%)	74.17%	2.50%	$134.54
OpenClaw / Hy3	14.00% (9.83%)	37.50%	5.83%	$11.95
OpenClaw / GLM 5.3	11.50% (6.17%)	27.50%	2.50%	$80.50

Omitted configurations: Codex/GPT-5.6-Luna, Hermes-Agent's Doubao-Seed-Evolving/Hy3/DeepSeek V4 Pro/GLM 5.3, OpenClaw's Gemini 3.7 Flash/Doubao-Seed-Evolving/Qwen3.8-Max/DeepSeek V4 Pro (all in Table 1).

• Overall level: the authors state that CLE is far from saturated—in Trial 2, the median L1 pass rate across the 17 configurations is 62.50%, and L5 is only 8.33%; Opus 5 fully passes 28/120 tasks in Trial 2.
• Feedback and revision: 16/17 configurations have a higher completion rate in Trial 2, with a median improvement of 5.33 pp; Sonnet 5 improves by 13.17 pp (greater than Opus 5's 5.50 pp), and OpenClaw/DeepSeek V4 Pro decreases by 1.33 pp (Figure 9B).
• Improvements do not necessarily extend to higher levels: the two Qwen3.8-Max configurations' L1–L3 pass rates rise but L4–L5 are unchanged (Table 1); Sonnet 5's number of fully completed tasks increases from 8 to 23.

Harness Comparison

• What was tested: Hermes-Agent vs. OpenClaw on 6 shared backbone models (Figure 16, Appendix F.3).
• Results: under both submission budgets, Hermes-Agent has a higher completion rate on every backbone; the average advantage increases from 9.56 pp in Trial 1 to 11.42 pp in Trial 2 (Trial 2 means 28.17% vs. 16.75%).
• Cost: in Trial 2, Hermes-Agent's average actor tokens and cost are 1.90 and 1.71 times those of OpenClaw, respectively; the authors note that these totals cover runs reaching different levels and cannot demonstrate efficiency at matched workload.

Model Generation Comparison

• What was tested: Gemini 3.6/3.7/3.8 Flash with Hermes-Agent, with the harness, task set, and order fixed (Table 7, Appendix F.4).
• Results: in Trial 1, the completion rate of 3.7 is 25.33%, higher than 3.6's 21.00% and 3.8's 21.17%; in Trial 2, 3.8 improves by 9.50 pp but is still lower than 3.7 (30.67% vs. 33.67%), and 3.8 consumes more tokens (795.78M vs. 486.25M, Trial 1).
• Authors' interpretation: across these three generations of models, performance does not improve consistently with model updates.

Other Ablations and Analyses

• Task progression (Figure 13): in Trial 1, every configuration has more than half of its tasks terminating after passing 0 or 1 levels (61/120 to 115/120); in Trial 2, only Opus 5 is below half (52/120).
• Correlation between L1 and overall: in Trial 2, the Spearman ρ between L1 pass rate and total completion rate is 0.75, and with L5 only ρ=0.42.
• Conditional pass rates (Figure 14): the median conditional pass rate between adjacent levels rises from 50.00% for L1→L2 to 75.00% for L4→L5 (Trial 1); the authors note that this is measured on increasingly favorable task subsets, and that the median denominator for L4→L5 is only 14 tasks.
• Resource trade-offs (Figure 10): in Trial 2, Codex/Terra and Sol have the same completion rate of 33.17%, but Terra spends $83.16/20.20h and Sol spends $210.83/28.20h, while Sol's number of fully completed tasks is 21, more than Terra's 15.

Limitations and Future Directions Noted by the Authors

• Feedback and knowledge retention not separated (Section 4.2): the Trial 1/2 comparison evaluates diagnostic feedback and the additional revision opportunity together; the authors propose a control that resets the harness to its initial knowledge state before each task while holding other conditions fixed, to assess the contribution of knowledge retained across tasks (the paper does not run this experiment).
• For post-training (Abstract): the authors state that the dataset can also be used to post-train LLMs to enhance continual learning ability; the paper itself does not conduct training experiments.
• Capability level coverage (Section 2.2): the authors divide continual learning ability into five cumulative levels T1–T5, state that CLE provides a foundation for evaluating T1–T5, and note that existing systems (such as Claude Cowork, Codex) have preset memory and skill workflows corresponding to T1, with early signs of T2 autonomy; the main experiments do not report results by level.
• The paper has no separate Limitations section; the above comes from the corresponding subsections of the main text.

Scope of Experimental Coverage

• Systems under test: 4 harnesses × 11 backbone models, 17 configurations in total, plus the generation comparison of Gemini 3.6/3.8 Flash (Table 1, Table 7).
• Protocol scope: all 120 task ladders are executed serially in one fixed order, model parameters are frozen, and only harness state is retained across tasks; only two submission budgets, K=1 and K=2, are tested.
• Judge reliability: the judge agent is DeepSeek V4 Flash; agreement with humans is based on 12 tasks, 60 submissions, and 732 items (Table 4), and repeatability is based on three scorings of 556 items (Table 5).
• Not reported in the paper: the number of repeated runs and variance for each run of the systems under test; the specific values of execution budgets such as context length and maximum steps for each configuration (the main text only mentions that execution budgets exist).
• Language and domains: of the 600 levels, 508 are in English and 92 in Chinese; each of the 12 occupational domains has 7–13 tasks (Section 3.5).

CLE (Continual Learning Exam) is a benchmark for evaluating the continual learning ability of LLM agents, addressing the problem that existing benchmarks only test single-task performance and cannot measure continuous improvement from experience accumulation.

• Design: 50+ industry experts write 120 real professional tasks, each organized into a five-level progressive task ladder (600 levels in total, covering 12 domains and 89 roles), with levels connected through five kinds of transferable knowledge; the environment contains 23,981 real input files and 19 stateful mock applications (1,024 APIs); the reviewer agent judges item by item against 16,125 human rubrics, and a level passes only if all are satisfied.
• Protocol: the parameters of the system under test are frozen, harness state is retained across tasks, and all task ladders are run serially; Trial 1 allows one submission per level with no feedback, and Trial 2 allows one revision after the first failure based on diagnostic feedback by rubric category.
• Main results: among 17 harness–model configurations, the highest Trial 2 completion rate is 47.67% for Claude Code/Opus 5, and only 23.33% of tasks pass all five levels; feedback plus revision improves 16/17 configurations, median +5.33 pp, but OpenClaw/DeepSeek V4 Pro declines by 1.33 pp, and improvements do not necessarily extend to higher levels (the two Qwen3.8-Max configurations show no change at L4–L5).
• Other findings: Hermes-Agent is higher than OpenClaw on all 6 shared backbones (Trial 2 mean 28.17% vs. 16.75%); in the Gemini 3.6/3.7/3.8 three-generation comparison, performance does not continuously improve across generations; the item-level agreement rate between the reviewer agent and human annotations is 96.86%.
• Authors' conclusion: CLE remains challenging for current models and harnesses, and can be used to develop and evaluate methods for acquiring, retaining and reusing experience, as well as for the continual learning ability of post-trained LLMs.