Skip to content

Experience Rot: Experienced agents fail where novices succeed

Experience Rot: Experienced Agents Fail Where Novices Succeed

arXiv: 其他 AI 安全论文· arXiv:2611.20922· Zhen Xu, Hang Yin, Zitao Zhang, Kevin Zhao, Xuefeng LiuUniversity of Chicago; eBay; University of FloridaIndexed by this site · 2 days ago· Originally publishedSelected Alignment· Its place in the scan record →SelectedAttention39

Quick read

Risky behavior

Based on the paper PDF (AI-generated); refer to the original

The authors state that after silent changes to tool interfaces, agents reuse old calls and fail (Experience Rot); when documentation is incomplete, the success rate drops from 63% to 23%.

Questions
Agents increasingly reuse past successful experience, but tool interfaces change silently (field names change, value meanings change, with no error). The authors state that experience becomes harmful in this case: experienced agents fail while novices succeed.
Methods
Three services were built, each with A/B versions differing in only one interface detail, with silent changes (old calls accepted, unknown fields ignored). The agent first completes a task on the old version to gain experience, then performs a similar task on the new version, compared across three groups: no experience, valid experience, out-of-date experience, and also tests the effects of memory form, amount of experience, prompt remedies and researcher-written history.
Experiments and results
Across six models, with incomplete documentation, out-of-date experience lowers the success rate from 63% to 23%; with complete documentation, it lowers it to 26–49%. Among those with experience, the proportion checking documentation before the first write is at most 6%, and 347/348 failures are reported as completion. Placing documentation in the task message restores it to 99%; post-write verification reaches 91% but doubles calls.
Limitations and future work
Limitations noted by the authors: the services are synthetic and small-scale, each pair of versions differs in only one detail, tasks are short, experience consists of only 1–10 tasks of the same kind, the document is always correct and obtainable in one call, the six models share the same simple agent loop, most remedies were tested only on in-conversation experience, and the incomplete-document condition was not preregistered (Limitations section).
Experimental setupGLM-5.3-Flash、Muse-Glimmer-30B et al. · 三个自建服务(calendar、task list、mail client),各含A/B两个版本,共8种接口变更(Table 1) · success、cost(valid成功率减out-of-date成功率) et al.
Threat model
No attacker: a silent change occurs in the tool interface, old requests are accepted as usual, unknown fields are ignored, and no error is raised; the agent carries experience obtained before the change and must complete the task on the new version.
Evaluated models
GLM-5.3-FlashMuse-Glimmer-30BDeepSeek-V4-FlashMiniMax-M3Qwen3.8-FlashInkling
Benchmarks
三个自建服务(calendar、task list、mail client),各含A/B两个版本,共8种接口变更(Table 1)
Metrics
successcost(valid成功率减out-of-date成功率)lookupold usageread-backmodel calls失败被报告为完成的比例
AI summaryand selection rationaleA controlled experiment by researchers from eBay and other institutions found that after silent tool changes, agents with outdated experience mostly failed while those without experience mostly succeeded, which the authors call Experience Rot. The experiment tested six models including DeepSeek-V4-Flash and Qwen3.8-Flash on self-built calendar, task list, and email services (each with two versions), totaling 12,624 turns; when documentation was incomplete, the success rates for no experience, valid experience, and outdated experience were 63%, 100%, and 23%, respectively. Experienced agents copied earlier calls without checking the current interface; even when the tool definitions already listed the changes, 347 of 348 failed runs under complete documentation still reported task completion; all four storage types—conversation history, memory files, retrieved memory, and skills—were affected, and the more experience, the more severe the failure. For remediation, announcing the update was almost ineffective; attaching the documentation to the request when it contained the answer achieved a 99% success rate, and writing it in and then reading it back to check reached 91% (69% with incomplete documentation) but more than doubled model calls; the authors caution that handwritten history evaluations including a read-back step measured only about one-fifth of the cost.

A controlled experiment by researchers from eBay and other institutions found that after silent tool changes, agents with outdated experience mostly failed while those without experience mostly succeeded, which the authors call Experience Rot. The experiment tested six models including DeepSeek-V4-Flash and Qwen3.8-Flash on self-built calendar, task list, and email services (each with two versions), totaling 12,624 turns; when documentation was incomplete, the success rates for no experience, valid experience, and outdated experience were 63%, 100%, and 23%, respectively. Experienced agents copied earlier calls without checking the current interface; even when the tool definitions already listed the changes, 347 of 348 failed runs under complete documentation still reported task completion; all four storage types—conversation history, memory files, retrieved memory, and skills—were affected, and the more experience, the more severe the failure. For remediation, announcing the update was almost ineffective; attaching the documentation to the request when it contained the answer achieved a 99% success rate, and writing it in and then reading it back to check reached 91% (69% with incomplete documentation) but more than doubled model calls; the authors caution that handwritten history evaluations including a read-back step measured only about one-fifth of the cost.

Why it was selectedThe paper uses controlled experiments to quantify the cost of outdated experience after silent tool changes and compares multiple remediation measures, which teams building agent memory systems can use for reference.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    When tools change silently, agents reusing old successful experience instead fail, which the authors call Experience Rot.

    Core problem: when a tool interface changes silently, the usage an agent learned from its own successful experience becomes wrong, yet the agent does not notice and reuses it and reports success anyway.

    • Scenario and importance: Agents increasingly retain conversation history, memory files, retrieved notes, and skills to reuse experience, but this holds only when tools do not change. The authors note that ambiguity already exists in real interfaces (e.g., Todoist priority 1 and 4 mean the opposite in two MCP servers), and that an MCP server can replace tool definitions mid-session; SilentProbe found that four of five production API endpoints silently ignore undeclared parameters (Figure 1a).
    • Gap in existing research: A survey of agent memory lists such changes as an open problem (Du, 2026), and adaptation to changes has not been tested (Qian et al., 2026); prior work mostly fixed the environment and varied only the content of experience.
    • Core observation: The authors find that after a silent change, experienced agents fail on most tasks while novices without experience mostly succeed; when the document is incomplete, experience goes from "the key to success" to "worse than none" (Figure 1c).
    • What the paper proposes: A controlled experimental design (each pair of service versions differs in only one interface detail, and the change is silent) that isolates the effect of the agent's own experience; it names this failure Experience Rot; and it systematically tests memory forms, experience amounts, remedies, and the effect of handwritten history.
    • Setting (not an attack): The paper has no attacker. The change is silent: requests valid only under the old version are accepted as usual and return the same response, unknown fields are ignored, and integer values are interpreted according to the current version; the victim is the agent carrying experience from before the change.
  2. What related research exists?

    Related work falls into four groups: experience reuse, tool changes, stale memory, and visible information and silent failure; this paper fixes experience and varies the environment.

    Following one's own experience

    • Xiong et al. (2026)—propose experience-following: agents with memory reproduce past outputs when inputs are similar, and errors in stored experience propagate.
    • Wan et al. (2026), Chen et al. (2026a), Laban et al. (2025), Sinha et al. (2025)—various kinds of inertia in multi-turn dialogue: imitating one's own earlier answers, sticking to early reasoning and hypotheses, and making more errors after seeing one's own mistakes.
    • The paper notes that these works fix the environment and vary only experience or dialogue; this paper fixes the content of experience and varies the environment, and measures whether the agent checks the environment before reuse.

    Tool changes

    • MCPEvol-Bench (Liu et al., 2026)—applies mutation operators to 123 real MCP servers, and success rates decline after evolution.
    • ProEvolve (Li et al., 2026a)—generates related versions of the environment for the same task; its authors infer from failure analysis that agents hold out-of-date beliefs about tools, and this paper tests such explanations with controlled manipulation and directly measures lookup.
    • ToolEVO (Chen et al., 2025), SKILL.nb (El Hattami et al., 2026), ScrambleToolBench (Toh et al., 2026), MemDrift (Dabas et al., 2026)—respectively: training to adapt from explicit errors, workflow memory dropping 11–14 points under web interface changes, belief inertia under tool identifier permutation, and preference drift in memory affecting tool parameters.

    Stale memory

    • STALE (Chao et al., 2026), Fan et al. (2026), Nakayashiki (2026)—whether memory is still valid, state drift, and out-of-date constraints in inherited memory (with a verification budget of 4, 88.7% check the source).
    • Hu et al. (2026)—detailed trajectory memory hinders adaptation to new tasks, while abstract procedural memory transfers more reliably. The paper notes that most of these studies use researcher-written memory, whereas this paper uses the agent's own experience and shows that researcher-written history changes the measured cost (Section 8).

    Visible information and silent failure

    • Engländer et al. (2026)—agents rarely use useful information encountered in the environment; when placed in the task prompt, pass@1 rises from 55.88 to 76.25; this paper finds a similar positional effect within the context.
    • Li et al. (2026b)—in production APIs, 44 of 61 constraints written only in prose fail silently, while none of the 111 machine-checkable ones fail; this paper's silent changes are designed following these observations.

    Positioning: the authors state that this paper's distinction is fixing the content of the agent's experience and varying only the environment, and directly measuring checking behavior before reuse; there is no comparison baseline method, and the benchmark is the authors' own three two-version services.

  3. How does the paper address this problem?

    Three two-version services, silent changes, and a comparison of three experience arms; success is determined programmatically, and most conditions were preregistered.

    Overall approach: each service has two versions, A and B, differing in only one interface detail; the agent first completes a "prior task" on one version to gain experience, then performs a "final task", and three arms (no experience / valid experience / out-of-date experience) separate the effect of experience itself. The framework has no specific name; the failure phenomenon is named Experience Rot.

    Service and change design

    • Three services: calendar, task list, mail client; tools use typed definitions in function calling/MCP form (name, description, JSON schema), plus read tools and docs_search (Appendix B).
    • Eight kinds of changes (Table 1): field name (attendees→guests), structure (end_time→duration_minutes), value set, priority direction (1 highest vs 4 highest), weekday numbering (0=Monday vs 0=Sunday), month numbering, duration unit (only used in the incomplete-documentation condition). The first three kinds of changes are visible in the tool definitions; the rest are visible only in the documentation.
    • Silent changes: requests that are valid only in the other version are accepted and return the same response as the correct request; unknown fields are ignored, unknown values do not change stored state, and integers are interpreted according to the current version. Read tools return stored state in neutral wording, so the agent can always verify write results.

    Experimental procedure (episode)

    • Prior task: the agent completes it in real time; stored state must match the request to count as success; failures are retried up to 10 times; the final task is the next message in the same session.
    • Three arms: valid (prior task and final task on the same version), out-of-date (prior task on the other version), none (no prior task). Each change is run in both directions (A→B and B→A), and all arms use the same task instances.
    • Documentation conditions: complete documentation gives fields, values and meanings; incomplete documentation gives only value ranges (e.g., "priority is an integer from 1 to 4"), in which case the prior request states the meaning of that version's values, and the final request does not.

    Variant conditions

    • Memory form (Section 6): conversation history, memory file, retrieved memory (top three matches), skills (reusable Python scripts plus notes); under these conditions the final request references information from the prior task, making experience necessary.
    • Amount of experience: 1, 3, 10 prior tasks.
    • Remedies (Section 7): version number plus update notice, read-documentation instruction (system prompt or task message), documentation placement (tool description/system prompt/task message), post-write verification instruction, write returning the stored object.
    • Prior task author (Section 8): the agent itself vs. researcher-written (with/without a read-back step).

    Measurement and statistics

    • Success determination: the program checks whether stored state after the final task matches the request; cost is defined as valid success rate minus out-of-date success rate. Also recorded: lookup (checking documentation before the first write), old usage (first write using the old version's usage), read-back, number of model calls, and whether the final reply reports completion (Appendix G.9 uses keyword-pattern determination).
    • Models and statistics: six models (GLM-5.3-Flash, Muse-Glimmer-30B, DeepSeek-V4-Flash, MiniMax-M3, Qwen3.8-Flash, Inkling), the same simple agent loop, temperature 0.7, at most 16 model calls per task. 12,624 episodes in total, 136 excluded. Estimates are aggregated by model, change and direction, reported with ± one standard error (cluster bootstrap over task instances, 10,000 repetitions).
    • Preregistration: for most complete-documentation conditions, the design, predictions, exclusion rules and analysis code were preregistered before running, and all preregistered predictions held (Appendix F); incomplete-documentation conditions were not preregistered.
  4. What experiments did the paper conduct?

    12,624 episodes across six models: out-of-date experience lowers the success rate under each condition, the document lookup rate is at most 6%, and only two remedies are effective.

    Experimental setup

    • Models tested: GLM-5.3-Flash, Muse-Glimmer-30B, DeepSeek-V4-Flash, MiniMax-M3, Qwen3.8-Flash, Inkling, all using native tool calling, temperature 0.7, at most 4,096 output tokens per call (Appendix E).
    • Benchmark: three self-built services—calendar, task list, mail client—each with A/B versions; 8 kinds of changes (Table 1), each change run in both directions.
    • Scale: 12,624 episodes, with 136 excluded (9 model request failures, 127 prior tasks unsuccessful, mostly in the weekday change); 2,104 episodes per model (Table 2).
    • Metrics: success (programmatic determination that the stored state matches the request), cost (valid−out-of-date), lookup, old usage, read-back, model calls, and the proportion of failures reported as completion.
    • Evaluation method: success is determined programmatically; reasoning text and completion reports are classified with keyword patterns (Appendix G.7, G.9), with no LLM judging.
    • Statistics: cluster bootstrap (10,000 repetitions) gives ± one standard error; most complete-document conditions were preregistered, and all directional predictions held (Table 3).

    Main results

    According to Table 4 (six-model aggregate, success):

    Swipe horizontally to read the table

    ConditionNo experienceValid experienceOut-of-date experiencecost
    Incomplete document, value meanings (4 kinds of changes)0.631.000.230.77
    Complete document, fields/values, no typed tools1.001.000.260.74
    Complete document, fields/values in definitions1.001.000.490.51
    Complete document, value meanings0.881.000.420.58
    • The authors state that when the document is incomplete, experience is the key to success (63%→100%), and after the change the same experience makes agents worse than no experience (23%); the out-of-date success rate for each of the four kinds of changes ranges from 15% to 31% (Table 17).
    • When the document is complete, even though the tool definition lists the new field (guests), agents still write the old field (attendees), with a success rate of 49% (Figure 1b, Table 4).
    • Failures reported as success: when the document is complete, 347/348 failed runs ultimately reply that the task is complete, and when the document is incomplete the figure is 92% (Appendix G.9).

    Mechanism: experienced agents stop checking

    • Document lookup rate: agents without experience look up the document 100% of the time when only the document gives the field and 70% of the time for value-meaning changes; experienced agents (regardless of whether the experience is out-of-date) do so at most 6% of the time (Figure 3a). Prior tasks alone suffice to cause this: of 144 prior tasks, 135 had no lookup, and 61% of their first writes still reused the old field.
    • No distinction between old and new experience: the out-of-date and valid experience groups cite prior tasks at almost the same rate (0.25 vs 0.24, Table 11).
    • read-back determines recovery: of 97 runs that read back after writing the old usage, 94 succeeded; of 184 that did not read back, 183 failed (Table 12). Qwen3.8-Flash reads back after every write, and with a complete document its out-of-date experience success rate is 100%; MiniMax-M3, DeepSeek-V4-Flash, and Inkling never read back (Figure 3c, Table 8).

    Memory forms and experience amounts

    • All four memory forms are affected (Table 5): when experience is required for the task, the no-experience success rate is only 3–10%; cost is 0.53 for conversation history, 0.27 for memory files, 0.20 for retrieved memory, and 0.30 for skills. 60–73% of memory files and 92% of skill notes record the old version's fields or values.
    • Notes cost less: the notes group looks up the document 24–34% of the time, versus 2% for the conversation group; the authors note that the reason was not tested and cite Hu et al. (2026) as saying abstract memory transfers more reliably.
    • More experience is worse (Table 7): with 1, 3, and 10 prior tasks, the out-of-date experience success rate falls from 46% to 33% and 26%, valid experience stays at 97–100%, and the lookup rate falls from 4% to 0%.

    Remedies and handwritten history

    • Telling agents about the change is almost ineffective: a version number plus update notification raises the success rate only from 46% to 50% (Table 6); adding a version label to the memory file lowers cost only from 0.27 to 0.20 (Table 5).
    • Pointing to the document works only when it is attached to the request and the document has the answer: a document-reading instruction in the system prompt lowers the success rate to 29% (prior-task compliance 98%, final task only 4%); the same instruction attached to the task message makes all agents look up the document, with a 99% success rate; placing the document itself in the task message gives 99%, while placing it in the system prompt or tool description gives 54–55%. When the document is incomplete, the instruction makes everyone look it up but the document has no answer, and the success rate remains 25% (Table 16).
    • Post-write verification works but doubles calls: the verification instruction raises the success rate to 91% (complete document) and 69% (incomplete document), with 6.5 and 10.6 model calls, versus 2.9 and 3.1 for valid experience; having writes return the stored object raises the success rate from 45% to 79% (4.6 calls), and of the remaining 77 failures, 59 received an object showing an error state and did not write again (Appendix H).
    • Handwritten history masks the problem (Figure 6, Table 7): with handwritten history containing a read-back step, the out-of-date success rate is 88% (the agent's own prior tasks only 46%), and measured cost is about 0.11 versus 0.54; removing the read-back step lowers the read-back rate to 17% and the success rate to 58%.

    Other ablations and analyses

    • By model (Table 8): Qwen3.8-Flash has an out-of-date experience success rate of 1.00, while MiniMax-M3 has only 0.05; all models have a success rate of at least 97% when the document is placed in the task message.
    • By change (Table 9): the reply-default change has no cost (out-of-date experience success rate 1.00, as agents explicitly set scope every time).
    • Changes visible in the tool definition are reused with the old usage less often: old usage is 69%, versus 99% for changes described only in the document (Section 9).
    • Adding a document-reading instruction under valid experience: the success rate stays at 100%, but model calls rise from 2.92 to 3.55 and input tokens from 3,839 to 5,361 (Appendix I).
    • The excluded episodes are concentrated in the weekday version (0=Monday) of three models, where prior tasks never succeeded; their weekday estimates are conditional on prior-task success (Limitations).
  5. What are some points that can be further explored?

    The authors list limitations including synthetic services, short tasks, and an always-correct document; the experiments cover six models, three services, and eight kinds of changes.

    Limitations and future directions noted by the authors

    • The services are synthetic and small-scale, and each pair of versions differs in only one interface detail; real changes often occur in groups and some produce partial errors, and the authors state that choosing a single silent change was to isolate the effect of experience (Limitations).
    • Tasks are short, and experience consists of only 1 to 10 prior tasks of the same kind (Limitations).
    • Under the complete-document condition, the document is always correct and obtainable in one call; in practice, documents may lag behind the service (Limitations).
    • Under the incomplete-document condition, prior requests directly state the meaning of values, and experience contains explicit rules; experience that the agent must discover by itself was not tested (Limitations).
    • Six models run the same minimal agent loop; agent frameworks with their own planning or verification steps may perform differently (Limitations).
    • Most remedies were tested only under the condition where experience is stored in the conversation history (Limitations).
    • The authors state that the reason agents stop checking has not been established; the data are consistent with experience-following, and the placement of text has an effect (Discussion/Mechanism).
    • Regarding tool design, the authors state that rejecting unknown fields can turn silent changes into errors, but this was not tested (Discussion).

    Scope of the experiments

    • The models tested are GLM-5.3-Flash, Muse-Glimmer-30B, DeepSeek-V4-Flash, MiniMax-M3, Qwen3.8-Flash, and Inkling, 6 in total, all using the same simple agent loop (Section 3.3).
    • The environment consists of 3 self-built services and 8 kinds of changes, each run in both directions; 12,624 episodes in total, 2,104 per model (Table 2).
    • Remedy evaluations cover: version numbers and update notifications, document-reading instructions (two positions), three document placement positions, post-write verification instructions, writes returning the stored object, and memory files with version labels or the current document (Section 7).
    • Memory forms cover four kinds: conversation history, memory files, retrieved memory, and skills; experience amounts cover 1, 3, and 10 prior tasks (Section 6).
    • The incomplete-document condition was not preregistered; standard errors treat models and changes as fixed and do not cover untested services or differences between models, and the authors state that this may be optimistic (Limitations).
  6. Summarize the main content of the paper

    Under silent tool changes, an agent's successful experience becomes a hidden risk: reusing old calls, not checking documentation, and reporting errors as success.

    An empirical study on agent memory risk: when tool interfaces change silently, an agent's own past successful experience becomes a source of errors, which the authors name Experience Rot.

    • Problem: agents reuse experience from conversation history, memory files, retrieved notes and skills, but this relies on tools staying unchanged; real interfaces change silently (old calls are accepted, unknown fields are ignored, no error is raised).
    • Method: the authors built three services—calendar, task list, mail client—each with A/B versions differing in only one interface detail; the agent first completes a prior task in real time to gain experience, then performs the final task, compared across three groups: no experience, valid experience, out-of-date experience; six models, 12,624 episodes, program-determined success, most conditions preregistered.
    • Main results: with incomplete documentation, valid experience raises the success rate from 63% to 100%, while out-of-date experience lowers it to 23%; with complete documentation, the no-experience success rate is 88–100%, and out-of-date experience lowers it to 26–49%, and the agent still writes old fields even when the tool definitions list the new ones (Table 4).
    • Mechanism: with experience, the proportion checking documentation before the first write is at most 6% (70–100% without experience); after writing old usage, 94 of 97 read-back cases recover, while 183 of 184 cases without read-back fail; 347/348 failed runs ultimately report task completion to the user.
    • Remedies: telling the agent about the change is almost ineffective (50%); documentation or a read-documentation instruction is effective only when placed in the task message and the documentation contains the answer (99%); post-write verification reaches 91% but raises model calls from 2.9 to more than 6.5.
    • Evaluation implications: with researcher-written history including a read-back step, the measured cost is about 0.11, whereas with the agent's own experience it is 0.54; the authors state that memory evaluations should use the agent's own trajectories and include incomplete documentation.
    • Authors' conclusion: storing "what worked" is not enough; memory systems also need a mechanism to detect that stored usage has become invalid; when documentation may be incomplete, verifying results after writing is the more reliable choice.
Read originalarxiv.org