When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization
When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization
The authors propose BadEvolve; merely injecting an evolution task is enough to make a self-evolving LLM agent write a backdoor into its own skills; on Kimi-K2.5, PI averages 79.6% ASR.
The attacker can inject at most B poisoning tasks into the evolution task set (controlling their instructions and reference answers), knows the self-evolution mechanism, and has access to D_evolve and the validation set D_val; cannot access D_test, the victim's backbone LLM, the evolved skills, or the solver/evolver trajectories; at test time can only attach a trigger to the target task.
- Self-evolving agents solidify execution experience into reusable skills; once a malicious rule is implanted into a skill, it persistently affects subsequent behavior; existing skill backdoor attacks assume skills can be directly modified, but this assumption does not hold when skills are built internally by a trusted evolver.
- Proposes BadEvolve: the attacker injects only poisoning evolution tasks, and an LLM optimizer iteratively rewrites the shared poisoning instruction; the attack is decomposed into four stages—Induction/Activation/Following/Effect—to provide staged feedback, and a failure-aware gate with rollback and failure memory is used to stabilize optimization.
- Across four self-evolution frameworks and three benchmarks, BadEvolve achieves an average ASR of 79.6% under PI (Kimi-K2.5), versus 37.6%/27.8% for the baselines TBA/TextGrad-Poison; under AC, DASR on WikiTQ×A-Evolve reaches 80.0% (Kimi-K2.5), with CACC and CACC0 differing within 0.2 pp.
- The authors state that future work will extend to other harness components such as memory and prompt and study defenses; experiments are conducted only on three table/document benchmarks and four backbones, the PI main results use only Kimi-K2.5, no defense is evaluated, and the query cost of attack optimization is not reported.
- Evaluated models
- Kimi-K2.5GPT-OSS-120BClaude Sonnet 4.6
- Benchmarks
- SpreadsheetBenchWikiTQOfficeQA
- Metrics
- ASRDASRCACCTACCCACC0
The researchers propose BadEvolve, a task-level backdoor attack framework in which the attacker can only inject poisoned tasks into the evolving task set, inducing a self-evolving LLM agent to write trigger-conditioned malicious rules into its own skills. The method breaks attack failure into four stages—Induction, Activation, Following and Effect—to locate problems, and uses rollback and failure memory to decide whether to retain each task update. Experiments cover four self-evolving frameworks—Trace2Skill, SkillGrad, SkillOpt and A-Evolve—and three benchmarks: SpreadsheetBench, WikiTQ and OfficeQA; with Kimi-K2.5 as the victim backbone, Payload Injection achieves an average attack success rate of 79.6%, compared with 37.6% for TBA and 27.8% for TextGrad-Poison, while clean-task accuracy is on average only 1.1 percentage points lower than unpoisoned evolution.
Why it was selectedThe paper presents attack success rates and utility costs of task-level poisoning across four self-evolution frameworks and multiple benchmarks, which can help understand the poisoning surface of skill self-evolution pipelines.
Paper analysis
What problem does this paper attempt to solve?
Studies task-level backdoors: the attacker injects only evolution tasks, causing the self-evolving agent to write the backdoor into its own skills.
Core problem: how to implant a backdoor into skill-based self-evolving LLM agents without directly modifying skills—the attacker can only inject poisoning evolution tasks, inducing the agent to write the malicious rule into skills itself.
- Setting and importance: in self-evolving agents, the solver executes evolution tasks and the evolver distills execution experience into persistent reusable skills. The authors argue that once a skill carries a malicious rule, it can repeatedly trigger malicious behavior in subsequent tasks without further attacker involvement, and illustrate the risk with a financial audit agent underreporting suspicious transactions.
- Shortcomings of existing methods: existing backdoor attacks on skill-based agents assume the attacker can directly manipulate the skill artifacts adopted by the victim; the authors point out that in the self-evolution setting skills are built internally by a trusted evolver, so this assumption rarely holds.
- Core observation: the natural approach is to iteratively optimize poisoning tasks by attack success rate, but the authors' preliminary analysis (Sec. 3.4) finds two challenges: (i) failures at different stages of skill induction and downstream execution produce indistinguishable end-to-end results, which cannot guide targeted updates; (ii) the stages are coupled, so failed updates may cancel existing progress, while discarding them outright repeats ineffective updates.
- Proposed: BadEvolve, a failure-aware task poisoning framework: it decomposes the attack into four stages—Induction, Activation, Following, Effect—to diagnose where failure occurs, and uses a failure-aware gate with rollback plus failure memory to stabilize optimization.
- Threat model (Sec. 3.2):
- Goal: make the solver exhibit the attacker-selected behavior on target tasks with triggers while remaining normal on benign tasks; the two attack objectives are Payload Injection (PI, attaching extra behavior without affecting correctness) and Answer Corruption (AC, making answers incorrect).
- Knowledge: knows the self-evolution mechanism, has access to the evolution task set D_evolve and a validation set D_val disjoint from both D_evolve and D_test.
- Capability: within budget B, can only inject poisoning tasks into D_evolve (controlling their instructions and reference answers); at test time can only append triggers to target tasks.
- Victim system: a skill-based self-evolving agent; the attacker cannot access D_test, the victim backbone LLM, the evolved skills, or the solver/evolver trajectories.
What related research exists?
Related work falls into two categories: self-evolution methods and agent backdoor attacks; baselines are TBA and TextGrad-Poison.
Self-evolution of LLM agents
- Early experience-reuse methods: Reflexion stores linguistic reflections on failed attempts, ExpeL distills reusable experience from trajectories, and Agent Workflow Memory induces workflows from historical executions; Voyager maintains an executable skill library that grows with exploration (Appendix A).
- Skill-based self-evolution: Trace2Skill distills local trajectory experience into transferable skills; SkillGrad iteratively patches skills according to execution feedback in a manner similar to gradient descent; SkillOpt compiles verifier-grounded skills from trajectories; A-Evolve evolves the harness (including skills) through an iterative solve–evolve loop.
- The paper notes that these methods assume execution experience is trustworthy by default, whereas this paper studies how this experience-to-skill pipeline can be exploited to implant backdoors.
Backdoor attacks against LLM agents
- Backbone and memory poisoning: BadAgent and others poison the backbone model via fine-tuning; AgentPoison and others poison the agent's memory or knowledge base so that malicious records are retrieved.
- Skill artifact poisoning: SkillTrojan and BadSkill embed backdoors directly into skill artifacts (executable skill code, bundled models).
- Parallel self-evolution poisoning work: PoisonedEvolution and SkillJack inject attacker-constructed trajectories into the evolution pipeline, requiring write access to the experience consumed by the evolver; TBA induces poisoned trajectories through attacker-constructed tasks, but uses a fixed hand-crafted strategy.
Baselines and benchmarks
- Baselines: TBA (a fixed hand-crafted strategy constructs poisoning tasks, each poisoning task carrying a trigger and a counterfactual demonstration of the malicious behavior plus a rule statement) and TextGrad-Poison (adapted from TextGrad's result-driven optimization, looking only at end-to-end attack success rate, with no staged feedback, rollback, or failure memory).
- Benchmarks: SpreadsheetBench, WikiTQ, OfficeQA; the self-evolution frameworks are Trace2Skill, SkillGrad, SkillOpt, A-Evolve.
The authors position this paper's differences as: (i) a more realistic threat model—only evolution tasks can be injected, with no access to victim skills or the experience consumed by the evolver; (ii) instead of a fixed hand-crafted strategy, staged feedback and a failure-aware gate are used to automatically optimize poisoning tasks.
How does the paper address this problem?
BadEvolve: an LLM optimizer iteratively rewrites the poisoning instruction, four-stage feedback locates failures, and rollback plus failure memory stabilize optimization.
Overall idea: the attacker does not directly modify skills; instead, it optimizes a shared poisoning instruction φ, appends it to B benign evolution tasks to form poisoning tasks; a surrogate agent runs self-evolution and is evaluated on the validation set, and an LLM attack optimizer iteratively updates φ. Framework name BadEvolve (Figure 3).
Problem Formulation
- Attack success indicator: under the PI objective, success is when the output exhibits the attacker-selected behavior; under the AC objective, success is when the task answer with the trigger is incorrect (Eq. 3).
- The optimization objective is to maximize the average attack success rate Jobj(𝒟poison) on the target task set under the budget constraint |D_poison| ≤ B (Eq. 4), i.e., selecting poisoning tasks so that the evolved poisoned skills trigger attack behavior on target tasks with triggers.
Poisoning Task Construction and Feedback-Driven Optimization
- Each poisoning task consists of a benign task with a trigger δ appended and a poisoning instruction φ appended; under PI the reference answer remains the correct answer, under AC it is an incorrect answer constructed according to predefined rules; the same φ is applied to all B tasks (Eq. 5).
- In each iteration round, the surrogate agent (solver and evolver) self-evolves on the poisoned evolution set to obtain skill S(i), then uses that skill to execute validation tasks with triggers and estimate the attack success rate (Eq. 6).
- The LLM attack optimizer q proposes updates based on the current φ, the score, and supporting evidence (the evolved skills, solver and evolver trajectories) (Eq. 7).
Stage-wise Feedback
- The propagation of the malicious rule is decomposed into four sequential stages: Induction (the rule is written into the skill), Activation (the task with trigger loads the skill containing the rule), Following (the solver executes the behavior described by the rule), Effect (the attack outcome is produced); each stage is judged by a binary indicator.
- The conditional pass rate of each stage is estimated on the validation set (the proportion passing this stage given that all previous stages passed, Eq. 8), and provided to the optimizer together with the score and evidence (Eq. 9), so that it updates targeting the failing stage rather than blindly rewriting.
Failure-aware Gate: Rollback and Failure Memory
- Rollback: the optimizer's output serves only as a candidate; it is retained only when the change in end-to-end success rate reaches the threshold ϵ, or when the change is less than ϵ but the stage-progress score improves; otherwise it rolls back to the previous candidate (Eq. 10). The stage-progress score g is defined as the sum of the cumulative pass rates of the stages.
- Failure memory: rejected candidates, together with their success-rate change, stage pass-rate change, and the optimizer's rationale, are recorded into memory (Eq. 11) and provided to the optimizer in subsequent updates (Eq. 12), to avoid repeating similar ineffective updates.
Algorithm and Key Parameters
- The procedure is in Alg. 1 of Appendix C: after evaluating the seed instruction φ(0), loop for L rounds; each round proposes a candidate, evaluates it, retains or rolls back according to the gate and records it, and finally outputs the poisoned evolution task set.
- Default parameters: poisoning budget B = 20, optimization rounds L = 5, gate threshold ϵ = 0.4, each candidate is evaluated with K = 5 surrogate self-evolution runs; the attack optimizer uses Claude Sonnet 4.6; φ(0) is a plain-text instruction that directly states the attack behavior (Appendix D.5).
What experiments did the paper conduct?
Average ASR under PI across four frameworks × three benchmarks is 79.6% (baselines 37.6%/27.8%), and DASR under AC reaches up to 80.0%.
Experimental Setup
- Backbones under test: Kimi-K2.5, GPT-OSS-120B, Claude Sonnet 4.6, Qwen3-235B; by default the solver and evolver share the same backbone, and the surrogate and victim share the framework and backbone.
- Self-evolution frameworks: Trace2Skill, SkillGrad, SkillOpt, A-Evolve.
- Datasets: SpreadsheetBench (SB), WikiTQ, OfficeQA; each benchmark is split into evolution/validation/test sets (40/40/100, OfficeQA evolution is 50, Table 6), and poisoning tasks are injected only into the evolution set.
- Baselines: TBA, TextGrad-Poison.
- Metrics: PI reports CACC (accuracy on clean test tasks), TACC (accuracy on tasks with triggers), ASR (proportion of tasks with triggers exhibiting attack behavior); AC reports CACC, TACC, DASR (proportion answering correctly without trigger but incorrectly with trigger); CACC0 is the test accuracy after evolving only on the clean evolution set.
- Repetitions: each attack performs 3 independent self-evolution runs starting from S0, and results are averaged; in the optimization phase each candidate is evaluated with K = 5 surrogate runs.
Main Results
PI main results (Table 2, victim backbone Kimi-K2.5, ASR unit %):
Swipe horizontally to read the table
Framework Method SB ASR WikiTQ ASR OfficeQA ASR Trace2Skill TBA / TextGrad-Poison / BadEvolve 27.0 / 0.0 / 92.0 18.7 / 30.0 / 69.0 46.3 / 16.7 / 69.3 SkillGrad TBA / TextGrad-Poison / BadEvolve 30.3 / 33.0 / 90.7 0.0 / 33.0 / 96.7 1.3 / 30.0 / 62.3 SkillOpt TBA / TextGrad-Poison / BadEvolve 97.3 / 28.0 / 100 33.3 / 0.0 / 66.7 47.0 / 37.6 / 85.3 A-Evolve TBA / TextGrad-Poison / BadEvolve 31.3 / 26.3 / 59.3 95.7 / 98.7 / 98.7 23.3 / 0.0 / 65.7 - Authors report: average ASR for TBA and TextGrad-Poison is 37.6% and 27.8% respectively, BadEvolve averages 79.6% (range 59.3%–100.0%); CACC averages only 1.1 pp lower than CACC0, and TACC drops at most 3.3 pp relative to CACC.
- Exceptions: on A-Evolve×WikiTQ, BadEvolve (98.7%) ties with TextGrad-Poison and is only slightly higher than TBA (95.7%); on SkillOpt×SB, TBA already reaches 97.3%.
- AC results (Table 3, WikiTQ×A-Evolve): on Kimi-K2.5, BadEvolve's TACC drops 66.4 pp relative to CACC0, DASR 80.0% (best baseline 68.5%); on Claude Sonnet 4.6 it drops 45.2 pp, DASR 54.1% (best baseline 18.0%); CACC differs from CACC0 by within 0.2 pp. Appendix Table 9: on Trace2Skill×WikiTQ (Kimi-K2.5), DASR is 34.0%, baseline maximum 14.1%.
Surrogate Backbone Transfer (RQ2)
- Setup: the victim is fixed as Kimi-K2.5, the surrogate backbone is changed to GPT-OSS-120B, Claude Sonnet 4.6, Qwen3-235B, with frameworks SkillOpt and Trace2Skill (Table 4).
- Results: average ASR 84.6% (range 59.7%–100.0%), CACC and TACC differ from CACC0 by within 4.6 pp; on WikiTQ×SkillOpt, when using GPT-OSS-120B or Qwen3-235B as surrogate, ASR is 100.0%, higher than the 66.7% when using the victim's own backbone.
- Authors' interpretation: BadEvolve remains effective and preserves utility in the gray-box setting where the victim backbone is unknown.
Ablation and Budget Analysis (RQ3)
- Ablation (Figure 4, Trace2Skill/SkillGrad×WikiTQ/OfficeQA, Kimi-K2.5): removing stage-wise feedback (w/o SF) drops ASR to 0.0% on OfficeQA×SkillGrad; removing rollback (w/o RB) drops ASR to 33.3% on WikiTQ×Trace2Skill; removing failure memory (w/o FM) drops ASR to 0.0% on OfficeQA×SkillGrad; CACC and TACC of each variant are close to CACC0.
- Poisoning budget (Figure 5, Trace2Skill, B ∈ {1,5,10,20,30,40}): the authors state that under both settings (WikiTQ×Kimi-K2.5, SB×Claude Sonnet 4.6) ASR generally rises as B increases, and CACC and TACC remain close to CACC0; the figure contains only axis and legend text, with specific values not labeled.
- Case (Table 8, WikiTQ×SkillOpt×Kimi-K2.5): TextGrad-Poison attributes low ASR each round to missing instructions, length, or wording, but the actual bottleneck is Induction; BadEvolve locates Induction via stage feedback (p_I = 0.4), and in round 2 raises p_I to 0.8 and validation-set ASR from 40.0% to 79.5%.
Other Analyses
- Motivation experiment (Table 1, TextGrad-Poison, GPT-OSS-120B): average ASR across 8 settings is 28.8%, of which 3 are 0.0%; Figure 2 shows that on SkillGrad×WikiTQ ASR falls from 53% in round 2 to 0% in round 3 and does not recover.
- Optimization stability (Figure 6, SB×Trace2Skill×Kimi-K2.5): the ASR of candidates retained by BadEvolve rises monotonically from 77.5% to 87.5%, and candidates in rounds 2 and 4 with ASR near 0% are rolled back.
- AC mechanism (Appendix E.2, Table 10): the poisoning instruction frames answer tampering as the final step of the solution, and the evolver distills it into a trigger-conditioned step; in one evolution, of 87 test tasks answered correctly without trigger, 80 are answered incorrectly after adding the trigger.
What are some points that can be further explored?
The authors state that future work will extend to components such as memory and prompt and study defenses; experiments cover three benchmarks and four backbones.
Limitations and future directions noted by the authors
- Extending to other harness components: The authors plan to extend BadEvolve to other harness components such as memory and prompt (Sec. 6 Conclusion and Future Work).
- Studying defenses: The authors plan to study defenses against this type of task-level backdoor attack (Sec. 6).
- The paper does not have a dedicated Limitations section; the above two items are the future directions explicitly stated in the conclusion.
Scope of experimental coverage
- Backbones tested: 4 in total: Kimi-K2.5, GPT-OSS-120B, Claude Sonnet 4.6, Qwen3-235B; the PI main results (Table 2) use only Kimi-K2.5, the AC main results (Table 3) use Kimi-K2.5 and Claude Sonnet 4.6, and GPT-OSS-120B and Qwen3-235B appear only in the motivation experiment (Table 1) and the surrogate transfer experiment (Table 4).
- Benchmarks: 3 in total: SpreadsheetBench, WikiTQ, OfficeQA, all spreadsheet/document-type tasks; the AC experiments are conducted only on WikiTQ (Appendix D.5 states the reason is that its answers are short table values).
- Self-evolution frameworks: 4 in total: Trace2Skill, SkillGrad, SkillOpt, A-Evolve; AC covers A-Evolve (main text) and Trace2Skill (Appendix E.1).
- Defense evaluation: The paper does not evaluate any defense method; defenses are proposed only as future work.
- Attack cost: The paper reports settings such as B = 20, L = 5, K = 5, ϵ = 0.4, but does not report the total number of queries or time required for attack optimization; evaluation uses each benchmark's built-in rule-based scoring (SpreadsheetBench compares target cells, WikiTQ/OfficeQA use exact match, Appendix D.1), with no LLM judge.
Summarize the main content of the paper
BadEvolve can implant a backdoor into a self-evolving agent by injecting only evolution tasks, with an average PI ASR of 79.6% and small utility loss.
A paper on task-level backdoor attacks against self-evolving LLM agents.
- Problem: skill-based self-evolving agents distill execution experience into persistent skills; existing skill backdoor attacks assume the attacker can directly modify skill artifacts; this paper studies the setting where the attacker can only inject poisoning evolution tasks and the victim itself learns the malicious rule into skills.
- Method: BadEvolve uses an LLM optimizer to iteratively rewrite a shared poisoning instruction, decomposes the attack into four stages—Induction, Activation, Following, Effect—to provide stage-wise feedback, and uses a failure-aware gate with rollback plus failure memory to stabilize optimization; defaults are B = 20 poisoning tasks, L = 5 rounds, K = 5 surrogate self-evolution evaluations per candidate.
- PI results: with Kimi-K2.5 as victim backbone, average ASR across four frameworks×three benchmarks is 79.6% (range 59.3%–100.0%), versus 37.6%/27.8% for baselines TBA/TextGrad-Poison; CACC averages only 1.1 pp lower than clean evolution (Table 2).
- AC results: on WikiTQ×A-Evolve, DASR reaches 80.0% (Kimi-K2.5) and 54.1% (Claude Sonnet 4.6), with best baselines 68.5% and 18.0% respectively (Table 3).
- Transfer and ablation: when the victim backbone is unknown, switching to other surrogate backbones gives average ASR 84.6% (Table 4); removing stage-wise feedback, rollback, or failure memory drops ASR to 0.0% or 33.3% in some settings (Figure 4).
- Authors' conclusion: stage-wise feedback and the failure-aware gate are key to an effective attack that preserves victim utility; the authors plan to extend to components such as memory and prompt in the future and study corresponding defenses.