Security of self-evolving LLM agent systems: threats, amplification effects, and case studies

Core problem: self-evolving LLM agents (which can autonomously update model parameters, memory, tools and architecture) introduce an entirely new threat landscape, but existing security analyses are all based on static attack-surface assumptions and lack a unified framework to characterize their full attack surface.

• Problem scenario: the authors point out that self-evolving agents continuously modify themselves through self-reward training, experience accumulation and self-referential frameworks; a single prompt injection may be written into long-term memory, distilled into weights, or encoded as a new tool, persisting across evolution cycles without the attacker needing to maintain access.
• Shortcomings of existing methods: the authors argue that traditional defenses (input filtering, sandbox boundaries, alignment constraints) rely on three assumptions that self-evolution breaks—the system is static, the trust anchor is immutable, and the threat is confined to a single session; existing surveys (such as the OWASP Top 10 and agent safety surveys) do not cover the evolution dimension.
• Core observation: the authors propose that self-evolution changes the nature of threats along three axes—from transient to persistent, from single-point to self-propagating, and from target to propagation vector; the agent's own evolution mechanism becomes attack infrastructure, and a T1-level (user interface only) attacker who succeeds once in poisoning can obtain persistent state compromise that traditionally required T4-level supply-chain penetration (the authors call this tier compression).
• Contributions of the paper: propose the MLAS (Module–Lifecycle Attack Surface) matrix (5 modules × 5 lifecycle stages, 25 cells in total), seven cross-module amplification effects, and a comparative case study of two open-source frameworks, OpenClaw and Hermes.
• Threat model (§2.3.1, terminology copied verbatim):
  • Goal: compromise the behavioral, safety properties or privacy guarantees of a self-evolving agent, classified by CIA+P (confidentiality, integrity, availability, privacy).
  • Knowledge: black-box or gray-box—can observe outputs and may know the general architecture, but does not need to know specific weight values or internal state.
  • Capability: can influence at least one input channel that enters the evolution loop (task input, environment observation, retrieved documents, tool responses, evaluation feedback, inter-agent messages), across five access tiers T1–T5; does not directly access model weights or training infrastructure; may be transient (single interaction) or persistent (multiple interactions across cycles).
  • Victim system: a self-evolving LLM agent system satisfying the three conditions of directed optimization, cross-session persistence, and autonomous control (Table 2).

Autonomous Agent Surveys

• Wang et al. (2024), Xi et al. (2023)—establish architectural taxonomies for LLM autonomous agents (brain–perception–action or planning–memory–tool) and survey applications. The paper notes that such work does not systematically analyze security threats, and security discussion is usually limited to subsections on prompt injection or hallucination.

Agent Security Surveys

• Dehghantanha et al. (2026)—systematizes the attack surface of agent tools and autonomy, and proposes metrics such as Unsafe Action Rate; the paper notes that it does not address the evolutionary dimension.
• Kim et al. (2026)—proposes a design-space framework covering tool, knowledge, and autonomy attack surfaces; Yu et al. (2025)—proposes the TrustAgent framework, decomposing trustworthiness into intrinsic (brain, memory, tools) and extrinsic (user, agent, environment) dimensions.
• The paper notes that the common problem of these surveys is treating agents as static systems, analyzing attacks on fixed architectures at a single point in time, and not modeling how evolutionary loops turn transient attacks into persistent, self-reinforcing, cross-generationally propagating damage.

Self-Evolving Agent Surveys and Empirical Work

• Tao et al. (2024)—surveys LLM self-evolution with a four-stage loop (experience acquisition, refinement, update, evaluation); Gao et al. (2025)—provides a taxonomy around what/when/how to evolve. The paper notes that both focus on evolution mechanisms and capabilities, with security mentioned only briefly as a future direction.
• Shao et al. (2026)—the paper states that it provides the first empirical evidence of misevolve in self-evolving agents (unsafe behavior arising from unguided experience accumulation), but it covers only the single paradigm of experience-driven contextual evolution and does not systematically cover all evolution modules and lifecycle stages.
• Safety fine-tuning and model merging (Safe LoRA 2024, DAM 2025, etc.)—the paper notes that they handle isolated components and do not analyze cross-module interactions.

Comparison Systems and Attacks Involved in the Case Study

• The case subjects are the open-source frameworks OpenClaw (evolution-augmented, over 160,000 GitHub stars) and Hermes (evolution-native, 140,000 stars within three months); attack scenarios are organized with reference to the OWASP ASI Top 10 and the threat taxonomy of Kim et al.

Positioning: The authors state that this paper is the first systematic security and privacy analysis specifically targeting self-evolving LLM agent systems, distinct from security surveys for static agents and surveys of evolutionary capabilities (Figure 3); the differences lie in full coverage of the 25 cells of the MLAS matrix, identification of evolution-specific transformation mechanisms, and empirical evidence on deployed systems.

Overall approach: first formalize self-evolving agents and the evolutionary lifecycle, and construct the MLAS matrix as the analytical skeleton; analyze each cell in a three-part structure of "exposed interface → threats and attacks → evolution-specific transformation effects"; then summarize cross-module amplification effects and defense principles; finally provide empirical evidence through a comparative case study of two open-source frameworks.

Formalization and Scope Definition

• Agent state: the state at evolution step t is a four-tuple θ_t = (M_t, C_t, T_t, W_t), corresponding to model parameters, non-parametric cognitive resources (system prompt, persistent memory, few-shot examples, workflow templates, user profile), tool/skill library, and architecture configuration; the evolutionary update is θ_t+1 = f(θ_t, τ_t, r_t), where τ_t is the interaction trajectory and r_t is the feedback signal, and part of f itself can be a component of θ (self-reference).
• In-scope conditions: three conditions must be satisfied simultaneously—directed optimization (presence of an explicit or implicit fitness signal), cross-session persistence, and autonomous control (no step-by-step human approval required); Table 2 accordingly excludes static agents, append-only memory agents, and human-in-the-loop tuning.
• Five evolutionary paradigms: model evolution, cognitive resource evolution, execution evolution, self-design evolution, and collective evolution, distinguished by the state component being modified.

MLAS Matrix and Cell-by-Cell Analysis Method

• Matrix structure: 5 functional modules (Brain, Cognitive Resource, Execution, Self-Design, Collective) × 5 lifecycle stages (Bootstrap, Propose, Evaluate, Commit, Serve), totaling 25 cells (Table 6); each cell is annotated with exposed interfaces and representative threats, with severity in five levels: catastrophic / critical / high / moderate / low (Figure 1).
• Three-part analysis: corresponding to three research questions—RQ1 identifies the exposed interfaces of each cell, RQ2 analyzes evolution-specific transformation effects (persistence, self-reinforcement, cross-generation propagation), and RQ3 catalogs threats and attacks and identifies defense failure points.
• Threat grading conclusions: the authors state that of the 25 cells, 17 are critical (no effective defense), 7 are high (existing defenses insufficient), and only 1 can be partially mitigated; the Self-Design row is catastrophic as a whole due to optimizer–optimizee collapse.

Seven Cross-Module Amplification Effects

• Seven effects (Table 14): Generational Accumulation (compounding accumulation of degradation across generations), Selective Amplification (evaluation systematically rewarding capability rather than safety), Deceptive Evolution (deceptive capability itself being evolutionarily optimized), Lamarckian Propagation (acquired experience directly inherited), Capability Ratchet (capability only increases, never decreases), Emergent Unpredictability (capability combinations producing unforeseen behavior), Optimizer–Optimizee Collapse (the system optimizing the whole, including its own safety mechanisms).
• Synergy chains: the authors identify four synergy chains, such as Lamarckian Propagation feeding into Generational Accumulation, and Optimizer–Optimizee Collapse acting as a meta-effect amplifying all the others (Figure 15).
• Defense principles: from three failure assumptions (static system, immutable trust anchor, session scope), four principles are derived—evolution-aware monitoring, immutable safety invariants, multi-generational audit trails, and defense coverage matched to the attack surface.

Comparative Case Study Design

• Case selection: OpenClaw and Hermes are selected according to four criteria: functional comparability, divergent evolution strategies, auditability, and ecosystem importance; both are self-hosted multi-channel AI assistants, but OpenClaw is evolution-augmented (learning requires explicit gating, artifacts are non-executable data, unified scanning), while Hermes is evolution-native (learning is automatically triggered on every interaction, artifacts are executable code, internal generated artifacts are exempt from scanning).
• Four-step process: architecture mapping to matrix cells → runtime security evaluation → cell-by-cell attack case construction → cross-case pattern analysis.
• Experimental configuration (Table 8): backbone model GPT-5; 10 attack scenarios per CIA+P category, 40 in total; each scenario is executed on Hermes's Path A (hub installation, scanned) and Path B (Background Review, not scanned) and the corresponding OpenClaw paths, totaling 160 runs; each scenario is repeated 3 times to control for sampling randomness.

Experimental Setup

• Systems under test: OpenClaw v2026.6.2 (TypeScript/ESM, Node.js 22+, MIT) and Hermes v0.15.1 (NousResearch, Python, MIT), both the latest stable versions as of June 2026 (Table 8).
• Backbone model: GPT-5; both runtime attacks and architecture-level cases invoke the agent end to end (Table 8).
• Attack scenarios: 40 standardized attack scenarios, 10 each for Integrity / Confidentiality / Availability / Privacy (CS-I1–I10, CS-C1–C10, CS-A1–A10, CS-P1–P10), mapped to 20 different MLAS cells.
• Paths: Hermes is divided into Path A (hub installation, static security scanning enabled) and Path B (Background Review autonomous evolution path, which does not invoke the scanner); OpenClaw uses the plugin installation path (unified scanning + manual approval queue).
• Run volume and repetitions: 40 scenarios × 2 paths × 2 frameworks = 160 runs; each scenario independently repeated 3 times to control LLM sampling randomness (Table 8).
• Success criterion: Attack success is defined as payload persistence (persisted); Tables 10–13 mark with ✓ (successful and persisted), ✗ (intercepted by the scanner), △ (partial success), and scores are the success rate across 3 repetitions. The paper does not state who or what program determined persistence.

Main Results

According to Figure 10 and Tables 10–13 (10 payloads per category × 3 times, GPT-5 backbone):

Swipe horizontally to read the table

Framework / Path	Integrity	Confidentiality	Availability	Privacy
Hermes Path B (evolution path, unscanned)	10/10 persisted	10/10	10/10	10/10
Hermes Path A (scanning path)	intercepted 1 (CS-I8), partially detected 1 (CS-I5)	intercepted 0	partially detected 1 (CS-A2)	partially detected 1 (CS-P7)
OpenClaw (approval gating)	all intercepted	all intercepted	all intercepted	all intercepted

• Overall numbers: The authors report a Path B attack persistence rate of 100% (40/40); Path A's LLM scanner intercepted only 1/40 (2.5%), with 3 additional partial detections (CS-I5, CS-A2, CS-P7, marked in Figure 10 as flagged in 1/3 runs but still successful); OpenClaw intercepted all 40 payloads through the architectural approval queue (100% interception), without relying on any detection heuristic.
• Authors' interpretation: The authors call this a "double failure"—the evolution path architecturally bypasses the scanner entirely, and LLM-mediated skill synthesis "washes" adversarial payloads into a form that evades detection; OpenClaw's interception comes from architectural gating rather than detection capability.
• Architecture mapping: Hermes activated 4 cells beyond the static baseline (Memory×Propose, Execution×Propose, Self-Design×Evaluate, Self-Design×Commit), and OpenClaw activated 2 (Brain×Propose, Memory×Propose) (Table 9); the abstract states that the evolution-native design activates 3.5× the attack surface cells, while §8.8 states that Hermes activates 7 different cells in total, covering all CIA+P categories, and OpenClaw 2, limited to Integrity.

Representative Attack Cases (§8.3–8.6)

• CS-I1 (indirect prompt injection → persistent skill backdoor): Injected instructions in a retrieved document were synthesized by the Background Review Agent into a Python skill with conditional branches; the Path A scanner did not detect it, and Path B wrote it directly to disk; the same injection in OpenClaw degraded only to a retrieval bias issue (memory is non-executable JSON).
• CS-C1 (unsanitized snapshot → cross-session credential leakage): The Hermes memory system persists raw conversations directly without sanitization; OpenClaw's counterpart is a two-round sanitization pipeline (regex + LLM revision).
• CS-A1 (recursive skill creation → resource exhaustion): Skill creation has no rate limit, and new skills trigger more review loops; OpenClaw's Q-value threshold (minAbsReward=0.15) acts as a rate limit.

Cross-Module Amplification Factors and Pattern Analysis (§8.7–8.8)

• AF-1 bias-to-action: Hermes defaults to "action over deliberation," which the authors state lowers the threshold for all four attack categories; OpenClaw's Q-value gating (minAbsReward=0.15, minRewardConfidence=0.55) is the opposite, defaulting to no action.
• AF-2 persistence without decay: Neither framework has memory decay or time-limit mechanisms; the authors state that because OpenClaw's memory is non-executable, the impact is limited to retrieval quality degradation.
• Four patterns summarized by the authors: evolution-native design expands the attack surface across all security properties; the existence of security mechanisms does not equal security; session-level injection becomes a generational persistent threat; amplification factors are multiplicative rather than additive.

Other Ablations and Analyses

• The paper does not set up independent ablation experiments; the dimension-by-dimension code-level comparison in §8.1/Table 9 (learning triggers, skill persistence form, scan coverage, memory sanitization, evolution scope, feedback loops) serves the ablation role.
• The conclusion section has another statement: the Hermes scanner "blocks 5/8 payloads," which is inconsistent with 1/40 (2.5%) in §8.2; the paper does not explain the relationship between the two numbers.
• Negative result: Path A's interception rate for the 10 Confidentiality payloads is 0 (Table 11), which the authors explain as a fundamental mismatch between the scanner's behavior orientation and the data-exfiltration nature.

Limitations and future directions noted by the authors

The paper has no separate Limitations section; in §10 "Future directions" the authors explicitly list five priority research areas:

• Longitudinal safety monitoring (§10): existing evaluation frameworks assess agents only at a single point in time; cross-generation safety auditing is needed to track drift that stays below the detection threshold of any single generation but accumulates to a critical level.
• Evolution-aware defense architectures (§10): immutable safety invariants that are architecturally immune to evolutionary modification are needed, as are attack-surface-matched defenses that cover all injection channels simultaneously.
• Population-level safety guarantees (§10): the "Simpson safety paradox" (individuals are safe but the population evolves unsafe behavior) requires new theoretical frameworks and population-level monitoring tools.
• Formal verification of self-evolution (§10): the authors call SEVerA a preliminary attempt, but extending formal methods to real self-evolving systems remains unsolved; the hardest subproblem is the case where the verification mechanism itself is also within the scope of evolution.
• Privacy by design for evolving systems (§10): privacy frameworks that go beyond single-training differential privacy and provide guarantees on evolutionary timescales are needed.

Experimental coverage

• Systems under test: the case studies cover two open-source frameworks, OpenClaw v2026.6.2 and Hermes v0.15.1 (Table 8); the 25-cell analysis of the MLAS matrix (§3–7) is theoretical analysis, not empirically tested cell by cell.
• Backbone model: both the runtime attacks and the architecture-level cases use only one backbone model, GPT-5 (Table 8).
• Attack scale: 40 attack scenarios (10 each for CIA+P), each scenario repeated 3 times, 160 runs in total (Table 8).
• Review and adjudication: the paper does not report who performs the adjudication of attack success/persistence (human, rules, or LLM reviewer), nor does it report the construction process for attack payloads or inter-author agreement.
• Defense evaluation: the defense mechanisms actually tested are Hermes's LLM safety scanner (Path A) and OpenClaw's approval gating + unified scanning; the four defense principles in §9.9 are design recommendations, and the paper does not implement or validate them.

This is a systematic security and privacy analysis targeting self-evolving LLM agent systems, which the authors state is the first systematic analysis in this direction.

• Problem to solve: Self-evolving agents can autonomously update parameters, memory, tools, and architecture; attack effects are permanently encoded, self-reinforce across generations, and propagate within populations; the three assumptions on which existing defenses rely—static systems, immutable trust anchors, and session scope—all fail simultaneously, and previously there was no unified framework to characterize their full attack surface.
• Method highlights: Proposes the MLAS matrix (Brain, Cognitive Resource, Execution, Self-Design, Collective × Bootstrap, Propose, Evaluate, Commit, Serve, 25 cells in total), conducting a "exposed interface → threat → evolutionary transformation effect" analysis cell by cell; summarizes seven cross-module amplification effects (Generational Accumulation, Selective Amplification, Deceptive Evolution, Lamarckian Propagation, Capability Ratchet, Emergent Unpredictability, Optimizer–Optimizee Collapse) and four defense principles.
• Matrix conclusions: The authors state that of the 25 cells, 17 are critical (no effective defense), 7 are high, and only 1 can be partially mitigated; the Self-Design row is catastrophic across the entire row due to optimizer–optimizee collapse (Figure 1).
• Case study: For OpenClaw (evolution-augmented) and Hermes (evolution-native), each ran 40 CIA+P attack scenarios × 3 times (GPT-5 backbone): the Hermes autonomous evolution path (Path B) had an attack persistence rate of 100% (40/40), while the scanning path (Path A) intercepted only 1/40 (2.5%); OpenClaw intercepted all 40 through architectural approval gating. Hermes activated 4 matrix cells beyond the static baseline, and OpenClaw activated 2.
• Authors' conclusions: Self-evolution turns every known class of attack from a session-level into a lineage-level persistent threat, and produces new attack classes such as self-reward manipulation, evolution hijacking, and echo-trap exploitation; the existence of security mechanisms does not equal security, and defenses that do not cover the evolution path provide no protection against evolution-mediated attacks.
• Authors' implications: The authors call for developing evolution-aware security frameworks, longitudinal safety monitoring, and formal verification of self-modifying systems before the deployment speed of self-evolving systems outpaces the capacity for security reasoning.