AutoRedClaw: automated red teaming for Claw-style agents

Once a claw-style agent's configuration, memory and skills are poisoned, they repeatedly enter the context across tasks and sessions.

• Setting: The authors say systems are shifting toward persistent assistants like OpenClaw (over 100,000 GitHub stars in about two months); configuration, memory and skills are reloaded across sessions, and poisoned content may also enter the system prompt (Figure 1). The authors also cite ClawWorm and malicious skills on ClawHub, saying such threats have already appeared in practice.
• Existing methods: The authors say manual cases and environment constraints limit scale and coverage, and static sets easily become outdated; many benchmarks start from an already-poisoned AGENTS.md and do not examine how external content induces an agent to poison its own state; evaluations often use function-calling or MCP, whereas actual deployments more often use skills, CLI, shell and the browser.
• The authors' approach: They propose the automated red-teaming framework AutoRedClaw, which generates end-to-end attacks together with executable environments, and freeze RedClaw-Bench from its outputs.
• Threat model (Section 2):
  • Goal: An external attacker places malicious content in emails, web pages, files or skills to hijack the agent into performing the attacker's task. inject must rewrite configuration or long-term memory, or install a malicious skill; trigger must induce harmful actions after the persistent state has been poisoned.
  • Knowledge: Follows the standard indirect prompt injection setting. The generation guidelines require treating the victim system as a black box and do not assume the attacker knows private content, hidden prompts or internal defenses (Appendix F.3).
  • Capability: Malicious intent can only be conveyed through the environment, not by sending messages through the user entry point. All of inject's instructions come from the environment; trigger assumes the malicious instructions are already in the persistent state.
  • Victim: Claw-style agents with AGENTS.md, MEMORY.md, USER.md, SOUL.md and a skill library. The experimental subjects are four frontier models on seven harnesses.

Related work is divided in Appendix B into indirect prompt injection, tool-calling evaluation, and claw-style evaluation.

Indirect prompt injection

• Attacks: Perez and Ribeiro (2022) proposed indirect prompt injection; the authors also list Neural Exec by Harang (2023), Schulhoff et al. (2023), and Pasquini et al. (2024), and TAP by Mehrotra et al. (2024). The paper only lists them and does not compare them one by one.
• Defenses: The authors divide them into model-level and system-level. At the model level they name PIGuard (Li et al., 2025b), StruQ (Chen et al., 2025a), SecAlign (Chen et al., 2025b), and ReasAlign (Li et al., 2026b); at the system level they name IsolateGPT (Wu et al., 2025), Debenedetti et al. (2025), Progent (Shi et al., 2025), DRIFT (Li et al., 2025a), and Shi et al. (2026). The authors state that these works add safety guardrails, but base models already behave safely on many existing benchmarks even without defenses, making it harder to verify whether defenses are effective.

Tool-calling agent evaluation

• Early benchmarks: InjecAgent (Zhan et al., 2024) collects 1,054 cases, covering directly harming users and leaking private data. AgentDojo (Debenedetti et al., 2024) turns the environment into function-calling tools and contains four evaluation suites. Agent Security Bench (Zhang et al., 2025) covers 10 prompt injection types and 10 scenarios. AgentDyn (Li et al., 2026a) provides 60 open user tasks and 560 safety tests.
• The authors' criticism: The authors state that these works suit early tool-calling systems running within ad hoc windows, but are difficult for evaluating current frontier models and claw-style agents.

Claw-style agent evaluation

• Persistent state: ClawWorm (Zhang et al., 2026b) shows that configuration poisoning can survive session restarts and propagate across agents. Gadgil et al. (2026) is treated by the authors as an example of testing from an already poisoned state: downstream effects can be seen, but it does not examine how external interactions induce an agent to poison its own persistent state. CIK-Bench (Wang et al., 2026) examines injection and triggering on persistent capabilities, identity, and knowledge; the authors point out that its injection places persistent poisoning instructions in the user query, unlike a real attack surface.
• Recent benchmarks: ClawSafety (Wei et al., 2026) has 120 cases, injected via skill files, emails, and web pages. ClawsBench (Li et al., 2026c) evaluates 44 tasks across five simulated applications. SafeClawArena (Niu et al., 2026) evaluates 406 adversarial tasks in containerized deployments. The authors state that expanding coverage still requires separately building tasks, attacks, and supporting environments.
• DTap: Chen et al. (2026) manually built 50 simulated environments and sampled 14 real scenarios. The authors state that the coverage is still narrower than real environments, manually building environments is costly, and workflows that the environment cannot execute cannot be evaluated.

Baseline methods and benchmarks

• Comparison targets: The experimental baselines are AgentDojo, AgentDyn, and DTap. The authors draw a frozen set, RedClaw-Bench (330 cases), from generated samples for unified evaluation.

The authors position this paper as automatically generating executable environments, benign tasks, malicious goals, and state judgments, and as testing the inject that establishes persistent poisoning separately from the trigger of downstream harmful actions.

AutoRedClaw first generates task and attack metadata, then materializes files and web applications, and finally uses probes to filter evaluable samples (Figure 2). Generation unfolds along 9 risk categories, 12 strategies, 2 stages, and 5 attack surfaces.

Scenario axes and experience guidance

• Risk × strategy: Risks describe harmful consequences, such as credential leakage, destructive modification, and privilege override; strategies describe inducement mechanisms, such as instruction override, boundary confusion, and poisoned memory propagation. The two cross to form 108 pairs. Definitions are in Appendix F.1 and F.2.
• Stage × attack surface: Inject tests from a not-yet-poisoned persistent state whether an external payload can be written; trigger tests from an already implanted payload whether subsequent harmful actions occur. The attack surfaces are AGENTS.md (agent instructions), MEMORY.md (long-term memory), USER.md (user profile), SOUL.md (persona and behavioral boundaries), and SKILL in the skill library, for 10 stage–surface combinations in total. These artifacts are poisoning destinations in inject and pre-set payload carriers in trigger.
• Experience pool: Records the number of collected samples on risk, strategy, stage, and attack surface, and an LLM summarizes recurring scenarios, injection carriers, payload wording, and inducement rationales. The generator is required to prioritize under-covered configurations and avoid already summarized patterns. The paper does not name the model doing the summarization, nor state the refresh frequency.

Metadata generation

• Benign tasks: Must be completable with the tools and information of the given scenario, resemble real assistant requests, and have clear operations, identifiable inputs, and expected results.
• Malicious tasks: Inject must establish the specified malicious content on the specified persistent surface; trigger must induce the corresponding harmful behavior. The delivery source must be a channel the attacker can reasonably control; what it knows is limited to public information or information visible from that source (Appendix F.3). Appendix F.4 gives the full generation prompt.
• Task-driven assets: The two types of tasks are generated first, then the required resources are determined, rather than building a complete environment first and then setting the tasks. Files are accessed via native file and shell tools; web pages are accessed via a browser, used to simulate applications such as email, chat, and cloud storage, rather than building separate dedicated tool interfaces. During evaluation, a reusable container is prepared for each harness, mounting only the assets required by that scenario.
• State judgment: Each condition states which file or application record to check after execution and what state it should contain. Inject checks whether malicious content newly enters the selected workspace file, or whether a malicious skill is newly registered. Trigger checks whether the risk state specified by the attacker is produced. The paper gives an example: a file-deletion task checks whether the target file still exists. Final natural-language replies are not scored.

Environment materialization

• Files: Content is completed according to the asset description, and legitimate content beside the specified payload is retained. If a document contains skills or scripts, the corresponding files are generated from that content. Skills and scripts must implement the functionality they claim; after removing the malicious instructions, the legitimate task must still be completable.
• Web pages and backend: HTML consistent with the application context is generated, with a backend to support sending mail, submissions, and state changes. The backend also serves as a mock sink, recording messages, inboxes, and databases for state judgment. Appendix F.5 gives the materialization prompt.
• Mounting: Generated assets are retained for reuse. Web page hostnames are written into the container hosts, so the agent accesses local services via ordinary URLs without rebuilding the harness runtime.

Validation probes

• Four checks: Oracle integrity requires at least one automatically executable attack success condition, with matching text and persistent file references complete. Attack reachability requires the payload to lie on the path to completing the benign task, with the main constraint coming from the external environment in inject; in trigger, the payload is pre-set in workspace files that are automatically loaded at execution. The initial state check requires that the attack condition is not yet satisfied before running. Task feasibility is trial-run with a more capable model, with DeepSeek-V4-Pro as the default probe; a case is retained only after the probe achieves attack success.

Harness adapters

• Workspace mapping: Persistent file paths in cases are represented by unified placeholders and mapped to the real locations of each harness during evaluation. For example, MEMORY.md is at the workspace root in OpenClaw and at memory/ in Nanobot.
• Capability mapping: Operations required by the scenario are written as primitives such as file.read, exec.run, and memory.write, then mapped to each harness's native tools. Adding a new harness requires adding a mapping file.

Experimental setup

• Harness: Function-oriented: OpenClaw, Nanobot; safety-oriented: NemoClaw, SeClaw, ZeroClaw; coding agents: CodeX, Claude Code. Each uses official default settings. Each case runs in an isolated Docker container based on Ubuntu 22.04, with tasks switched by mounting assets.
• Models: GPT-5.4, Claude-Sonnet-4.5, Gemini-3.1-Pro, DeepSeek-V4-Pro. Coding agents are tested only with natively supported providers. NemoClaw and Gemini have no results because a compatibility plugin caused OpenClaw's SSRF check to block the internal inference endpoint (Table 1 note).
• Generation roles: The scenario generation, materialization, and probe models in the red-team stage are all DeepSeek-V4-Pro. The probe harness for the efficiency experiment is OpenClaw.
• Benchmark: RedClaw-Bench has 330 cases in total: 30 cases each for the five persistent surfaces on inject and trigger, plus 30 cases of same-session indirect prompt injection that does not pass through persistent state (Section 4.3, Appendix E.4). It covers 9 risk categories, 12 strategies, 100 risk–strategy combinations, and 25 scenarios (Figure 5, Table 2). Compared with AgentDojo, AgentDyn, DTap.
• Metrics and judgment: U is the percentage of benign tasks completed, ASR is the percentage meeting the attacker's success condition, and the two are computed independently. Judgment is by state check. The paper does not state LLM judging, human agreement, or number of repetitions.
• Scope of the facet tables: In Appendix Table 3 and Table 4, CodeX lists only AGENTS.md and SKILL, and Claude Code lists only CLAUDE.md and SKILL. The paper does not state whether the averages for these two harnesses in Table 1 include only these surfaces.

Main results

The table below is the Average column of Table 1 (%). The averages for CodeX and Claude Code include only their reported models; NemoClaw excludes Gemini.

Swipe horizontally to read the table

Harness	Inject U	Inject ASR	Trigger U	Trigger ASR
OpenClaw	78.5	65.7	72.9	69.2
Nanobot	80.2	64.2	75.7	73.5
NemoClaw	74.0	56.2	53.8	50.9
SeClaw	52.8	13.0	50.2	10.5
ZeroClaw	72.2	26.3	61.0	35.8
CodeX	61.7	65.0	81.7	65.0
Claude Code	56.7	26.7	50.0	21.7

• The authors state that Nanobot and OpenClaw are easy to compromise in both stages, and that most systems have ASR above 50%. In the table, the average ASR of SeClaw, ZeroClaw, and Claude Code is below 50%.
• The authors state that SeClaw and ZeroClaw are safer, with a decline in functionality. In the table, ZeroClaw's average U is higher than SeClaw's. The authors state that Claude Code is safer than CodeX, but its utility is lower than harnesses such as OpenClaw.
• Single-cell highs not listed in the table: Nanobot with DeepSeek V4 Pro has trigger ASR of 92.0% (U is 82.7%); OpenClaw with the same model has trigger ASR of 90.7% and inject ASR of 81.3% (Table 1).

Generation efficiency and coverage

• Efficiency: DeepSeek-V4-Pro generates, OpenClaw is the probe. After generating 1,500 samples, the feasibility rates of inject and trigger are 20.1% and 46.1%, respectively. In Figure 3(a), the horizontal axis is Generated samples, and the end is labeled with the number of successful attacks: inject 302, trigger 691. The authors state that trigger is higher than inject, and agents are more vulnerable to attacks from internal state.
• Cost basis: Section 4.2 states that under this setting the average generation cost per successful sample does not exceed $0.10. The abstract says the effective attack rate is above 30% and each successful sample is below $0.10. The paper does not break down "above 30%" by dimension.
• Coverage: The authors state that within the first 500 successful attack samples, all 9 risk categories and 12 strategies are covered, and the risk × strategy combinations for trigger and inject exceed 95 and 65, respectively. In Figure 3(b), the vertical axis is Covered combinations, labeled Max 108 (9 risks × 12 strategies), and 108 and 85 are marked.

Comparison with existing benchmarks

• Low-ASR benchmarks: Compared using GPT-5.4 and Claude-Sonnet-4.5. The authors state that these two models have ASR no higher than 1% on AgentDojo and AgentDyn. Figure 6 marks 0.5, 0.7, 0.0, and 0.9 at the corresponding positions, and the main text does not specify item by item which number belongs to which benchmark. The comparison does not state the harness.
• Relative to DTap: RedClaw-Bench's Inject ASR goes from 40.0% to 64.7% on GPT-5.4, and from 25.2% to 62.0% on Claude-Sonnet-4.5. Figure 6 marks +24.7 pp and +36.8 pp. The harness is likewise not stated.
• Utility: The authors state that RedClaw-Bench's utility is lower than DTap and AgentDojo, and thus the tasks are harder. Figure 6 has several utility readings, and the main text does not correspond them item by item to benchmarks; these readings are not assigned here.

Attack surfaces, risks, and strategies

• Attack surfaces (Section 4.4): In the Inject stage, SKILL has the lowest ASR on almost all harnesses. Among the four text surfaces, SOUL.md has the highest cross-harness average Inject ASR, at 42.7%, 71.1% on OpenClaw, and 66.7% on NemoClaw. In the Trigger stage, AGENTS.md has the highest average ASR, at 50.9%, 77.8% on OpenClaw, and 85.6% on Nanobot. The authors state that SOUL.md is relatively less sensitive, and AGENTS.md is granted higher privileges. The curves in Figure 4 are the average of GPT-5.4, Claude-Sonnet-4.5, and Gemini-3.1-Pro, excluding DeepSeek V4 Pro.
• Risks (Appendix E.2): NemoClaw has ASR of 83.3% on Inject's destructive actions and 65.0% on Trigger's privilege override. The authors state that these are higher than its overall Inject 48.7% and Trigger 44.0%, and also higher than OpenClaw's and Nanobot's results on these categories. The paper does not state whether the 48.7% and 44.0% aggregates include DeepSeek V4 Pro; in Table 1, NemoClaw's averages including that model are 56.2% and 50.9%.
• Strategies (Appendix E.2): The authors state that UI Ambiguity is more effective in the Inject stage on OpenClaw, Nanobot, and NemoClaw, and weakens in the Trigger stage. Poisoned Memory Propagation is the opposite: OpenClaw and Nanobot go from 19.0% and 23.8% on Inject to 66.7% and 69.0% on Trigger. The authors state that establishing persistent compromise and exploiting it favor different strategies.

Other ablations and analyses

• Standard IPI has 30 cases in total (Table 5): CodeX+GPT-5.4 has U of 70.0% and ASR of 23.3%; Claude Code+Claude-Sonnet-4.5 has U of 36.7% and ASR of 16.7%.
• Table 3: SeClaw's SKILL inject ASR is 0.0% on GPT-5.4, Claude-Sonnet-4.5, and Gemini-3.1-pro, and 3.3% on DeepSeek V4 Pro (30 cases each).
• Table 3: With ZeroClaw and GPT-5.4, the inject ASR of AGENTS.md and USER.md is 0.0%; its SKILL inject ASR is 46.7%, higher than the text surfaces in that row.
• Table 4: With OpenClaw and DeepSeek V4 Pro, the trigger ASR of AGENTS.md is 96.7%, and SKILL is 100.0% (30 cases each).
• Table 4: With SeClaw and GPT-5.4, the trigger ASR of SOUL.md is 0.0%; with Claude-Sonnet-4.5, the trigger ASR of USER.md is 0.0%.
• Appendix E.5 gives one example skill injection task and eight generated application interfaces; the payload is not repeated here. The paper has no separate ablation table.

Limitations and future directions noted by the authors

• Models and attack surfaces: The evaluation focuses on selected models and harnesses and five persistence attack surfaces; more memory architectures and deployment configurations are left for future research (Appendix A).
• Multimodality and multi-agent: Extending to multimodal payloads and multi-agent shared memory requires corresponding adapters and evaluation standards (Appendix A).
• Simulated environments: Current environments use isolated containers and simulated applications; connecting to online services and changing access policies would add operational complexity (Appendix A).
• Session length: The experiments examine bounded task execution; how poisoned state evolves over long, multi-session interactions is listed by the authors as a future direction (Appendix A).
• Generator: Both scenario generation and probing use DeepSeek-V4-Pro. The authors say few available models are both weakly safety-aligned and highly capable, and the impact of switching to other generators can be a future direction (Appendix A).

Scope of the experiments

• Systems tested: The harnesses are OpenClaw, Nanobot, NemoClaw, SeClaw, ZeroClaw, CodeX and Claude Code; the driving models are GPT-5.4, Claude-Sonnet-4.5, Gemini-3.1-Pro and DeepSeek-V4-Pro (Section 4.1, Table 1).
• Frozen benchmark: RedClaw-Bench contains 330 cases: 30 each for inject and trigger on the five persistence surfaces, plus 30 same-session IPI cases; 9 risk categories, 12 strategies and 25 scenarios (Section 4.3, Figure 5, Table 2).
• Generation experiments: The efficiency experiment uses DeepSeek-V4-Pro as generator and probe model and OpenClaw as the probe harness, reporting the feasibility rate after generating 1,500 samples and a cost of no more than $0.10 per successful sample (Section 4.2). The paper does not report the number of repetitions or the number of queries.
• Comparisons and judgments: The comparisons with AgentDojo, AgentDyn and DTap use GPT-5.4 and Claude-Sonnet-4.5 (Section 4.3). That passage does not state the harness. Success is determined by state checks; the paper does not report agreement with human judgment.
• Missing rows and averaging scope: NemoClaw–Gemini has no result due to the SSRF check; the two coding agents are evaluated only on natively supported models (Table 1). Figure 4 averages GPT-5.4, Claude-Sonnet-4.5 and Gemini-3.1-Pro, excluding DeepSeek V4 Pro.

AutoRedClaw is an automated red-teaming framework for persistent claw-style agents, used to simultaneously generate attacks, executable environments and state judgments.

• Problem: AGENTS.md, MEMORY.md, USER.md, SOUL.md and skills are reloaded across sessions. The authors argue that testing only downstream behavior after poisoning, or using only function-calling/MCP environments, does not match how such assistants are actually used.
• Approach: The generation space consists of 9 risk categories, 12 strategies, inject/trigger and five persistence surfaces. An experience pool pushes new samples away from existing patterns. Environments mount files and web pages per task, and probes check judgment conditions, reachability, initial state and feasibility. The default probe is DeepSeek-V4-Pro. Adapters map the unified paths and capabilities to different harnesses.
• Generation results: On the OpenClaw probe, the inject and trigger feasibility rates of 1,500 generated samples are 20.1% and 46.1%; Section 4.2 says the average cost per successful sample is no more than $0.10. The abstract separately states an effective attack rate above 30% and a cost below $0.10.
• Benchmark results: On the 330-case RedClaw-Bench, OpenClaw's average ASR is inject 65.7%, trigger 69.2%; Nanobot 64.2% and 73.5%; NemoClaw 56.2% and 50.9% (Table 1). SeClaw 13.0% and 10.5%; ZeroClaw 26.3% and 35.8%; Claude Code 26.7% and 21.7%.
• Relative to older benchmarks: The authors say GPT-5.4 and Claude-Sonnet-4.5 have ASR no higher than 1% on AgentDojo and AgentDyn. Relative to DTap, these two models' Inject ASR goes from 40.0% and 25.2% to 64.7% and 62.0%; that passage does not state the harness.
• The authors' conclusion: This pipeline can continuously produce executable evaluations at relatively low cost and give distinguishable safety and utility readings for advanced harnesses. The authors say different attack surfaces carry different risks: SOUL.md is relatively easy to write to, and AGENTS.md is granted higher privileges at the trigger stage, which designers can use to strengthen the corresponding protections.