AgentSafeEvo: An Adversarial Co-Evolutionary Framework for LLM Agent Safety

1. 01What problem does this paper attempt to solve?

   The authors propose AgentSafeEvo, which makes adversarial task construction and safe execution co-evolve across multiple attack types.

   How to make adversarial task construction and safe execution co-evolve across diverse tasks and attack types to improve the robustness of tool-using agents without degrading task performance.

   • Scenario: The authors state that LLM agents complete tasks such as search, document processing, and scheduling through multi-turn tool interactions, but tool use expands the attack surface to user queries, tool descriptions, and environment observations. Malicious content can rewrite task instructions, distort tool behavior, or mislead subsequent decisions, and induce unsafe actions during execution.
   • Existing shortcomings: The authors state that adversarial training on a fixed attack set does not update the attack distribution as defender capability changes. Language-level co-evolution mainly covers harmful text in isolated interactions; ARLAS's attacker only performs indirect prompt injection in observations of a given web task, leaving adversarial task construction outside co-evolution.
   • Observation: Task-level zero-sum rewards favor tasks that lower the defender's average reward, but do not directly account for within-task reward variance; the defender's within-group normalization largely removes cross-task reward scale differences. The authors hypothesize that giving higher relative weight to tasks with greater variance in safety and utility rewards can sustain continued co-evolution.
   • Training game: The attacker generates user queries, tool definitions, and environment observations from the scenario, seed intent, and environment information to induce unsafe behavior; the defender calls tools over multiple turns and gives a final answer. The update signal is the safety and utility of the defender's execution. The paper does not characterize this game as white-box, black-box, or no-box.
   • Proposed method: AgentSafeEvo jointly optimizes adversarial task generation and safe execution in the above game, with learnability-guided task weighting. Training data covers 50 agent scenarios, about 7,000 tools, and six attack types, including adversarial tasks for attacker initialization and seeds for co-evolution.

   Collapse this answer

2. 02What related research exists?

   The authors distinguish this paper from language-level co-evolution and from ARLAS, which only modifies given observations.

   The authors distinguish this paper from language-level attack-defense co-evolution and from ARLAS, which only injects into observations of a given task: the attacker must construct user queries, tool definitions, and environment observations across attack types.

   General co-evolution

   • SPAG (Cheng et al., 2024): Uses competitive language games to improve reasoning. The paper only cites it as an example of competitive settings.
   • MAPoRL (Park et al., 2025): Jointly trains multiple LLMs to enhance collaborative problem solving. The paper does not compare safety results with it.

   Safety-oriented co-evolution

   • Language-level attack-defense: purple-teaming (Zhou et al., 2024), Self-RedTeam (Liu et al., 2025), SEAS (Diao et al., 2025), non-cooperative game alignment (Paulus et al., 2025), and MAGIC (Wen et al., 2026) optimize attack and defense together. Self-RedTeam uses a single model for online self-play serving both sides; MAGIC alternately optimizes separate attacker and defender for adversarial prompt rewriting and safe responses, respectively. The authors state that such methods mainly cover harmful language generation in relatively isolated interactions, with limited coverage of multi-turn tool interactions and environmental changes.
   • ARLAS (Wang et al., 2025): Extends co-evolution to agent safety, training an attacker to inject malicious instructions into observations of a given task in a predefined web environment. The authors criticize that this attacker is limited to refining indirect prompt injection, and adversarial task construction is outside the co-evolution process.

   Agent safety evaluation and alignment

   • Evaluation: ToolEmu (Ruan et al., 2024) and R-Judge (Yuan et al., 2024) assess tool-use safety and risk awareness; AgentHarm (Andriushchenko et al., 2025) and InjecAgent (Zhan et al., 2024) focus on harmful task compliance and indirect prompt injection; AgentDojo (Debenedetti et al., 2024) evaluates prompt injection attack and defense in interactive tasks; Agent-SafetyBench (Zhang et al., 2024) evaluates responses and actions in diverse environments; MSB (Zhang et al., 2026) examines attacks on tool signatures, parameters, and responses in MCP environments.
   • Inference-time methods: Prompt Sandwiching (Schulhoff, 2024) and Spotlighting (Hines et al., 2024) separate trusted instructions from untrusted content; Prompt Guard 2 (Meta, 2025) and PIGuard (Li et al., 2025) filter potentially malicious content before it influences behavior.
   • Training alignment: Sha et al. (2025) handle threats from user requests and tool responses through sandboxed agent interactions.

   Baselines and benchmarks

   • Baselines: Original Qwen3-8B; Spotlighting with Delimiting; Prompt Guard 2; Static-AT (fixed attacker after SFT, only updating the defender, without task weighting).
   • Benchmarks: Safety side is Agent-SafetyBench, AgentHarm, AgentDojo, MSB; general capability side is BFCL-V3, ALFWorld, ToolHop. Attacker initialization and RL seeds come from TOUCAN tasks.

   The authors state that this paper incorporates adversarial task construction into co-evolution, rather than only language-level rewriting or only injecting into observations of a given task.

   Collapse this answer

3. 03How does the paper address this problem?

   First supervised fine-tune the attacker, then jointly update attack and defense with GRPO and learnability task weighting.

   AgentSafeEvo couples adversarial task generation and defensive learning into a game between an attacker π_A and a defender π_D. Figure 2 and Algorithm 1 have two phases: red-team attacker initialization, and joint adversarial training over multi-turn tool interactions.

   Problem Setting

   • Interaction: The agent calls external tools over multiple turns. Untrusted content can appear in user queries, tool descriptions, or environment observations.
   • Six attack types: direct prompt injection, indirect prompt injection, tool description injection, misinformation, harmful tool execution, and user query jailbreak, totaling 13 subtypes (Table 6).
   • Attacker: Input is the agent scenario, seed intent, and environment information; output is the user query, tool definitions, and environment observations. Tasks containing malicious content are called adversarial agent tasks.
   • Defender: Calls tools over multiple turns for a given task; the trajectory records tool calls, environment observations, and the final answer.

   Red-Team Attacker Initialization

   • Cold start: The authors state that the initial attacker cannot stably generate valid adversarial tasks. About 20,000 source tasks are sampled from TOUCAN; Gemini 3 Flash Preview converts about 10,000 into adversarial tasks according to specified attack intents and synthesizes construction trajectories, which are then used for supervised fine-tuning together with about 10,000 benign samples.
   • Objective: Equation (1) is the per-token negative log-likelihood of the attacker's output. The target output includes the user query and the task environment configuration specifying available tools and their returned observations.
   • Templates: One set of prompts each for red-team and benign cases; input is the scenario, intent, and tool definitions; output is the user query and tool functions. Appendix C.1 gives the full prompts. The defender system prompt is in Appendix C.2, roughly stating to complete separable legitimate parts and ignore or reject illegal, harmful, unauthorized, or conflicting instructions.
   • SFT settings: Qwen3-8B is fine-tuned with LLaMA-Factory. Table 7: maximum sequence length 16384, batch size 8, learning rate 3×10−5, 2 epochs, warmup ratio 0.03, DeepSpeed ZeRO-3.

   Reward Design

   • Safety: Qwen3.8-27B judges each trajectory as safe or unsafe, with rewards of +1 or −1.
   • Utility: The same model independently judges whether legitimate requirements are completed, with rewards of +1 or −1. Progress toward a harmful goal does not count as completing legitimate requirements; when the overall request is harmful, a safe refusal receives a utility reward of +1. The authors' rationale is that optimizing only safety could make the defender refuse the entire task.
   • Defender reward: Defined as R_D(τ_i,j)=R_safety(τ_i,j)+R_utility(τ_i,j), i.e., the sum of safety and utility; the maximum reward occurs only when both safe and legitimate requirements are completed.
   • Attacker reward: For each task, the mean defender reward over K trajectories is taken and negated, so the task-level reward is zero-sum. Higher attacker reward corresponds to lower average defender reward. The full review prompt is in Appendix C.3.

   Joint Optimization and Learnability Weighting

   • Seeds and grouping: RL data contains 6,000 red-team seeds and 6,000 benign seeds, to improve adversarial robustness while preserving general capability. Each seed generates M tasks, and the defender samples K trajectories per task. The defender advantage is normalized within the K trajectories of that task; the attacker advantage is normalized within the M tasks of the same seed. When the within-group reward standard deviation is 0, all advantages in that group are set to 0.
   • Weighting: The authors hypothesize that giving larger relative weight to tasks with greater variance in the defender's combined reward can sustain continued co-evolution. The weight is w_i=1+λ(σ_D⁽ⁱ⁾−σ̄_D)/(max_i′|σ_D^(i′)−σ̄_D|), where σ_D is the within-task standard deviation of defender reward and λ controls the deviation from unit weight. When the denominator is 0, that group's weight is 1. The same weight is multiplied onto both advantages after their respective GRPO normalization, without changing the outcome rewards or the grouping.
   • Loss: Both sides use the clipped GRPO surrogate objective. User inputs, tool outputs, and environment observations remain in the context and do not enter the loss. Table 8 sets the KL coefficient to 0.0.
   • RL settings: Attacker learning rate 1×10−6, defender 3×10−6; rollout counts are 5 and 8, respectively; 4 GPUs per role; 120 steps; batch size 64; λ=0.5; rollout temperature 1.0; maximum prompt length 6144, maximum response length 16384. The pipeline is based on PettingLLMs and verl, with rollout using vLLM v0.19.1. The paper does not state whether M and K equal the above rollout counts.

   Training Data and Attack Types

   • Composition: 50 scenarios (Table 4), about 7,000 tools, with operation categories in Table 5. Figure 4 counts 10,310 red-team tasks: harmful tool execution 28.5% (2,935), direct prompt injection 27.0%, user query jailbreak 13.6%, indirect prompt injection 10.9%, tool description injection 10.4%, misinformation 9.7%.
   • Type distinctions: Indirect prompt injection has instructions in external content encountered only after interaction; misinformation provides false, outdated, or unreliable claims rather than executable instructions; harmful tool execution requires specific actions in the trajectory that advance a harmful goal, intent alone is insufficient; user query jailbreak relies on social engineering cues and need not explicitly demand ignoring higher-priority rules. Subtype descriptions are in Table 6.
   • Non-released parts: Appendix B states that, to reduce misuse risk, the red-team data synthesis pipeline, including its prompts and implementation details, is not released.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The authors state that on Qwen3-8B, the mean of each safety metric is better than the listed baselines, and ALFWorld and ToolHop are also higher.

   Experimental Setup

   • Models: Both attacker and defender backbones are Qwen3-8B. The reward model is Qwen3.8-27B, judging safety and utility separately during training. Figure 3(b) additionally evaluates the attacker on Llama3.3-70B, Qwen3-235B, DeepSeek-V4-Pro, and GPT-5.
   • Training data: SFT samples about 20,000 source tasks from TOUCAN; TOUCAN contains over 1.5 million trajectories from 495 real MCP servers. About 10,000 of these are converted to adversarial tasks by Gemini 3 Flash Preview and merged with about 10,000 benign samples. RL uses 6,000 red-team seeds plus 6,000 benign seeds.
   • Baselines: The original Qwen3-8B; Spotlighting with Delimiting, which marks all tool outputs as untrusted content with delimiters; Prompt Guard 2 (written as Llama Prompt Guard 2 in the appendix), which performs injection detection only on tool output text, with flagged content replaced before the next step, and the detector not seeing the user task or conversation history; Static-AT, which uses a fixed SFT attacker and generates tasks online, training only the defender without task weighting.
   • Safety benchmarks: Appendix E.1 states that Agent-SafetyBench has 2,000 test samples and 349 interactive environments; AgentHarm has 110 explicitly malicious base behaviors and 440 task variants, with harm score using human-written scoring rubrics; AgentDojo has 97 user tasks and 629 safety test samples; MSB has over 2,000 attack instances. Metrics are ASR, harm score, UA, and NRP. The main text states that ASR measures whether an attack induces unsafe behavior, UA measures whether the intended user task is completed under attack, and NRP combines task performance under attack with attack success; Appendix E.1 states that NRP combines task performance and attack resistance into a joint measure of robustness. The paper does not give the NRP formula, nor does it specify the external review model for each benchmark's ASR.
   • General capability: BFCL-V3, ALFWorld, and ToolHop report accuracy. Appendix E.2 states that ALFWorld evaluates 109 tasks and follows AgentBench-FC in AgentRL; ToolHop has 995 queries and 3,912 tools. The paper does not give the number of samples for BFCL-V3.
   • Protocol: Evaluation sampling uses temperature 0.6 and top_p 0.95. Section 4.2 reports means and standard deviations of three independent runs for safety and general capability. Cross-evaluation uses 100 out-of-training seeds. Figure 3 does not mark the number of repetitions.

   Main Results

   According to Table 1, the following numbers are means±standard deviations of three runs on Qwen3-8B. Lower ASR and harm score are better; higher UA and NRP are better.

   Swipe horizontally to read the table

   Metric	Qwen3-8B	+Spotlighting	+Prompt Guard 2	+Static-AT	+AgentSafeEvo
   ASB ASR↓	49.11±0.51	48.52±0.78	44.13±0.64	31.01±0.33	17.84±0.24
   Harm score↓	50.37±3.63	53.00±2.60	54.67±1.07	30.00±2.05	6.53±0.38
   Dojo ASR↓	22.20±0.57	19.25±1.04	16.44±0.45	6.71±0.43	6.18±0.18
   Dojo UA↑	53.64±0.90	57.82±1.11	47.35±0.52	61.43±0.15	66.52±1.43
   MSB ASR↓	60.43±0.97	58.53±0.73	59.02±1.57	38.64±2.24	31.25±1.24
   MSB NRP↑	28.12±1.30	29.55±0.46	29.71±1.29	40.64±0.93	46.00±0.54

   • Authors' interpretation: The authors state that AgentSafeEvo has the best mean on all reported metrics across the four safety benchmarks. Higher UA is interpreted by the authors as better task completion under attack, and higher NRP as a better combination of task performance and attack resistance.
   • General capability: In Table 2, AgentSafeEvo's ALFWorld accuracy is 43.72±3.03% and ToolHop 48.04±1.40%, higher than Static-AT's 30.89±0.43% and 39.43±0.09%. BFCL-V3 is 65.98±0.14%, lower than the original model's 66.13±0.22% and Static-AT's 66.73±0.50%. The authors argue that co-evolved tasks provide training signals for planning and tool use, while function-calling accuracy is largely maintained.
   • Exceptions in the table: Spotlighting and Prompt Guard 2 have higher AgentHarm harm scores than the original model, and Prompt Guard 2 has lower AgentDojo UA than the original model. The paper does not separately explain these regressions of the baselines relative to the original model.

   Co-evolution and Cross-Model Transfer

   • Cross-evaluation: Using 100 out-of-training seeds, attackers at different stages generate tasks, and defenders at each stage are evaluated on the same task sets. The main text states that for the base defender, the SFT attacker's ASR is 55.81% and the final attacker's is 92.59%; for the final attacker, ASR goes from 92.59% with the base defender to 49.82% with the final defender. Figure 3(a) labels these as 55.8, 92.6, and 49.8. In the figure, rows decrease from left to right and columns increase from top to bottom. The authors state that later attackers have higher ASR against every defender, and later defenders have lower ASR against every attacker.
   • Transfer: Figure 3(b) compares the SFT attacker with the final attacker co-evolved against Qwen3-8B. The figure labels are: Qwen3-8B 55.8 and 92.6, Llama3.3-70B 43.7 and 80.7, Qwen3-235B 50.9 and 81.1, DeepSeek-V4-Pro 26.2 and 39.3, GPT-5 12.3 and 22.5. The main text confirms Llama3.3-70B from 43.7% to 80.7% and GPT-5 from 12.3% to 22.5%. The paper does not state whether Qwen3-8B in the figure is the base or trained defender; its labels match the 55.8 and 92.6 for the base defender in Figure 3(a).
   • Authors' interpretation: The authors state that the evolved attacker has higher ASR on all four additional target models and argue that attack improvements are not limited to the defender used in co-evolution. The paper does not explain why absolute values on DeepSeek-V4-Pro and GPT-5 are lower than on Llama3.3-70B.

   Ablations

   Table 3 still targets Qwen3-8B. The note states that "no attacker update" is Static-AT. The paper does not restate the number of repetitions for Table 3, but the table gives standard deviations.

   Swipe horizontally to read the table

   Variant	ASB ASR↓	MSB ASR↓	MSB NRP↑	ALFWorld↑	ToolHop↑
   AgentSafeEvo	17.84±0.24	31.25±1.24	46.00±0.54	43.72±3.03	48.04±1.40
   No attacker update	31.01±0.33	38.64±2.24	40.64±0.93	30.89±0.43	39.43±0.09
   No task weighting	22.91±0.35	32.89±0.66	41.48±0.21	36.70±0.75	40.07±0.58
   No defender weighting	14.13±0.33	28.47±1.03	43.04±0.73	29.05±0.43	40.80±0.22
   No attacker weighting	15.52±0.27	34.92±0.89	41.87±0.33	34.25±1.89	40.63±0.29

   • Co-evolution: The authors report that, relative to training only the defender, unweighted co-evolution lowers ASR on Agent-SafetyBench and MSB by 8.10 and 5.75 percentage points respectively, and ALFWorld accuracy goes from 30.89% to 36.70%. The authors argue that online task generation alone cannot explain these differences, because the fixed attacker still generates tasks in the interaction loop.
   • Shared weighting: The authors state that, relative to unweighted co-evolution, shared task weighting improves all reported metrics in the table. When only the attacker or only the defender is weighted, ALFWorld accuracy goes from the unweighted 36.70% to 29.05% and 34.25%, respectively.
   • Exceptions: The authors report that both one-sided weightings have lower ASR on Agent-SafetyBench than the full method, and when only the attacker is weighted, MSB ASR is also lower at 28.47%, versus 31.25% for the full method. The authors argue that shared weights help robust execution and general capability rather than lowering ASR alone; NRP also accounts for task performance under attack and attack resistance.

   Other Ablations and Analyses

   • Middle cell of Figure 3(a): the SFT attacker's ASR against the RL step 120 defender is 26.9; the RL step 120 attacker's ASR against the RL step 90 defender is 74.2.
   • The cross-evaluation text states 55.81%, 92.59%, and 49.82%, while Figure 3(a) labels them as 55.8, 92.6, and 49.8.
   • The paper does not report separate sweeps of λ, learning rate, rollout count, or temperature.
   • Figure 3(b) does not mark error bars or the number of repetitions.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors plan to extend to multi-agent delegation and state that benchmark results do not establish safety in all deployment scenarios.

   Limitations and Future Directions Noted by the Authors

   The paper has no separate Limitations section. Below are only the future directions and boundaries explicitly stated in the main text.

   • Multi-agent: The Conclusion states that future work will extend AgentSafeEvo to collaborative multi-agent settings, focusing on security risks from communication and task delegation.
   • Deployment boundaries: The Ethics Statement states that improvements on evaluated benchmarks do not establish safety in all deployment settings; actual deployment requires further evaluation of the tools, permissions, and environments available to the agent.

   Experimental Coverage

   • Backbones and transfer targets: The attacker and defender backbones are Qwen3-8B; target models in Figure 3(b) are Qwen3-8B, Llama3.3-70B, Qwen3-235B, DeepSeek-V4-Pro, GPT-5. The reward model is Qwen3.8-27B.
   • Benchmarks and repetitions: Safety evaluations are Agent-SafetyBench, AgentHarm, AgentDojo, MSB; general capability is BFCL-V3, ALFWorld, ToolHop. Section 4.2 reports the mean and standard deviation of three independent runs for the first two main evaluation categories.
   • Comparisons: Baselines are original Qwen3-8B, Spotlighting with Delimiting, Prompt Guard 2, and Static-AT (Section 4.1). Ablations include removing attacker updates, removing task weighting, and removing only one side's weighting (Table 3).
   • Attack data and process: Red-team tasks are organized into six types and 13 subtypes, with Figure 4 counting 10,310; RL seeds are 6,000 red-team and 6,000 benign. Cross-evaluation uses 100 out-of-training seeds, with stages including SFT and RL steps 30, 60, 90, 120.
   • Unreported evaluation information: The paper does not report agreement between reward model judgments and human judgments, nor does it specify the external review model used for ASR on each safety benchmark. The Reproducibility Statement and Appendix B state that the red-team data synthesis pipeline is not released.

   Collapse this answer

6. 06Summarize the main content of the paper

   Attack-defense cooperative reinforcement learning trains tool-use agents; the authors state it can improve robustness while maintaining general capabilities.

   AgentSafeEvo is an attack-defense co-evolution reinforcement learning framework for tool-using LLM agents, used to simultaneously update adversarial task construction and safe execution across diverse tasks and attack types.

   • Problem: Malicious content can appear in user queries, tool descriptions, or environment observations, and can induce unsafe actions across multi-turn tool calls. The authors argue that fixed attack sets, language-level attack and defense, and approaches that only modify observations of given tasks do not place adversarial task construction into co-evolution.
   • Method: The attacker is first supervised fine-tuned on adversarial samples synthesized from TOUCAN tasks, then jointly updated with the defender using GRPO. The defender reward is the sum of trajectory-level safety and utility; the attacker reward is the negative of the mean defender reward on that task. Learnability weights adjust both updates according to the within-task standard deviation of defender rewards. Training covers six attack types, and RL uses both red-team seeds and benign seeds.
   • Safety results: On Qwen3-8B, AgentSafeEvo achieves an Agent-SafetyBench ASR of 17.84%, an AgentHarm harm score of 6.53, an AgentDojo ASR of 6.18% and UA of 66.52%, and an MSB ASR of 31.25% and NRP of 46.00 (Table 1). The authors state that these means are better than the listed baselines.
   • Capability and transfer: ALFWorld accuracy is 43.72%, ToolHop 48.04%; BFCL-V3 is 65.98%, lower than the original model's 66.13% and Static-AT's 66.73% (Table 2). Relative to the SFT attacker, the final attacker's ASR on Llama3.3-70B goes from 43.7% to 80.7%, and on GPT-5 from 12.3% to 22.5% (Figure 3b, main text). The authors argue that both attack and defense improve during co-evolution, and that attack improvements are not limited to the defender used in training.
   • Authors' conclusion: The authors argue that updating the task generation policy with defender feedback and weighting both sides with the same task-level signal can strengthen the robustness of tool agents while maintaining general capability. Future plans include extending to multi-agent communication and task delegation; the ethics statement also states that improvements on benchmarks cannot establish safety in all deployment settings.

   Collapse this answer

Related papers