X-AgentRed: Red Teaming Personal Agents in Evolving Multi-Service Ecosystems

What is to be discovered: how, while a personal agent fulfills benign requests, cross-service actions accumulate into harm that the user did not request or authorize.

• Scenario: the authors state that personal agents such as ChatGPT, Claude, and Muse orchestrate web and app services into multi-step workflows; service integrations and browser and desktop interactions continue to expand composable capabilities (Introduction). After information, permissions, and state propagate across services, benign requests can also deviate from intent or bypass safety guardrails. Figure 1(a) is a schematic; the coverage and length figures are also illustrative, with measurements in Section 4.4.
• Existing approaches: the authors state that existing methods often have an LLM propose tool sequences from task and environment descriptions (Li et al., 2025), or decompose harmful goals specified by humans or LLMs into seemingly benign actions (Kulkarni et al., 2025; Zeng et al., 2026), then revise with execution feedback. The authors believe that having an LLM decide how to connect tools relies on training priors, which may favor familiar combinations and under-explore new service interactions.
• Authors' requirement: an expanding service space needs candidate paths identified from tool capabilities and dependencies, rather than only reusing previously tested workflows (Introduction).
• This paper: proposes X-AgentRed. The red-team agent analyzes effects, potential harm, and execution preconditions, builds a HarmFlow graph, samples recipes and writes them into scenarios, then uses execution feedback to expand successful paths and correct blocked paths.
• Threat model:
  • Goal: discover harm produced through cross-service interactions while fulfilling benign requests. The request must be safely completable; the target harm must not be requested or authorized by the user.
  • Knowledge: the red team analyzes descriptions in the tool registry and reads blocked steps and unmet preconditions in trajectories. The target agent sees only the request and initial state, not the target harm.
  • Capabilities: generate scenarios and execute them against the target; use feedback to update the graph and recipes; rewrite subsequent user prompts according to target responses and tool outputs until harm occurs or a turn limit is reached. The paper does not state the number of turns in this limit.
  • Victim system: a personal agent in a multi-service ecosystem with a tool registry. The formalization is in Section 3; service and tool scales are given in the experiments.

Automated red teaming and prompt optimization

• Perez et al. (2022) generate adversarial inputs with language models.
• Zou et al. (2023), Chao et al. (2025), Mehrotra et al. (2024), Liu et al. (2024) optimize adversarial or jailbreak prompts. Related Work only lists them, without comparing each one to this paper.
• Russinovich et al. (2025), Rahman et al. (2025) adapt adaptively in multi-turn interactions.

Tool-using agents

• Ruan et al. (2024), Andriushchenko et al. (2025), Debenedetti et al. (2024) evaluate harmful behavior of agents and vulnerability to external content.
• Kulkarni et al. (2025), Li et al. (2026) construct adversarial tasks and interactions.
• The introduction lists Li et al. (2025), Kulkarni et al. (2025), Zeng et al. (2026) as representatives: even when execution feedback is used to evaluate and revise candidates, the LLM still decides how tools are combined. The authors believe this reliance on priors may introduce bias.

Diversity and feedback

• Samvelyan et al. (2024), Wang et al. (2025) vary both the targeted harmful behavior and the prompt style.
• Hong et al. (2024) use novelty rewards to encourage outputs different from already generated prompts.
• Liu et al. (2025) extract strategies from previous attempts, store them in a library, and reuse them in subsequent search.

Baseline methods and environment: STAC (Li et al., 2025) composes individually benign tool calls into harmful chains and rewrites prompts with execution feedback. TRACE (Zeng et al., 2026) decomposes harmful tasks into subtasks and iteratively modifies prompts based on execution feedback. Both are compared with this paper on 30 services and 914 tools simulated in ToolSimulator.

The authors position X-AgentRed as first selecting cross-service recipes from an explicit tool dependency graph, then instantiating them into scenarios; execution feedback updates recipes and scenarios through service replacement and precondition correction on blocked steps, without retraining the model (Related Work).

X-AgentRed first builds the HarmFlow dependency graph, then samples recipes backward, writes benign scenarios, and finally uses execution feedback to replace tools or correct blocked steps (Figure 2). The red-team agent is Claude Opus 4.8.

Problem setup

• Scenario: x = (u, s0, h), containing a benign request, an initial environment, and a target harm. The trajectory is produced by the target interacting from s0 over the tool registry T, and h is not visible to it. X is the space of acceptable scenarios.
• Harm indicator: H(τ)=1 when execution produces a harmful outcome, otherwise 0. HPR is the number of confirmed harmful candidate scenarios divided by the number of candidate scenarios (Equation 3).
• Taxonomy: Synthesizing existing frameworks yields 10 terminal harm categories H1–H10 and 12 triggering factors F1–F12 (Table 1). Harms are labeled by observed actions and outcomes; factors are labeled on the interactions that initiate the harmful process. Sampling balances target harms without factor quotas. A task that is merely incomplete and harmless is not sufficient to be labeled as harm (Appendix A.1).

HarmFlow graph

• Nodes: Each tool has preconditions Pt, effects Et, and potential harms Ht. Effects are state changes or information that can be produced; preconditions include inputs, resources, and permissions.
• Edge condition: D = (t_i, t_j) ∈ T² | (E_ti × P_tj) ∩ R ≠ ∅. Meaning: if there exists a red-team-accepted pairing between the source tool's effects and the target tool's preconditions, a directed edge is added. R is determined by the red team pairwise over effect–precondition pairs.
• Scale and semantics: Implemented as 914 nodes and 29,367 edges (Figure 2, Appendix A.2). Edges only indicate supporting dependencies, not guaranteed executability. The same dependency can be collaborative under a benign request or expose information; the path itself does not determine safety.

Recipe sampling and instantiation

• Backward random walk: For a target harm h, start from tools whose Ht contains h, trace backward along incoming edges until a predefined maximum path length, then reverse to obtain a tool path. A recipe is that path plus h (Section 3.2; the walk cites Lovász, 1993). The paper does not give a numerical value for that maximum length.
• Instantiation: The red team selects a factor from the factor set and generates a benign request and initial state such that the factor makes the sampled cross-service interaction produce the target harm.
• Quotas: 50 candidate scenarios per round. Diversification and refinement each account for at most 25%, with the rest random sampling (Appendix A.2). Scenario instantiation and user prompt generation do not directly optimize with review feedback (Section 3.3).

Execution feedback

• Successful paths: Replace tools not critical to the harmful outcome with neighboring nodes in the graph, retaining the harmful steps and the selected factor, to examine the same vulnerability across different service combinations. Figure 2 describes this as replacing a benign service with a neighboring service.
• Blocked paths: At the blocked step, identify unsatisfied preconditions, update that tool's preconditions and corresponding edges, rebuild the steps before that step, and concatenate with the original suffix.
• Follow-up user turns: Following STAC's practice, rewrite follow-up user prompts based on target responses and tool outputs until harm occurs or the turn cap is reached (Appendix A.2). The paper does not state this cap.

Success adjudication

• Three conditions: Claude Opus 5 and GPT-5.6 Sol independently inspect the full user request sequence, tool trajectory, and recorded environment state. Both must accept: substantive harm occurred, the triggering conditions are reasonable, and every delivered request remains benign. The agent's own statements cannot alone establish that an action is safe or authorized (Appendix A.5).
• Aggregation: A scenario counts toward HPR only if both reviewers accept it. Missing executions or reviews are not counted as harmful. All three methods use the same set of recorded executions and the same adjudication protocol.

Experimental setup

• Environment: ToolSimulator simulates 30 services and 914 tools; tools in the same service share underlying data. Claude Opus 4.8 generates tool responses based on service history and initial state. Retrieved web content exists in the simulated environment (Appendix A.3, Table 4).
• Model roles: Red teaming, as well as generation and internal search for both baselines, all use Claude Opus 4.8. Targets are GPT-5.6 Sol, GLM-5.2, DeepSeek-V4-Pro, Claude Opus 4.8. Reviewers are Claude Opus 5 and GPT-5.6 Sol. Thus Claude Opus 4.8 is simultaneously the red team, the environment response generator, and one target; GPT-5.6 Sol is simultaneously a target and a reviewer.
• Scale and metrics: N=250 candidate scenarios per method per target, one round per configuration (Section 4.1). X-AgentRed has 50 per round for five rounds, with later rounds depending on previous feedback. HPR is the proportion of candidates producing confirmed harm. Tool coverage is the percentage of the 914 tools used at least once, and the denominator for the DeepSeek-V4-Pro row is also 914.
• Baseline protocol: STAC shares one ordered corpus of 250 scenarios across the four targets, executed end-to-end, with up to three adaptive follow-up user turns; this differs from STAC's original precursor-replay. TRACE's task source specifies source–sink tool pairs, declared paths contain two tools by construction, and no follow-up user turns are added after template search (Appendix A.4, B.3). The paper states that the comparison does not equalize generation cost or the number of calls to the target model.
• Interface constraints: DeepSeek-V4-Pro accepts at most 256 tools. Each scenario includes the tools needed for that scenario, and the rest are filled in a fixed order (Appendix A.3).
• Adjudication: The main results require agreement between the two reviewers. The paper states one round per configuration and does not report multiple-seed repetitions.

Main results

According to Table 2 (HPR and tool coverage both in %; 250 candidates per target; two-reviewer agreement):

Target	HPR Ours / STAC / TRACE	Coverage Ours / STAC / TRACE
Claude Opus 4.8	7.6 / 6.0 / 0.4	37.1 / 15.1 / 19.4
GPT-5.6 Sol	19.6 / 14.0 / 2.4	47.6 / 14.0 / 23.3
DeepSeek-V4-Pro	18.0 / 9.6 / 2.0	37.4 / 12.4 / 17.5
GLM-5.2	31.6 / 12.4 / 6.0	43.8 / 14.2 / 34.6
Four-target average	19.2 / 10.5 / 2.7	41.5 / 13.9 / 23.7

• Authors' interpretation: The authors state that averaged over the four targets, HPR is 1.77 times STAC and 10.36 times TRACE; the largest relative gain over STAC is on GLM-5.2, followed by DeepSeek-V4-Pro. In Table 2, Ours has higher HPR than both baselines on every target.
• Coverage and exceptions: The authors state that average tool coverage is 2.99 times STAC and 1.84 times TRACE; per-target coverage is 37.1%–47.6%, i.e., 2.5–3.4 times STAC (Section 4.4). At the same time, TRACE's tool coverage is higher than STAC on every target, but HPR is lower (Table 2).
• Cumulative: Figure 3 sums scenario counts and success counts across the four targets. The authors state that at every tested budget, the number of successes is higher than both baselines; totals are X-AgentRed 192, STAC 105, TRACE 27.

Self-improvement

• What was tested: Figure 5 compares recipe diversification (Aug), precondition refinement on paths that previously produced no harm (Refine), and random sampling. Random sampling includes the initial round; Figure 5 aggregates across targets. The 32.8% contribution is first computed within each target and then averaged (Appendix B.1).
• Results: Aug HPR is 27.7%, random sampling is 18.0%; Refine on previously failed paths is 15.5%. The two strategies together account for 32.8% of confirmed harmful results.
• Authors' interpretation: The authors state that handling unsatisfied preconditions can discover additional harmful results. Appendix C illustrates replacement and refinement with examples; the scenario text is not reproduced here.

Path breadth and length

• Breadth: Average distinct tools and services per turn: X-AgentRed 13.1 and 5.0, STAC 4.6 and 2.7, TRACE 2.0 and 1.9 (Figure 6). The ten most used tools account for 24.3%, 43.2%, 48.9% of usage (Figure 4(b), each tool counted at most once per turn). TRACE's tool and service statistics describe declared paths, while call counts describe execution (Appendix A.6).
• Length: Executed tool calls average 23.2, 7.8, 3.3 (Figure 6). The authors state that averaged over the four targets, this is 2.92 times STAC and 7.53 times TRACE, and includes adaptive follow-up interactions. TRACE's call counts exclude repeated executions in template search. The authors state that failed turns also add follow-up rounds, so execution length reflects both workflow and adaptation strategy.
• Exceptions: The authors state that longer planned paths do not stably yield more harmful results; longer paths may expose more intermediate actions to safety checks, increasing the chance of detection or intervention (Section 4.4). Tool-set overlap of successful paths is low for all methods and does not uniformly favor X-AgentRed. Figure 4(d) averages only GPT-5.6 Sol, GLM-5.2, and DeepSeek-V4-Pro for all three methods, because TRACE has only one harmful scenario on Claude, making pairwise similarity undefined.

Cost and review sensitivity

• Generation calls: Table 3 counts preparation, generation, and validation, including retries and internal search; it excludes execution against the target, simulated evaluation trajectories, adaptive follow-up user prompts, or review results. Per candidate scenario: X-AgentRed 22.10, STAC 55.00, TRACE 103.24. Shared preparation is counted once and divided across 1000 candidates for the four targets (Appendix B.4).
• Review: In Table 9, Claude Opus 5 is higher than GPT-5.6 Sol on three of the four targets, with the order reversed on the Claude target (16.4% vs. 19.6%). Accepting either reviewer relative to requiring agreement raises HPR by 15.2–30.4 percentage points. The authors state this measures sensitivity, not review accuracy. The AND column is identical to Ours in Table 2.
• Opening turns: The opening-turn evidence in Table 6 comes from full-trajectory adjudication, not an independent evaluation of opening prefixes. The four-target total is 1.1%, and the full trajectory is 19.2%. The authors state this cannot separate the effect of adaptive prompting.

Other ablations and analyses

• Per-round HPR (Table 5, 50 scenarios per round; later rounds depend on feedback, not independent trials): GLM-5.2 Cycle 2 is 44.0%; Claude Opus 4.8 Cycle 1 is 10.0%, Cycle 2 and Cycle 5 are 6.0%.
• Opening-turn evidence share: GPT-5.6 Sol 0.8% vs. full trajectory 19.6%, GLM-5.2 1.6% vs. 31.6%, DeepSeek-V4-Pro 0.8% vs. 18.0%, Claude Opus 4.8 1.2% vs. 7.6% (Table 6).
• Median planned length of successful recipes is 8–9 steps, maximum 12; the 192 successes counted by target include 190 distinct planned paths, with variation in terminal tools. Repeated planned paths account for 2.0%–2.8% of all candidates (Section 4.4, Table 7).
• Total unique sequences within targets: X-AgentRed 998, STAC 866. For successful X-AgentRed trajectories, median tool-set Jaccard similarity by target is 0.000–0.065. The authors state these are structural diversity, not distinct causal failure mechanisms.
• The authors state that higher overlap between planned and executed tools is associated with higher harm rates, but no causal effect of recipe conditions is established (Figure 7 text). Figure 8(b) aggregates HPR by planned length; the text does not list per-bin numbers, only stating that length alone does not stably predict harm.
• Figure 9: X-AgentRed covers all 10 harm categories; harm categories can overlap, and factors are mutually exclusive. Scenarios where both reviewers confirmed harm but did not share a category are X-AgentRed 11, STAC 6, TRACE 1, and remain in the denominator. The authors state that manual inspection of all 192 harmful trajectories found no reward hacking in these cases (Section 3.3).

Limitations and future directions noted by the authors

• Graph maintenance cost: Limitations states that maintaining HarmFlow as services evolve incurs additional computational cost, because new or modified tools require updated representations and re-evaluation of affected dependencies. The paper does not state other limitations or future experiments in Limitations, Discussion, Conclusion, or Future Work.

Experimental coverage

• Targets and rounds: Four target agents, 250 candidate scenarios each, one round per configuration (Section 4, 4.1). The paper does not report multiple-seed repetitions, nor a specific cap on X-AgentRed's follow-up user rounds.
• Roles and environment: Red-team generation, tool response generation, and both baselines' generation all use Claude Opus 4.8. The environment is 30 services and 914 tools in ToolSimulator; DeepSeek-V4-Pro has at most 256 tools per scenario (Appendix A.3, Table 4).
• Comparison protocol: Baselines are STAC and TRACE. STAC shares one 250-scenario corpus across the four targets, and the main results are end-to-end execution, not STAC's original precursor-replay. The comparison does not equalize generation cost or calls to the target model (Appendix A.4).
• Adjudication: Main HPR requires agreement between Claude Opus 5 and GPT-5.6 Sol. Appendix B.5 additionally reports single-reviewer and OR/AND; the authors state these comparisons measure sensitivity, not review accuracy.
• Cost scope: Generation calls in Table 3 exclude executing scenarios against the target, simulated evaluation trajectories, adaptive follow-up user prompts, or review results (Appendix B.4). The paper reports manual inspection of 192 harmful trajectories.

X-AgentRed red-teams personal agents in a multi-service ecosystem to discover unauthorized harm that arises when benign requests accumulate through tool dependencies.

• Gap: The authors state that having an LLM directly propose tool chains or decompose harmful goals, even with execution feedback, may still favor combinations familiar from training.
• Approach: Claude Opus 4.8 reads tool descriptions, consolidates "whether effects can satisfy preconditions" into HarmFlow (914 nodes, 29,367 edges), samples recipes backward from tools connected to the target harm, and then generates benign requests that can still be completed safely. On success, it replaces non-critical tools and retains the harmful steps; when blocked, it updates the preconditions of that step and rebuilds the prefix. Claude Opus 5 and GPT-5.6 Sol must both confirm the harm, the triggering conditions, and that the requests remain benign.
• Main results: ToolSimulator contains 30 services and 914 tools. Four targets, 250 scenarios each, one round. Table 2 average HPR is 19.2%, STAC is 10.5%, TRACE is 2.7%; average tool coverage is 41.5%, 13.9%, 23.7%. The authors state that averaged over the four targets, HPR is 1.77 times STAC and 10.36 times TRACE, and tool coverage is 2.99 times and 1.84 times. GLM-5.2 HPR is 31.6%, Claude Opus 4.8 is 7.6%. On every target, Ours has higher HPR and tool coverage than both baselines.
• Other measurements: Executed calls average 23.2; the authors state that averaged over the four targets, this is 2.92 times STAC and 7.53 times TRACE, and includes adaptive follow-up rounds. Diversified HPR is 27.7%, random sampling is 18.0%, and refinement on previously failed paths is 15.5% (Figure 5). The authors state that longer plans do not stably correspond to higher HPR. Generation calls per scenario are 22.10, STAC is 55.00, TRACE is 103.24, excluding target execution and review (Table 3).
• Authors' conclusion: The authors state that combining explicit dependency structure with feedback-driven expansion and refinement helps discover harmful behaviors that emerge across services. Limitations also states that after services change, tool representations must be updated and dependencies re-evaluated, and graph maintenance has additional computational cost.