Securing Multi-Agent Systems by Design
Securing Multi-Agent Systems by Design
The authors propose ReMA, a response manipulation attack against multi-agent systems, and SecMAS, a secure aggregation rule; the authors state that baseline aggregation rules reach an ASR of up to 1.00 under ReMA, and SecMAS reduces the ASR of each attack to close to 0.00.
The attacker controls one or more malicious agents and can arbitrarily manipulate the responses they send to the manager agent; multiple malicious agents can collude; the goals are divided into untargeted (deviating from the ground-truth final solution) and targeted (producing the attacker-specified final solution). The attacker has white-box access to the malicious agent; knowledge of benign agent responses, the aggregation rule, and the manager agent backbone LLM is divided into two levels: known/unknown. The victim system is a multi-agent system running according to the framework in Figure 1.
- In multi-agent systems, a malicious agent controlled by an attacker can manipulate the manager agent's aggregation result through carefully constructed responses, causing the final solution to be incorrect. The authors argue that existing attacks lack optimization against the aggregation rule, and existing aggregation rules lack theoretical safety guarantees.
- The authors propose ReMA: it formalizes untargeted/targeted attacks as an optimization problem, iteratively optimizing the malicious response with three LLM components: a response generator, a surrogate manager agent, and an attack critic; they also propose SecMAS: it selects, by semantic similarity between responses, the response most consistent with a sufficiently large subset as the aggregated solution, and provides a theoretical safety guarantee.
- Evaluated across five application scenarios, six aggregation rules, and multiple threat models: baseline aggregation rules reach an ASR of up to 1.00 under ReMA; SecMAS reduces the ASR of all attacks (including adaptive attacks targeting it) to close to 0.00, and its utility difference from baseline aggregation rules is not statistically significant at the 0.05 level.
- The authors point out that the theoretical guarantees depend on the assumption that benign agents' response embeddings cluster near the ground truth, and that the guarantees may fail when the assumption does not hold; SecMAS may still be vulnerable to stronger adaptive attacks. The experiments cover only settings such as 10 agents per group and 2 malicious agents, and no human evaluation or inter-rater agreement is reported.
- Evaluated models
- Gemma-4-31BQwen-3.8-27BLlama-4-Scout-17B-16EGPT-5.6-SolGPT-5.6-TerraGPT-5.6-Luna
- Benchmarks
- 作者用 GPT-5.6-Sol 生成的五场景(education、finance、healthcare、legal、news)用户任务集,每场景 100 个经 Claude Opus 5 验证的任务,共 500 个
- Metrics
- ASRASVPNA
Researchers propose ReMA, a response manipulation attack against multi-agent systems, and SecMAS, a defensive aggregation rule with theoretical safety guarantees. Multi-agent systems use aggregation rules to combine the responses of individual agents, and a malicious agent can manipulate the final result by submitting malicious responses; ReMA formalizes untargeted and targeted attacks as optimization problems and iteratively refines malicious responses based on feedback from a surrogate multi-agent system to maximize attack effectiveness. SecMAS uses semantic similarity between agent responses to select the response most consistent with a sufficiently large subset of responses as the aggregation result. Evaluations show that existing aggregation rules are all vulnerable to ReMA, and that SecMAS withstands all evaluated attacks, including adaptive variants, while achieving utility comparable to existing aggregation rules.
Paper analysis
What problem does this paper attempt to solve?
Malicious agents can manipulate the aggregation result of a multi-agent system; the paper proposes the ReMA attack and the SecMAS secure aggregation rule.
Core problem: in multi-agent systems, a malicious agent controlled by an attacker can manipulate the manager agent's aggregation result by constructing malicious responses, causing the system to produce an incorrect final solution, and existing attacks and defenses have not systematically addressed this problem.
- Problem scenario: a multi-agent system decomposes a user task into subtasks, delegates them to groups of agents, and then merges the responses using an aggregation rule. The authors argue that this structure allows a malicious agent to indirectly inject malicious content into the manager agent.
- Shortcomings of existing attacks: the authors state that heuristic attacks (TAMAS, MAST, AiTM) lack systematic optimization of the malicious response; existing optimization-based attacks (such as ObliInjection) do not explicitly consider the aggregation rule and often require white-box access to the target LLM, with limited threat model settings that may underestimate the risk.
- Shortcomings of existing defenses: the authors state that existing aggregation rules mostly evaluate responses one by one, do not consider relationships among responses within the group, and lack theoretical safety guarantees against malicious agents.
- Contributions of the paper: propose the response manipulation attack ReMA (formalizing untargeted/targeted attacks as an optimization problem and solving it iteratively) and the secure aggregation rule SecMAS (selecting responses by semantic consistency), and provide a theoretical safety guarantee and experimental evaluation.
- Threat model:
- Goal: untargeted attacks cause the final solution to deviate from the ground-truth; targeted attacks cause the system to produce the attacker-specified final solution.
- Capability: the attacker controls one or more malicious agents and can arbitrarily manipulate the responses they send to the manager agent; multiple malicious agents can collude.
- Knowledge: white-box for the malicious agent (backbone LLM, system prompt, subtask); for the three items of benign agent responses, aggregation rule, and manager agent backbone LLM, two levels are considered respectively: known/unknown.
- Victim system: a multi-agent system running iteratively according to the three-step framework in Figure 1 (subtask delegation, subtask solving, aggregation).
What related research exists?
Related work falls into three categories: multi-agent malicious agent attacks, secure aggregation rules, and federated learning secure aggregation; the paper points out the shortcomings of each.
Malicious agent attacks in multi-agent systems
- Heuristic attacks: TAMAS [9] induces malicious responses by modifying the malicious agent's system prompt; MAST [24] and AiTM [7] directly prompt the LLM to generate malicious responses; Prompt Infection [11] studies prompt injection between LLMs within multi-agent systems. The paper points out that such attacks lack systematic optimization of the malicious response, and their attack effectiveness is limited.
- Optimization-based attacks: ObliInjection [21] uses gradient-based optimization to construct malicious responses, causing the LLM agent to produce the output desired by the attacker; [27] and [28] also use optimization methods to attack LLM agents. The paper points out that such attacks do not explicitly consider the manager agent's aggregation rule, and their threat model settings are limited (e.g., ObliInjection requires white-box access to the manager agent's backbone LLM).
- Relationship to prompt injection: The authors analogize malicious agent attacks to prompt injection attacks [13, 22]: a malicious agent is equivalent to indirectly injecting malicious content into the manager agent's input.
Secure aggregation rules
- CP-WBFT [30]: assigns a confidence score to each response and selects the one with the highest score as the aggregated solution; it has two variants, PCP (agents self-report confidence) and HCP (a trained confidence estimation model).
- DecentLLMs [8]: uses multiple reviewer LLMs to score responses along multiple dimensions, takes the geometric median, and then selects the highest-scoring response.
- The paper points out that these rules mostly evaluate responses one by one, do not consider relationships among responses within the group, and lack theoretical safety guarantees; simple mechanisms such as Summarization were broken by optimization-based attacks in the paper's experiments.
Federated learning and secure aggregation
- Federated learning secure aggregation: [2, 3, 26] and others defend against malicious model updates sent by malicious clients [5]. The paper points out that the key difference is that federated learning aggregates numerical vectors, whereas multi-agent systems aggregate text, so these rules cannot be directly applied; coordinate-wise rules [26] are also not applicable to semantic embeddings, because a single embedding dimension has no independent semantics.
Baselines and benchmarks: The attack baselines are TAMAS, MAST, AiTM, and ObliInjection; the aggregation rule baselines are Summarization, CP-WBFT-PCP, CP-WBFT-HCP, DecentLLMs, and DeSummary (using DataSentinel [14] for prompt injection detection followed by summarization). The paper follows the five application scenarios of TAMAS [9], but because the TAMAS tasks have no ground-truth, the authors generated 500 user tasks with ground-truth themselves.
Positioning of this paper: The authors state that ReMA is an optimization-based attack that explicitly considers the aggregation rule and supports multiple realistic threat models; SecMAS explicitly exploits relationships among responses within the group and provides a theoretical safety guarantee against malicious agents.
How does the paper address this problem?
ReMA formalizes the attack as an optimization problem and solves it iteratively with three components; SecMAS selects the response most consistent with the majority by semantic similarity.
The overall approach has two parts: on the attack side, ReMA is proposed, which formalizes response manipulation as an optimization problem and iteratively optimizes the malicious response using feedback from a surrogate multi-agent system; on the defense side, SecMAS is proposed, which selects the aggregated solution based on the intuition that benign majority responses are semantically consistent, and provides a theoretical guarantee.
Problem formalization (ReMA)
- Untargeted attack: find a set of malicious responses that minimizes the probability that the manager agent produces the ground-truth final solution, i.e., min Pr(â(t, Gh) = y∗); equivalently, in each malicious iteration, minimize the probability that the aggregation rule produces the ground-truth solution for that subtask.
- Targeted attack: find a set of malicious responses that maximizes the probability of producing the attacker-specified final solution y_tar; in each malicious iteration, GPT-5.6-Sol first derives the target aggregated solution y_i^tar for that subtask, and then the probability that the aggregation rule produces it is maximized.
- The authors state that this formalization applies to any aggregation rule conforming to the framework in Figure 1, and can therefore be used to construct adaptive attacks against new rules (including SecMAS).
Iterative solution of ReMA
- Three components: a response generator (generates/improves malicious responses), a surrogate manager agent (simulates the target system's aggregation process), and an attack critic (evaluates the surrogate aggregated solution and gives a 0–10 score and reflective feedback), all based on LLMs; in the experiments, GPT-5.6-Sol is used as the backbone for the response generator and the attack critic.
- Three steps per round (Figure 2): generate b=4 candidate malicious response sets → merge with benign responses and aggregate via the surrogate manager agent → the attack critic scores them and retains the c=2 candidates with the highest scores, with the feedback entering the next round; after at most omax=15 rounds, the one with the highest total score is taken.
- Candidate diversity: the authors encourage candidate sets to explore different strategies, such as explicit misleading instructions and plausible-looking reasoning packaging; Appendix Figures 9 and 10 give the complete system prompts for the response generator and the attack critic.
Extensions under different threat models
- Unaware of benign responses: use different backbone LLMs, frameworks, and random seeds to generate d=8 surrogate benign responses in place of the real benign responses.
- Unaware of the aggregation rule: randomly assume a surrogate aggregation rule and switch it every ∆=3 rounds; finally evaluate candidates using all surrogate rules used, and take the one with the highest average score.
- Unaware of the manager backbone LLM: randomly select three of six backbone LLMs to form a surrogate LLM pool, and the attack critic jointly evaluates the aggregated solutions under each surrogate model, avoiding overfitting to a single model.
SecMAS aggregation rule
- Intuition: when benign agents are the majority, the semantic consistency among benign responses is higher than their consistency with malicious responses.
- Scoring and selection: use an embedding model (Qwen3-Embedding-8B in the experiments) to convert responses into embeddings and compute pairwise cosine similarities; for each response, take its m_i − f_i − 2 most similar responses as neighbors, and the response score is the sum of similarities with its neighbors S(rik) = ∑l ∈ Nik sikl; select the response with the highest score as the aggregated solution.
- Rationale for the number of neighbors: taking m_i − f_i − 2 neighbors leaves f_i + 1 non-neighbors, at least one of which is benign, so that the score of a benign response can be bounded by a lower bound on the similarity between benign responses; the paper states that this also supports the theoretical guarantee.
- Theoretical guarantee: under Assumption 1 (benign response embeddings concentrate near the ground-truth solution embedding with high probability; Figure 4 gives the empirical distribution), Theorem 1 gives a lower bound on the cosine similarity between the aggregated solution and the ground-truth solution under untargeted attacks, and Theorem 2 gives an upper bound on the similarity between the aggregated solution and the attacker-specified solution under targeted attacks; the proofs are in Appendices A and B.
- Adaptive attacks: when the attacker knows the aggregation rule, the surrogate manager agent is configured with the same rule and its mechanism is described to the response generator; the authors state that in the experiments SecMAS remains safe against such adaptive attacks.
What experiments did the paper conduct?
Five scenarios, 500 tasks, six aggregation rules, multiple threat models: the baseline rules were broken by ReMA, and under SecMAS the ASR of each attack is close to 0.
Experimental Setup
- Multi-agent systems: one system for each of five application scenarios (education, finance, healthcare, legal, news), each containing one manager agent and a group of agents in four roles, with 10 agents per group by default; each agent randomly combines two frameworks (AutoGen, CrewAI) and six backbone LLMs (Gemma-4-31B, Qwen-3.8-27B, Llama-4-Scout-17B-16E, GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.6-Luna).
- Task data: GPT-5.6-Sol was used to generate user tasks with ground truth, Claude Opus 5 verified correctness, and 100 were kept per scenario, 500 in total; the ground-truth aggregate solutions of subtasks were also generated by GPT-5.6-Sol.
- Attack configuration: for each user task, one group was randomly selected and 2 of its agents were designated as malicious; ReMA parameters omax=15, b=4, c=2; the target solution of the targeted attack was generated by GPT-5.6-Sol.
- Baselines: attack baselines TAMAS, MAST, AiTM, ObliInjection (when the backbone LLM is unknown or the model is closed-source, five shadow LLMs are used: Ministral-3-8B, Llama-3.1-8B, Qwen3-8B, Falcon-H1-7B-Instruct, Gemma-4-12B); aggregation rule baselines Summarization, CP-WBFT-PCP, CP-WBFT-HCP, DecentLLMs, DeSummary (detector is DataSentinel).
- Metrics and judging: ASR is judged by GPT-5.6-Sol as to whether the final solution achieves the attack goal (outputs Yes/No, prompt in Appendix Figure 13); ASV is a continuous value based on cosine similarity ((1−cos)/2 for untargeted, (1+cos)/2 for targeted); PNA measures utility when there is no attack. SecMAS sets f_i=2.
- Number of repetitions: in the utility experiments, each aggregation rule was run 5 times per scenario, reporting mean and standard deviation, with a two-sided Welch's t-test; the number of repetitions for the attack experiments is not reported in the paper.
Main Results
According to Table 1 and 2, the ASR of ReMA on each aggregation rule under the strongest threat model (knowing benign responses, the aggregation rule, and the backbone LLM) (untargeted/targeted, range across five scenarios):
Aggregation rule untargeted ASR targeted ASR Summarization 0.93–0.99 0.80–0.91 CP-WBFT-PCP 1.00 1.00 CP-WBFT-HCP 0.84–0.97 0.75–0.87 DecentLLMs 0.83–0.94 0.74–0.85 DeSummary 0.89–0.94 0.72–0.90 SecMAS 0.00–0.02 0.00 - The authors state that baseline aggregation rules have higher ASR and ASV than the no-attack baseline under all types of attacks, and that ReMA achieves the highest ASR/ASV under most threat model settings; under the strictest threat model (knowing none of the three), ReMA's untargeted/targeted ASR against Summarization (News) is still 0.61/0.49.
- The authors state that SecMAS has the lowest ASR/ASV against all attacks (including adaptive ReMA using SecMAS as the surrogate rule) under all threat models and scenarios, comparable to when there is no attack; under the strongest threat model, the ASR of each attack drops to close to 0.00 (the highest in the SecMAS row of Table 2 is 0.02).
- The authors explain that untargeted ASV being generally lower than targeted ASV is due to the definition: when embeddings are orthogonal, untargeted ASV is only 0.50, but the final solution may already deviate markedly from the ground truth.
Utility Comparison
- What was measured: PNA of each aggregation rule when there is no attack (Table 3), with each rule run 5 times per scenario.
- Results: the mean PNA of the six rules all fall between 0.88 and 0.91; the two-sided Welch's t-test in Table 4 shows that all p-values between SecMAS and each baseline are greater than 0.05 (minimum 0.130).
- Authors' interpretation: no statistically significant difference was detected at the 0.05 significance level, and SecMAS's utility is comparable to the baselines.
Ablation and Parameter Analysis
- LLM pool size (Figure 5, knowing the aggregation rule and backbone LLM, not knowing benign responses): ASR/ASV decrease as the backbone LLMs in the pool increase from 2 to 8; the authors believe that increasing backbone LLM diversity can improve system security.
- Number of agents within a group (Figure 6, same threat model): when there is only 1 agent in the group, ReMA's untargeted/targeted ASR are both 1.00 (single point of failure); as the number of agents increases to 5–20, ASR/ASV decrease slightly.
- Number of malicious agents (Figure 7, same threat model): ASR/ASV rise as malicious agents increase; the authors state that when 30% of agents are malicious, SecMAS still maintains very low ASR/ASV.
- Cross-aggregation-rule transfer (Table 5, knowing benign responses and the backbone LLM, not knowing the aggregation rule, with a single surrogate rule fixed): ASR/ASV are generally slightly lower than when switching surrogate rules, but still higher than all baseline attacks; transfer between similarly designed rules is stronger, for example, when attacking Summarization with DeSummary as the surrogate, untargeted ASR/ASV are 0.92/0.34, higher than 0.77/0.28 under the same threat model in Table 1.
- Effect of f_i (Figure 8, malicious agents fixed at 2): when f_i=1 (less than the actual number of malicious agents), SecMAS is broken by ReMA; when f_i≥2 the defense is effective, and PNA hardly changes with f_i; the authors suggest that developers conservatively set f_i = m_i/2 − 1 when the number of malicious agents is unknown.
What are some points that can be further explored?
The authors acknowledge that the theoretical guarantee relies on assumptions about benign response quality, and that strong adaptive attacks may still be effective; the experiments cover five scenarios and fixed-scale configurations.
Limitations and future directions noted by the authors
- Theoretical assumptions may not hold (Section 9): the theoretical safety analysis relies on the assumption that benign agents can correctly solve subtasks, i.e., that their response embeddings concentrate near the ground-truth solution embedding; the authors state that when this assumption is violated, SecMAS's theoretical guarantee may not hold (the experiments provide only empirical support via the distribution in Figure 4).
- Stronger adaptive attacks (Section 9): the authors state that SecMAS may still be vulnerable to stronger adaptive attacks, especially in scenarios where the response quality of benign agents varies greatly and the theoretical assumption does not hold; exploring such adaptive attacks is future work.
Scope of the experiments
- Tested system configurations: five application scenarios (education, finance, healthcare, legal, news), each system with one manager agent plus four agent groups, by default 10 agents per group, of which 2 are malicious; the backbone LLMs are six: Gemma-4-31B, Qwen-3.8-27B, Llama-4-Scout-17B-16E, GPT-5.6-Sol/Terra/Luna, and the frameworks are two: AutoGen and CrewAI.
- Attack and defense coverage: attack baselines TAMAS, MAST, AiTM, ObliInjection; aggregation rules Summarization, CP-WBFT-PCP, CP-WBFT-HCP, DecentLLMs, DeSummary, SecMAS; the threat models cover eight combinations of knowledge of the three items: benign responses, aggregation rule, and backbone LLM (some rules do not involve the backbone LLM item).
- Sample size: 100 user tasks per scenario, 500 in total; the utility experiments run 5 times per rule per scenario with a t-test; the paper does not report the number of repetitions for the attack experiments.
- Ablation scope: LLM pool size 2–8, number of agents per group 1–20, number of malicious agents 1–4, f_i from 1–4, cross-rule transfer; the proportion of malicious agents goes up to 30% (the related statement in Figure 7).
- Evaluation method: ASR is judged by GPT-5.6-Sol, and task correctness is verified by Claude Opus 5; the paper does not report human evaluation or the agreement between the reviewer models and humans.
Summarize the main content of the paper
The ReMA attack breaks existing aggregation rules, and SecMAS aggregates by semantic consistency and resists the tested attacks in experiments and theory.
A multi-agent system security paper that proposes both an attack and a defense: on the attack side, ReMA exposes the vulnerability of existing aggregation rules; on the defense side, SecMAS provides secure aggregation with theoretical guarantees.
- Problem: the attacker controls a minority of malicious agents and, by constructing responses, manipulates the manager agent's aggregation result, causing the system to produce a final solution that deviates from the ground truth (untargeted) or that the attacker specifies (targeted).
- ReMA: formalizes the two types of attacks as optimization problems, iteratively optimizing malicious responses using three LLM components—response generator, surrogate manager agent, and attack critic (omax=15 rounds)—and extends to threat models where benign responses, the aggregation rule, or the backbone LLM are unknown.
- SecMAS: uses an embedding model to compute cosine similarity between responses, scores each response by summing its m_i − f_i − 2 most similar neighbors, and selects the highest-scoring response as the aggregate solution; under the assumption that benign response embeddings cluster near the ground truth, Theorem 1 and 2 give theoretical guarantees against untargeted/targeted attacks, respectively.
- Attack effectiveness: under the strongest threat model, ReMA's untargeted/targeted ASR against CP-WBFT-PCP reaches 1.00, and untargeted ASR against the remaining baseline rules is 0.83–0.99 (Table 1, 2); under the strictest threat model, it is still 0.61/0.49 against Summarization (News).
- Defense effectiveness: SecMAS reduces the ASR of each attack, including adaptive ReMA, to 0.00–0.02 under all threat models and scenarios (Table 2); when there is no attack, its utility has no statistically significant difference from the baselines (minimum t-test p-value 0.130, Table 4).
- Authors' conclusions and boundaries: the authors state that when malicious agents are a minority, selecting the response most consistent with a sufficiently large subset can defend against such manipulation; they also acknowledge that the theoretical guarantees depend on assumptions about benign response quality, and that stronger adaptive attacks may still threaten SecMAS.