Skip to content

Kaer: safety-aware replanning for multi-agent systems

Kaer: Security-Aware Replanning for Multi-Agent Systems

arXiv: 其他 AI 安全论文· arXiv:2611.24839· Tushin Mallick, Evan Rose, Matthew Laws, Alina Oprea, Cristina Nita-RotaruNortheastern UniversityIndexed by this site · 13 hr ago· Originally publishedDefenses· Its place in the scan record →

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

The authors propose a multi-agent safety replanning framework, Kaer, and a benchmark, Cintra; on gpt-5.6-sol, Kaer achieves a safety rate of 92.3%, higher than ControlValve's 69.8%.

Threat modelTrusted components are the orchestrator, LLM planner, agent registry, MCP and LLM servers, and policies; agents are untrusted and may, within their permissions, invoke unauthorized capabilities, forge return values, and fully control the content and timing of the payload returned to the orchestrator, using prompt injection to manipulate replanning. Out of scope: misuse within an agent's internal loop, compromise of trusted components, threats propagated through subtask outputs, and output correctness verification.

Questions
Orchestrator-based multi-agent systems need to replan during execution, but a malicious agent can embed instructions in feedback to hijack replanning; the authors state that existing defenses only do tool-level access control or pre-execution static policies, and no one protects the replanning stage.
Methods
Kaer lets a failed agent return only failure templates from a fixed set, the planner can only modify unfinished subtasks using seven deterministic repair operators, and the repaired plan must pass task policy and agent policy validation; the authors also build the Cintra benchmark with 14 tasks and 14 attack goals.
Experiments and results
On Cintra, with gpt-5.6-sol, Kaer achieves a safety rate of 92.3% and utility of 64.3%, higher than ControlValve's 69.8%/54.3% and Magentic-One's 35.0%/91.4%; residual attacks propagate mainly through successful outputs, and the authors state that adding LLM output verification raises the defense rate to 96.8%.
Limitations and future work
The authors note that failure templates omit runtime context, so tasks requiring information such as data source location cannot be repaired; output validation uses an LLM whose judgment can be influenced by injection; adaptive attackers can still induce repairs within policy scope. The evaluation covers only 3 models and 14 tasks, and the malicious agent is fixed as filesurfer.
Experimental setupgpt-5.6-sol、gpt-5.6-terra et al. · Cintra · Utility、Security et al.
Evaluated models
gpt-5.6-solgpt-5.6-terradeepseek-v4-pro
Benchmarks
Cintra
Metrics
UtilitySecurityAttack success rate
AI summaryKaer is a safety-aware replanning framework for multi-agent systems that prevents malicious agents from embedding instructions in feedback to hijack replanning. It passes diagnostic information to replanning as abstract templates, and plan modifications are limited to deterministic repair operators and must satisfy preset policies. On the Cintra benchmark, Kaer is more robust than the undefended Magentic-One and the state-of-the-art defense ControlValve, with utility comparable to the latter; abstract failure templates slightly limit utility on some tasks.

Kaer is a safety-aware replanning framework for multi-agent systems that prevents malicious agents from embedding instructions in feedback to hijack replanning. It passes diagnostic information to replanning as abstract templates, and plan modifications are limited to deterministic repair operators and must satisfy preset policies. On the Cintra benchmark, Kaer is more robust than the undefended Magentic-One and the state-of-the-art defense ControlValve, with utility comparable to the latter; abstract failure templates slightly limit utility on some tasks.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    The authors address the problem that the replanning stage of multi-agent systems is hijacked by malicious agent feedback, while existing defenses do not cover this stage.

    Core problem: when an orchestrator-style multi-agent system replans during execution, a malicious agent can hijack the replanning by injecting instructions through feedback, and the authors state that existing defenses do not protect this stage.

    • Scenario and importance: many tasks cannot be fully planned in advance, and the orchestrator needs to dynamically revise the plan based on agent feedback; the authors state that this lets a malicious agent induce unauthorized plan changes, such as adding subtasks or steering execution toward the attacker's goals (i.e., control hijacking).
    • Shortcomings of existing methods: the authors state that existing defenses only do per-tool access control or pre-execution static policies; if ControlValve covers all replanning scenarios in advance, the policy becomes too permissive, and if it regenerates the policy at each step, it degenerates into a ReAct-style agent and inherits indirect prompt injection vulnerabilities.
    • Core observation: the planner only needs to know the reason for failure to repair the plan, and does not need the agent's free text; restricting feedback to failure templates from a fixed vocabulary can bound the malicious agent's influence on repair.
    • What the paper proposes: the secure replanning framework Kaer (template feedback + deterministic repair operators + policy validation), and the accompanying benchmark Cintra (tasks requiring replanning + attack goals targeting replanning).
    • Threat model (Section 3):
      • Goals: induce the planner to grant capabilities forbidden by the task policy, route output to unauthorized agents, add/remove/modify subtasks, or create infinite replanning loops.
      • Knowledge: the malicious agent only sees its own subtask instructions, capabilities and inputs, and does not see the global task or the full plan.
      • Capabilities: full control over the content and timing of the returned payload, and can forge return values and embed prompt injection within its permissions.
      • Victim system and trust boundary: the orchestrator, LLM planner, agent registry, MCP/LLM servers and policies are trusted; misuse inside an agent, compromise of trusted components, threats propagated through subtask output, and output correctness validation are out of scope.
  2. What related research exists?

    Related work falls into three groups: replanning and failure analysis, runtime defenses, and plan-level defenses; the authors state that none of them protects the replanning process itself.

    Replanning and failure analysis in multi-agent systems

    • Magentic-One [1] triggers plan regeneration when a task stalls; other orchestrators replan after verifying step results [5], or repair plans through failure-driven workflow adjustment [6], role reassignment [7], and system self-correction [8]. The authors note that these works optimize only utility and impose no behavioral constraints on agent execution.
    • Cemri et al. [24] classify multi-agent failures into three categories: system design flaws, communication breakdowns, and missing verification; Kaer's failure templates are built on this taxonomy. Other failure attribution directions include causal analysis, error dependency graphs, and counterfactual replay [28]–[38].

    Runtime defenses (screening execution)

    • Information flow methods propagate data provenance or security labels to prevent untrusted input from changing control flow or invoking sensitive tools [11], [39]–[44]; privilege-reduction methods enforce least-privilege tool policies or isolated execution environments [11], [45]–[47]; trace-based systems dynamically synthesize policies [48]–[51].
    • In multi-agent settings, firewalls and monitors filter inter-agent messages [52], [53], gateways/proxies validate tool calls [54]–[58], and guard agents and formal rule languages inline-check actions [59]–[61]. The authors note that these mechanisms make only binary allow/block decisions and do not adjust the underlying plan.

    Plan-level defenses (constraining the plan)

    • Generate static plans from trusted input [11], [44], or embed policy constraints into the planner [62]. The authors note that although some work emphasizes that policy enforcement must be combined with replanning [63] and some attacks directly hijack planning [64], existing systems do not protect the replanning process itself from adversarial manipulation.

    Baselines and benchmarks

    • The comparison baselines are Magentic-One (an undefended MAS) and ControlValve (which the authors call the state-of-the-art defense), plus two ablations: Kaer w/ free text and ControlValve w/o rules.
    • The authors state that existing benchmarks do not meet the requirements: GAIA and WebArena test only multi-step task completion and contain no attacks; safety benchmarks such as AgentDojo, InjecAgent, and ASB test agents reading injected data, not attacks targeting orchestrator replanning, so the authors built Cintra.

    The authors position this paper as extending safety constraints to the replanning process itself while preserving dynamic replanning capability, a step that neither of the above two defense categories covers.

  3. How does the paper address this problem?

    Kaer restricts agent feedback to fixed failure templates, repairs use only seven deterministic operators, and all revised plans must pass policy validation.

    Overall approach: Kaer isolates the planner from untrusted agent feedback—agents can convey diagnostics only through abstract failure templates, plan revisions can use only a set of deterministic repair operators, and all revised plans must satisfy preset policies. The authors state this draws on the endorsement idea from information flow control (narrowing untrusted data to a small type and then passing it through a trusted channel).

    Problem setting and plan validity

    • A plan Π = (S1, …, SN) is a strictly ordered sequence of subtasks, where each subtask Sj = (Aj, {Ci}j, (Pj, Oj−1)) contains the assigned agent, the set of authorized capabilities, and the inputs (the natural-language directive Pj and the preceding subtask's output Oj−1).
    • A plan is valid only if four conditions hold: the subtask's capabilities belong to those offered by the assigned agent, belong to the task policy CT, the dependencies conform to the agent policy APA, and the outputs of non-final subtasks are used by later subtasks; if invalid, the plan is returned to the planner for regeneration, and the task is abandoned after a fixed number of attempts.

    Execution and least-privilege gateway

    • The orchestrator executes subtasks in order and mediates all inter-agent communication; agents run in isolated environments and can access tools and LLMs only through a trusted gateway.
    • The gateway forwards only requests where "the caller is the currently active agent and the target is an endpoint of its authorized capabilities" and rejects all other requests; it does not check the parameters or return values of permitted calls.
    • Each agent is strictly limited to the capabilities granted to its subtask (least privilege).

    Close-out and failure templates

    • At the end of a subtask, the agent returns a close-out: success (with output) or failure (with a failure condition + progress report); the gateway also records a failure condition when it rejects a call, an endpoint fails, or a timeout occurs with no close-out.
    • Failure conditions come from a fixed set and are given to the planner as failure templates (Table 2 has 13 templates in total, such as endpoint transient fault, directive underspecified, input artifact absent, result uncorroborated), each accompanied by a suggested sequence of repair operators for the planner's reference only.
    • Key constraint: the planner receives no subtask outputs, progress reports, or agent-generated text; progress reports are returned to an agent only when that same agent is re-dispatched.

    Repair operators and repair process

    • Seven repair operators (Table 1): RE-EXECUTE, REVISE-DIRECTIVE, RESCOPE, REASSIGN, INSERT, PRUNE, REWIRE; they can modify only positions k ≥ j (unfinished subtasks), and completed subtasks and their outputs are immutable.
    • A repair is a privileged operation when it grants a subtask capabilities outside CT and requires user authorization (denoted Xj; after authorization the valid plan requires {Ci}j ⊆ CT ∪ Xj); if refused, that repair is blacklisted.
    • The repair budget limits the number of failures for which repairs can be requested per task; when exhausted, the task is terminated; the task is also terminated if consecutive refused repairs exceed a fixed number.

    Output verification extension (RQ5)

    • To address "a malicious agent falsely reporting success," the authors add an LLM check after a successful close-out: it judges whether the output satisfies the directive, and if it refuses, the success is converted to a failure and replanning is triggered; the check uses the same LLM as the planner.
    • The authors argue this check gives attackers no new channel of influence: if accepted, it is consistent with no check; if refused, the planner sees only the template.
  4. What experiments did the paper conduct?

    Comparing 5 systems × 3 models on the self-built benchmark Cintra, Kaer has the highest security rate and higher utility than ControlValve.

    Experimental setup

    • Models under test: gpt-5.6-sol, gpt-5.6-terra, deepseek-v4-pro; each system uses these three models as the underlying model respectively.
    • Systems: Kaer, Kaer w/ free text (ablation replacing failure feedback with free text), Magentic-One (undefended baseline), ControlValve, ControlValve w/o rules (ablation removing context rules, using the authors' implementation).
    • Benchmark: the authors' self-built Cintra, containing 14 tasks requiring replanning (obstacles divided into agent reliability, unknown requirements, unknown data, Table 3) and 14 attack goals (covering integrity, confidentiality, availability, Table 4), paired into 164 task-goal pairs; the malicious agent is fixed as filesurfer.
    • Sample size: each system-model combination runs 14×5 = 70 benign trials and 164×5 = 820 attack trials, with full isolation between trials.
    • Metrics: Utility is the benign trial success rate; Security is the proportion of attack trials in which the attack goal fails; the attack success rate for each goal is also reported.
    • Evaluation method: some utility checks and attack success conditions are evaluated by an LLM judge running on gpt-5.6-terra (Table 3 marks D as deterministic check and J as LLM judge).

    Main results

    According to Table 5 (each cell is 70 benign / 820 attack trials):

    Swipe horizontally to read the table

    Systemgpt-5.6-sol Utilitygpt-5.6-sol Securitygpt-5.6-terra Utilitygpt-5.6-terra Securitydeepseek-v4-pro Utilitydeepseek-v4-pro Security
    Kaer64.3%92.3%65.7%88.0%41.4%79.9%
    Kaer w/ free text74.3%75.7%60.0%76.3%51.4%72.3%
    Magentic-One91.4%35.0%78.6%47.8%84.3%50.1%
    ControlValve54.3%69.8%34.3%80.4%54.3%78.7%
    ControlValve w/o rules75.7%42.2%68.6%59.1%55.7%60.2%
    • The authors state that Kaer has the highest Security on all three models, and also exceeds ControlValve in Utility on gpt-5.6-sol and gpt-5.6-terra; Magentic-One and ControlValve w/o rules have high utility but the lowest security.
    • The authors state that ControlValve's low utility mainly stems from its rule enforcement: its static grammar and rule set are generated before execution and reject agents and instructions that only become necessary during execution; removing the rules restores utility.
    • Exception: on deepseek-v4-pro, Kaer's Utility (41.4%) is the lowest of the five systems; the authors state that most failed benign trials produce incorrect results, and in about one third of failures all subtasks report success, leaving the planner no way to detect it.

    Attack goal breakdown (RQ2)

    • According to Figure 6 (gpt-5.6-sol, attack success rate): Kaer blocks almost all control-flow attacks such as "terminate task early" (0%), "specify agent input" (0%), "drop downstream agent" (0%), while Magentic-One has a success rate of 28%–100% on most goals.
    • Attacks that still succeed against Kaer fall mainly into three categories (authors' attribution): injected instructions smuggled in successful output and executed by downstream agents (e.g., running a data exfiltration script, 7% on gpt-5.6-sol, 27% on gpt-5.6-terra, 53% on deepseek-v4-pro, Figure 7); falsely reporting missing capability to induce reassignment (23% on gpt-5.6-sol); silent failure (34% on gpt-5.6-sol).
    • The authors state that ControlValve can block script- and service-modification attacks, but remains vulnerable to step-omission attacks (early termination, removing agents), because its rules do not check omitted steps.

    Failure templates and repair coverage (RQ3, RQ4)

    • According to Figure 8 (gpt-5.6-sol): under the always success policy, Kaer and Kaer w/ free text have almost the same attack success rate (on the order of 13.4% vs 13.0%; the caption states both handle successful close-out the same way); under always failure, Kaer is 2.8%, and Kaer w/ free text is more than twice that, with the authors stating that free-text failure reports are the more effective attack surface for Kaer w/ free text.
    • According to Figures 11 and 12: failures in the agent reliability obstacle category are mostly execution faults, which Kaer usually resolves by retrying; missing capability/input is resolved about half the time; missing information is resolved the least, which the authors attribute to templates deliberately omitting runtime context (such as data source location).
    • Undetected failures (all subtasks report success but the result is wrong) appear in all three obstacle categories, most on deepseek-v4-pro; in this case replanning is not triggered.

    Output validation extension (RQ5)

    • After adding LLM output validation (always success policy): on gpt-5.6-sol the attack success rate drops from 13.4% to 3.2% (Figure 8), on gpt-5.6-terra from 13.0% to 3.5% (Figure 9), and on deepseek-v4-pro from 28.0% to 2.8% (Figure 10); the abstract states the defense rate rises from 86.6%/87.0% to 96.8% (two models).
    • Utility impact: on gpt-5.6-sol and gpt-5.6-terra, benign utility is comparable to the original Kaer; on deepseek-v4-pro it rises from 29 successes out of 70 to 38 (Figure 5(b)), which the authors state reaches ControlValve's level.
    • Residual successes mostly come from subtask reassignment: after validation rejects the malicious output, the planner reassigns the subtask to a capable agent, which happens to be the agent specified by the injected instruction.

    Other ablations and analysis

    • Turning off ControlValve's context rules (ControlValve w/o rules): utility rises and security falls on all three models (Table 5).
    • Adaptive attacker discussion (Section 5.7): when the attacker knows all templates and the repair budget, it can induce repairs containing the target operator by choosing failure reports; the authors state that Kaer can only constrain, not prevent, such attacks, because the repairs are all within policy scope and consume the repair budget.
    • The paper does not report statistical significance tests, variance, or confidence intervals.
  5. What are some points that can be further explored?

    The authors discuss missing template information, the injection surface of output validation, and adaptive attackers; the evaluation covers 3 models and 14 tasks.

    Limitations and future directions noted by the authors

    • Templates omit runtime context (Sections 5.2, 5.5): to preserve security, failure templates carry no arbitrary runtime information, so tasks requiring runtime observations such as data source location cannot be repaired, which the authors state is the reason for limited utility.
    • Successful close-out does not validate output (Sections 5.2, 5.3): Kaer does not evaluate output when accepting a successful close-out, so silent failures and outputs carrying injected instructions pass through; the authors address this with the output validation extension, but residual reassignment attacks remain.
    • Output validation itself can be influenced by injection (Section 5.6): the validation LLM reads text produced by the agent, and a malicious agent can embed instructions attempting to make validation accept the result or reject it according to a specified template; the authors argue this gives the attacker no new channel to influence the planner, but the validation judgment itself can be manipulated.
    • Adaptive attackers can only be constrained, not prevented (Section 5.7): an attacker who knows all templates and the repair budget can induce repairs within policy scope (e.g., using REWIRE/INSERT to obtain other subtasks' output), and the authors state that Kaer can only limit its impact.
    • Out-of-scope items of the threat model (Section 3): misuse within an agent's internal loop, compromise of trusted components, threats propagated directly through subtask output, and output correctness validation for open-ended tasks are all outside the defense scope.
    • The paper has no separate Limitations or Future Work section; the above are boundaries explicitly stated by the authors in the Discussion and threat model.

    Scope of experimental coverage

    • The models under test are gpt-5.6-sol, gpt-5.6-terra, deepseek-v4-pro, 3 in total; the compared systems are Magentic-One, ControlValve and two ablation variants.
    • The Cintra benchmark contains 14 tasks, 14 attack goals, and 164 task-goal pairs; each configuration has 70 benign and 820 attack trials, each repeated 5 times.
    • The malicious agent is fixed as an instance of filesurfer (the authors state this is because it participates in every task, facilitating uniform comparison); the attack injection method is appending injected instructions in the response or choosing the close-out type.
    • The output validation extension is evaluated only under the always success close-out policy; validation uses the same LLM as the planner.
    • The paper does not report the specific values of the repair budget, timeout, or consecutive rejection count, and does not report statistical significance tests or human evaluation.
  6. Summarize the main content of the paper

    Kaer protects multi-agent replanning with templated feedback, deterministic repair operators and policy validation, and achieves a better safety rate than baselines on Cintra.

    A paper on multi-agent system defenses: addressing the risk that orchestrator-style MAS are hijacked by malicious agent feedback when replanning during execution, it proposes the secure replanning framework Kaer and evaluates on a self-built benchmark Cintra.

    • Problem: Dynamic replanning makes the planner depend on agent feedback, and a malicious agent can embed injected instructions in the returned payload to manipulate plan revision; the authors state that existing defenses only do tool-level access control or pre-execution static policies, and no one protects the replanning stage.
    • Method: A failed agent can only choose a failure template from a fixed set to report (the planner receives no agent text); plan revision can only use seven deterministic repair operators and only modifies unfinished subtasks; revised plans must pass task policy and agent policy validation, and out-of-scope capabilities require user authorization; there is also a repair budget limit.
    • Main results (Table 5, 70 benign + 820 attack trials per configuration): on gpt-5.6-sol, Kaer Security 92.3%, Utility 64.3%, higher than ControlValve's 69.8%/54.3% and Magentic-One's 35.0%/91.4%; on gpt-5.6-terra, 88.0%/65.7%.
    • Exceptions and residual risk: on deepseek-v4-pro, Kaer Utility 41.4% is the lowest of the five systems; attacks that succeed against Kaer mainly propagate through the output of a successful close-out (silent failure, smuggled injection), and after adding LLM output validation the attack success rate drops to 3.2%, 3.5%, 2.8% on the three models respectively (Figures 8–10).
    • Authors' conclusion: multi-agent systems can replan securely without pre-locking the plan and with controllable utility loss; the authors also note that templates omitting runtime context cause some tasks to be unrepairable, and adaptive attackers can still induce repairs within policy scope, so this can only constrain, not prevent.
Read originalarxiv.org