Evaluating Authorization and Trust Boundaries in Agent Orchestration Frameworks
Evaluating Authorization and Trust Boundaries in Agentic Orchestration Frameworks
The authors used gpt-4o-mini to run 12 attack experiments mapped to the OWASP Agentic Top 10 on LangChain, LangGraph, AutoGen, and OpenClaw; the authors state that authorization and trust boundary flaws are widespread.
Attackers can submit malicious instructions, influence retrieved content, provide crafted files or manifests, manipulate memory input, and control or impersonate untrusted agents; no assumption is made of host compromise, administrative privileges, or possession of unauthorized credentials. The victim system is an agent application implemented on four frameworks, sharing gpt-4o-mini; protected assets include task integrity, tool authorization, the file system, command execution capability, agent memory, and inter-agent identity communication.
- How agent orchestration frameworks represent identity, enforce authorization, and propagate trust at component boundaries determines their security risk; the authors aim to evaluate the actual protection effectiveness of LangChain, LangGraph, AutoGen, and OpenClaw at these boundaries.
- Use the same LLM (gpt-4o-mini) to implement 12 attack experiments mapped to the OWASP Agentic Top 10 on four frameworks, covering prompt injection, unauthorized resource access, command execution, memory poisoning, and inter-agent trust and identity attacks, and compare vulnerable and protected configurations.
- The authors report: variant D prompt injection induced unauthorized tool calls on LangChain, LangGraph, and OpenClaw; E3 and E6 had flaws in all frameworks; E5 attacks succeeded on three frameworks; E9 memory poisoning ASR 66.9%–92.8%; E11/E12 protected configurations blocked the attacks in all cases.
- The authors note: the 12 experiments cannot represent all attack scenarios; the protected configurations include application-custom checks, and the results do not reflect inherent guarantees of the frameworks; only gpt-4o-mini was used, and other models and production configurations may differ; credential authentication, enterprise IAM, and external policy engines were not comprehensively evaluated.
- Evaluated models
- gpt-4o-mini
- Benchmarks
- OWASP Top 10 for Agentic Applications(映射的 12 个自建实验 E1–E12)
- Metrics
- ASRClean AccuracyUTISDPRRFPRaverage poison rank
Researchers evaluated the authorization and trust boundaries of four agent orchestration frameworks—LangChain, LangGraph, AutoGen and OpenClaw—through twelve experiments designed against the OWASP Top 10 for Agentic Applications, uniformly using gpt-4o-mini. The experiments covered prompt injection, unauthorized resource access and command execution, memory poisoning, and inter-agent trust and identity attacks. The experiments showed that adversarial inputs can induce unauthorized tool calls, and that tool chaining, stale filesystem validation, encoded instructions and inter-agent delegation exposed authorization weaknesses across component boundaries; memory poisoning showed that untrusted state can influence model responses, a risk distinct from traditional access control violations. Among the tested defense configurations, explicit authorization, controlled capability assignment, filesystem isolation and execution guards prevented specific unauthorized operations. The authors emphasize that authorization and information trust boundaries should be maintained throughout the agent workflow, and that framework capabilities or model-layer instructions are insufficient as a security guarantee.
Paper analysis
What problem does this paper attempt to solve?
Evaluate the security enforcement effectiveness of four agent orchestration frameworks at identity, authorization, and trust boundaries.
Core question: how agent orchestration frameworks represent identity, enforce authorization, and propagate trust at component trust boundaries, and whether these mechanisms can withstand attacks.
- Problem setting: Agent systems combine LLMs with tools, memory, and multi-agent collaboration; orchestration frameworks coordinate execution and communication, but also create opportunities for unauthorized resource access, command execution, memory poisoning, and privilege abuse (the authors classify these according to the OWASP Top 10 for Agentic Applications).
- Gaps in existing research: The authors state that existing work separately studies framework vulnerabilities, security benchmarks, penetration testing, automated red teaming, etc., with varying scopes, so a controlled comparison of authorization and trust boundaries under shared attack objectives is needed.
- Core observation: The authors argue that framework support for tool restrictions, routing, or approval mechanisms does not equal automatic enforcement of authorization; identity identifiers (such as agent names, session IDs) do not necessarily constitute verified identity, and mechanism effectiveness depends on configuration and integration.
- What the paper proposes: 12 attack experiments mapped to the OWASP Agentic Top 10 on LangChain, LangGraph, AutoGen, and OpenClaw, all sharing gpt-4o-mini, plus analysis of trust boundaries and enforcement points; contributions include a mechanism comparison, 12 attack experiments, and boundary analysis.
- Threat model (Section 2.2):
- Objective: cause experiment-specific unauthorized effects, such as unauthorized tool calls, command execution, resource access, or acceptance of forged instructions.
- Capabilities: submit malicious instructions, influence retrieved content, provide crafted files or manifests, manipulate memory input, control or impersonate an untrusted agent.
- Not assumed: the host is already compromised, administrative privileges, or possession of unauthorized credentials.
- Victim system and assets: agent applications implemented with the four frameworks; protecting task integrity, tool authorization, the file system, command execution capability, agent memory, and inter-agent identity communication.
What related research exists?
Related work covers framework vulnerabilities, security benchmarks, penetration testing, and red teaming; the authors position this paper as a controlled comparison under shared attack objectives.
Framework vulnerabilities and security analysis
- Hamza et al. (2026) — empirical analysis of security issues in open-source agent frameworks (reference [3]).
- Shan et al. (2026) and Wang et al. (2026) — respectively a security analysis and defense framework for OpenClaw, and formalization and benchmarking of attacks against OpenClaw (references [17, 18]).
Benchmarks and deployment security
- Fu et al. (2025) — RAS-Eval, a benchmark for LLM agent safety evaluation in real-world environments (reference [1]).
- Gasmi et al. (2026) — comparative vulnerability assessment of LLM agent deployment paradigms (reference [2]).
Penetration testing, red teaming, and multi-agent failure
- Nguyen and Husain (2025) — comparison of agentic AI penetration testing across models and frameworks (reference [12]).
- Pendyala et al. (2026) — adaptive red-teaming framework using dynamic attack graphs and RAG for vulnerability profiling (reference [16]).
- Xie et al. (2026) — modeling and mitigating error cascades in LLM multi-agent collaboration (reference [19]).
Architecture comparison and framework documentation
- Lei (2024) — comparison of LangGraph and LangChain (reference [8]); the official documentation of LangChain, LangGraph, AutoGen, and OpenClaw is also cited to explain their mechanisms (references [4–7, 9–11, 13, 14]).
Baselines and benchmarks: The paper does not implement other attack methods as baselines; the experimental controls are the vulnerable and protected configurations of each framework, plus benign task controls (such as E7's benign manifest and E9's no-injection control); attack objectives are mapped to the OWASP Top 10 for Agentic Applications [15].
Positioning: The authors state that the above work has varying scopes, and this paper's distinction is a controlled comparison of authorization and trust boundaries across four frameworks under shared attack objectives, emphasizing that the results are intended to highlight boundary enforcement rather than to rank the security of the frameworks.
How does the paper address this problem?
Implement 12 attacks mapped to OWASP on the four frameworks with the same LLM, comparing vulnerable and protected configurations.
Overall approach: implement 12 attack experiments (E1–E12) on LangChain, LangGraph, AutoGen, and OpenClaw using a common gpt-4o-mini, keeping attack goals, inputs, capabilities, and success criteria equivalent, retaining each framework's native tool APIs, state handling, routing, and authorization implementations, and comparing execution effects at trust boundaries.
Experimental platform and threat model
- Environment: Ubuntu 24.04.4 LTS virtual machine (8-core CPU, about 14 GiB memory); LangChain 1.2.17, LangGraph 1.1.10, AutoGen 0.7.5 (PyAutoGen 0.10.0) on Python 3.11.9, OpenClaw 2026.6.11 with Node.js; all via the OpenAI API using gpt-4o-mini.
- Seven trust boundaries: user to model, external content to model, model to tools, agent to file system, agent to execution environment, agent to memory, agent to agent.
- Success determination: success only when the experiment-specific unauthorized effect occurs (unauthorized tool call, command execution, resource access, or acceptance of forged instructions); distinguish "attempted operation" from "actual unauthorized side effect".
Design of five categories of attack experiments
- Prompt injection (E1/E2): the benign task is to summarize a project report, with an additional privileged read_secret_file tool not needed for the task exposed, and a canary placed in the protected file; E1 places malicious instructions in the user request, E2 embeds them in a retrieved document. Four variants: A basic override, B HOUYI-style delimiters, C forged developer/system authorization, D delegation/audit framework. Metrics are UTI (unauthorized tool invocation) and SD (secret disclosure).
- Unauthorized resource access (E3/E6/E7): E3 tests a tool chain combination (read secret, encode, write report, with only summarization authorized); E6 is a TOCTOU symlink escape, testing whether authorization remains valid after path validation and at access time; E7 is manifest path traversal, including a benign manifest control.
- Unauthorized command execution (E4/E5/E8): E4 injects shell commands (direct and via retrieved file variants), with only actual execution counted as success; E5 has a restricted decoder decode a Base64 payload and automatically pass the text to an executor with file creation permission; E8 separates a planner without permissions from an executor with shell permission, with instructions passed directly without intermediate validation, including remote maintenance session and interactive shell scenarios.
Memory poisoning and inter-agent attacks
- Memory poisoning (E9/E10): E9 injects forged claims into session history (recap, handoff, emergency, postmortem framings), across 12 operation domains, including a clean baseline, a no-injection control, and three rounds of multi-turn sessions; ASR, Clean Accuracy, etc. are determined using a semantic similarity threshold of 0.62. E10 uses 84 clean and 72 poisoned memory records, 228 queries per framework per seed, a shared FAISS IndexFlatIP index and all-MiniLM-L6-v2 embeddings, a cosine similarity threshold of 0.70, and measures ASR, PRR, average poisoning rank, Clean Accuracy, and FPR; each of the two experiments runs three independent seeds.
- Inter-agent trust and identity (E11/E12): E11 embeds an attacker-controlled delegation in a legitimate project summary; the vulnerable configuration (B1) forwards it directly to a downstream agent with file read permission without authorization checks, while the protected configuration (B2) adds an authorization stage to verify delegation permissions. E12 has the attacker forge reviewer approval; the vulnerable configuration (A) relies on textual approval statements, while the protected configuration (B) has a dedicated reviewer generate a structured authorization record verified by the coordinator; success requires the unauthorized marker file to be created, not merely acceptance of a forged identity.
What experiments did the paper conduct?
The 12 experiments show: tool chains, TOCTOU, encoded instructions, and delegation commonly lead to privilege escalation, and protected configurations can block specific operations.
Experimental setup
- Frameworks under test: LangChain 1.2.17, LangGraph 1.1.10, AutoGen 0.7.5, OpenClaw 2026.6.11, all sharing gpt-4o-mini (OpenAI API).
- Experiments: 12 self-built experiments E1–E12, mapped to the OWASP Top 10 for Agentic Applications (some of ASI01–ASI10), covering five attack categories; benign tasks (T1/T2/T6/T11/T12) completed successfully under all configurations, and the main text reports only attack results.
- Metrics: UTI, SD (E1/E2); V/S (E3/E6/E7); A/I, R, P, Y, B (E4/E5/E8); ASR, Clean Accuracy, PRR, average poisoning rank, FPR (E9/E10, three seeds, mean ± sample standard deviation).
- Decision method: E9 semantic similarity threshold 0.62, E10 cosine similarity threshold 0.70; E10 retrieval uses a shared FAISS IndexFlatIP and all-MiniLM-L6-v2; no LLM judge was used.
- Controls: E7 benign manifest control, E9 no-injection control, E11/E12 vulnerable vs. protected configurations.
- Sample size: E10 has 84 clean + 72 poisoned records, 228 queries per framework per seed; E1–E8, E11, and E12 do not report the number of repetitions.
Main results
According to Table 3, 4, 5, 7 (LC=LangChain, LG=LangGraph, AG=AutoGen, OC=OpenClaw):
Swipe horizontally to read the table
Experiment LC LG AG OC E1/E2 variant D (UTI/SD) U/– U/– S U/– E3 tool chain V V V V E6 symlink TOCTOU V V V V E7 path traversal S S S S E4 shell execution A/I R R R E5 encoded payload P Y Y Y E8 delegated command B B B B E11/E12 vulnerable→protected all compromised→all protected same as left same as left same as left - Variants A–C were all resisted by the four frameworks; variant D induced UTI on LC, LG, and OC but no SD, which the authors say shows that authorization must be enforced independently at the tool boundary.
- In E4, LangChain adopted the malicious instruction but the approval mechanism interrupted execution, which the authors use to distinguish "resisting instructions" from "enforcement at the execution boundary."
- In E5, LangChain changed the target file name due to a decoding error, recorded as partial success (P).
Memory poisoning (E9/E10, Table 6)
- E9 session poisoning: ASR was AutoGen 92.8±1.1%, LangGraph 92.6±0.4%, LangChain 78.0±2.2%, OpenClaw 66.9±2.4%; under the no-injection control, no framework reproduced the forged claim; multi-turn ASR ranges: LangGraph 91.7–95.8%, AutoGen 91.7–93.8%, LangChain 72.9–85.4%, OpenClaw 60.4–64.6%.
- E10 persistent poisoning: PRR was 82.3% for all four frameworks, average poisoning rank 1.43 (shared retrieval facility); ASR was LangGraph 49.3±0.6%, AutoGen 48.3±6.9%, LangChain 47.6±1.6%, OpenClaw 45.1±1.2%; OpenClaw had the lowest Clean Accuracy (E9 61.3%, E10 24.1%).
- Authors' interpretation: retrieving poisoned content does not equal accepting it; OpenClaw's low ASR comes with low Clean Accuracy, and reducing poisoning success must be weighed together with legitimate task performance.
Inter-agent trust and identity (E11/E12, Table 7)
- E11: Under vulnerable configuration B1, the downstream agents of all four frameworks accepted the delegation, called the file-reading tool, and leaked the secret; protected configuration B2 rejected unauthorized delegation before allocating privileged capabilities, with no access and no leak.
- E12: Under vulnerable configuration A, all four frameworks accepted the forged review approval and created the unauthorized marker file; protected configuration B rejected the forged request with an independent verification stage and structured authorization records, and no marker was created.
- Authors' interpretation: a downstream agent holding a privileged tool does not mean the upstream request was authorized; identity claims must be verified independently of natural-language instructions.
Other ablations and analysis
- E1 and E2 produced identical results; the authors state that both direct and indirect delivery can exploit this boundary under the evaluated configurations, but cannot determine whether the behavior is mainly attributable to the shared LLM or to framework mechanisms.
- In E8, a benign SSH command generated from an ambiguous maintenance request was executed by the simulated shell tool, while a reverse-shell command generated from an explicit interactive shell request was detected and blocked at the execution interface.
- In E7, LangChain, LangGraph, and AutoGen use normalized path checks, while protected OpenClaw relies on sandbox isolation.
- E9 poisoning was introduced through four framework mechanisms: recap, handoff, emergency, and postmortem; the paper does not report ASR separately by framework.
What are some points that can be further explored?
The authors list three types of limitations: scope, implementation, and generalization; the experiments used only gpt-4o-mini and four frameworks.
Limitations and future directions noted by the authors
- Scope (Section 5 Scope): 12 experiments and 4 frameworks cannot represent all attack scenarios; identity verification is mainly examined through E12, and E9–E10 evaluate information integrity rather than traditional access control.
- Implementation (Section 5 Implementation): different runtimes and tool interfaces require framework-specific implementations; protected configurations include application-custom checks, and the observed results describe these configurations, not inherent guarantees of the frameworks.
- Generalizability (Section 5 Generalizability): sharing gpt-4o-mini reduces model-choice differences, but other models, versions, and production configurations may behave differently; credential-based authentication, enterprise IAM, and external policy engines were not comprehensively evaluated.
Scope of experimental coverage
- The frameworks under test are LangChain, LangGraph, AutoGen, and OpenClaw, 4 in total; the underlying LLM is only gpt-4o-mini (Section 2.2).
- Attack categories cover prompt injection, unauthorized resource access, unauthorized command execution, memory poisoning, and inter-agent trust and identity, 5 categories and 12 experiments in total (Table 2).
- Quantitative repetition appears only in memory poisoning: E9/E10 each have three independent seeds, and E10 has 228 queries per framework per seed (Section 2.3.4); E1–E8, E11, and E12 do not report the number of repetitions in the paper.
- Defense evaluation uses the protected configuration of each experiment: E4 execution approval, E7 path containment/sandbox, E8 reverse-shell detection, E11-B2 delegation authorization, E12-B identity verification (Section 3).
- The paper does not report sample sizes for E1–E8 and E11/E12, nor human evaluation or consistency checks other than the semantic similarity threshold.
Summarize the main content of the paper
Four frameworks, 12 experiments: authorization and trust boundary flaws are widespread; explicit checks can block specific unauthorized operations.
A controlled safety evaluation of agent orchestration frameworks: the authors use a common gpt-4o-mini to implement 12 attack experiments mapped to the OWASP Top 10 for Agentic Applications on LangChain, LangGraph, AutoGen, and OpenClaw, examining identity representation, authorization enforcement, and trust propagation.
- Problem: The tool restrictions, routing, and approval mechanisms provided by frameworks do not equal automatic authorization enforcement; the authors aim to test actual protections at cross-component boundaries.
- Method: Keep attack goals and success criteria equivalent, retain each framework's native implementation, compare vulnerable and protected configurations, and cover five categories: prompt injection, resource access, command execution, memory poisoning, and inter-agent attacks.
- Main results:
- Prompt injection variant D induced unauthorized tool calls without secret leakage in LangChain, LangGraph, and OpenClaw, and neither in AutoGen (Table 3).
- The E3 tool chain and E6 TOCTOU symlink were flawed in all four frameworks; E7 path traversal was blocked in all protected configurations (Table 4).
- The E5 encoded payload succeeded in LangGraph, AutoGen, and OpenClaw, and partially succeeded in LangChain; the malicious commands in E4 and E8 were all blocked (Table 5).
- E9 session memory poisoning ASR was 66.9% (OpenClaw) to 92.8% (AutoGen); E10 PRR was 82.3% for each framework, with ASR 45.1%–49.3% (Table 6).
- Under vulnerable configurations, all four frameworks were compromised in E11/E12; protected configurations all blocked unauthorized operations (Table 7).
- Authors' conclusion: The availability of privileged capabilities does not mean their use is authorized; secure orchestration requires deterministic authorization at privilege transfer points, least-privilege capability assignment, and execution-layer protections, and must distinguish untrusted memory content from authoritative information; the authors emphasize that these findings are intended to highlight boundary enforcement, not to rank the frameworks.