guardscope: measuring which actions runtime controls inspect in LLM agent systems
guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems
The authors propose guardscope, which uses static analysis plus kernel-level ground-truth checking to measure which agent action paths runtime controls actually inspect; the measurements confirm that 5 of 15 paths bypass all sub-kernel designs.
- Enterprise-deployed runtime controls (such as agent gateways) can only inspect actions that pass through their deployment point, while agents can take effect through multiple paths; the authors state that previously no work had measured this deployment-location-determined coverage against kernel-level ground truth.
- The authors enumerate 15 action paths and 8 topology designs and derive the coverage result for each pair of combinations; guardscope has two stages: static analysis (predicting inspectable paths from control source code) and empirical measurement (running a fixed workload and checking control decisions against Tracee eBPF kernel records).
- Using two harnesses, Pi and OpenClaw, and real controls (Microsoft agent governance toolkit, agentgateway, ActPlane), eight designs were tested empirically; the authors report that the results confirm the derived conclusions for the tested action classes: 5 paths bypass all sub-kernel designs, and the T7 empirical coverage (6 allowed, 1 denied, 8 bypassed) is below the theoretical upper bound for the kernel position.
- The authors note three limitations: the reference workload implements only 4 of the 15 paths, with the remaining paths relying solely on design analysis; only coverage was measured, not security effectiveness; and kernel-level ground truth itself cannot identify which tool an action belongs to, carries no payload content, and mixes in unrelated processes.
- Evaluated models
- agent-os-kernel (agent governance toolkit)agentgatewayPi in-process policy engineActPlane
- Benchmarks
- guardscope reference workload(15 个动作实例,实现 P2、P4、P6、P7 四条路径)
- Metrics
- checkedbypassedpreventedindeterminate
Researchers from Vulcan Research, Keio University and Standard Chartered Bank propose the open-source framework guardscope, which measures which actions runtime controls actually inspect in LLM agent systems. The framework enumerates 15 action paths and 8 deployment topologies for agents; the static stage predicts inspectable paths from control source code, and the empirical stage reconciles control decision records under a fixed workload against Tracee eBPF kernel-level observations, classifying each action as inspected, bypassed or blocked. The evaluation uses two harnesses, Pi and OpenClaw, and real controls such as the Microsoft agent governance toolkit, agentgateway and ActPlane to validate eight designs; the paper finds that 5 of the 15 paths bypass all designs below the kernel, and that deployed solutions mostly cluster in designs with narrower coverage. The authors note that the workload implements only 4 of these paths, that the measurement is of coverage rather than security effectiveness, and that the code is open-sourced on GitHub.
Why it was selectedThe paper presents a method for verifying runtime control coverage with kernel-level observations, which teams deploying agent controls can use to check their own gaps across the 15 action paths.
Related events and news
0 events · 1 news mentions
News that mentions this paper
- 公众号 大模型安全研究cites the paper's id or link大模型安全周报:别只关注通过率,聚焦边界控制和安全覆盖面
3. 把运行时控制的覆盖度做成可测量、可对账的工程指标:guardscope: Measuring Which Actions Runtime Controls Check in LLM Agent Systems(https://arxiv.org/abs/2611.15223)
Paper analysis
What problem does this paper attempt to solve?
Runtime controls can only inspect actions that pass through their deployment point; the authors state that previously no work had measured this coverage against kernel-level ground truth.
Core question: which action paths of an LLM agent can runtime control actually inspect, and this coverage had previously not been measurable against kernel-level ground truth.
- Problem setting: Enterprises deploy LLM agents to handle internal data and services, and use runtime controls such as agent gateways to inspect their behavior in real time; but a control can only evaluate actions that pass through its deployment point, while an agent can take effect through multiple paths beyond tool calls (e.g., the harness directly initiating network connections, or launching a tool server as a local subprocess).
- Gaps in existing work: The authors state that existing work separately does behavior monitoring, gateway policy checking, guardrails, access control, and evaluation, but no work measures the coverage of deployed controls determined by deployment position against kernel-level ground truth.
- Core observation: A control's deployment position simultaneously determines which paths it can inspect and what information it can inspect; kernel-level monitoring (eBPF) can see almost all actions in the agent process tree, but its records do not contain payload content and are mixed with unrelated processes.
- What the paper proposes: Enumerate 15 action paths and 8 topology designs (T0–T7) and derive the coverage result for each pair of combinations; propose the guardscope framework, whose static stage predicts inspectable paths from control source code, and whose empirical stage runs a fixed workload and, against kernel-level ground truth, classifies each action instance as checked, bypassed, or prevented.
- Scope assumptions: The paper assumes general-purpose agent harnesses (such as Claude Code, Gemini CLI, Pi, OpenClaw, etc.) and general-purpose runtime controls; it assumes a single-agent setting, and the authors state the analysis can be extended to multi-agent (Section 4.1).
What related research exists?
The authors divide related work into five categories: monitoring, gateways and policy checking, guardrails, access control, and evaluation and benchmarks, and point out that no work measures the coverage determined by deployment position.
Behavior monitoring
- AgentOps, ClawSentry, Watson—respectively define agent observability data, monitor through a gateway, and trace the agent's reasoning process.
- AgentSight—uses eBPF to intercept LLM traffic and kernel events and correlates the two data streams across process boundaries; Tetragon and Falco provide kernel-level process activity records for general workloads.
Agent gateways and policy checking
- Envoy AI Gateway, Apache APISIX, etc. provide extensible filter chains; agentgateway, LiteLLM Proxy, Bifrost, etc. apply the same structure to LLM traffic; Microsoft Agent Governance Toolkit (AGT) adds governance modules at the gateway.
- Cloud control planes (Microsoft Foundry, Agent 365) enforce equivalent policies within the provider boundary; policies are written in policy-as-code languages such as Rego, Cedar, and CEL.
Guardrails
- NeMo Guardrails provides input/output moderation and enforcement checks; LlamaFirewall combines jailbreak detection, goal alignment checks, and code scanning; Llama Guard and Presidio perform model-based classification and sensitive-content detection, respectively.
- GuardAgent maps protection requests to guardrail code; ShieldAgent extracts verifiable safety rules from policy documents and evaluates action sequences.
Access control
- Classical access control: Ganie integrates RBAC into agent deployment, and AgentGuard performs ABAC checks on each tool call.
- Capability systems: CaMeL extracts capabilities from trusted user queries and enforces control-flow/data-flow policies; RTBAS and APPA perform information-flow control; SHarD, SkillGuard, SEAgent, etc. place the enforcement point in harness configuration or multi-agent settings.
- Policy-language systems: Progent enforces least privilege with symbolic rules; AgentFlow uses an SMT solver to decide policies; FORGE uses Datalog; AgentSpec evaluates rules at action execution; ActPlane enforces policies in the kernel with eBPF and reports that tool-call interception misses system actions on indirect execution paths.
Evaluation and benchmarks
- AgentDojo measures prompt injection attacks and defenses on shared tasks; the authors state that the adaptive evaluation study [39] is the only published adaptive test of a policy-based monitor, testing Progent on AgentDojo with a single attack template, and the defense held.
- There are also individual attack reports such as causality laundering, visual confused deputy, and cross-app context poisoning, as well as empirical measurement of policy-as-code artifacts and a catalog of prompt injection defense design patterns [16].
Positioning of this paper: The authors state that related work shows runtime controls occupy various enforcement positions from gateway to kernel, but does not explain what actions of the agent each position can actually inspect; this paper turns this relationship into repeatable measurement.
How does the paper address this problem?
First enumerate 15 paths and 8 topology designs and derive the coverage matrix, then use guardscope's static prediction plus kernel ground-truth checking to turn the analysis into repeatable measurement.
Overall approach: first derive "which paths are inspected by which deployment design" into a matrix, then use the guardscope framework to turn the derivation into verifiable measurement—prediction in the static stage, checking against kernel-level ground truth in the empirical stage.
Action paths and topology designs (Section 3)
- 15 paths, 9 families: induced bottom-up from the routes by which actions take effect in real harnesses, including in-process tool execution (P1), direct source operations (P2), subprocesses (P3), direct network egress (P4), local intermediaries (P5), HTTP/stdio/local IPC tool server calls (P6–P8), wrapper calls (P9), agent delegation (P10), daemon submission (P11), background processes (P12), dispatch-free output consumption (P13), non-tool protocol operations (P14), reverse protocol interactions (P15) (Table 1).
- How the path basis was formed: the authors and two AI research models (Z.ai's GLM-5.3 and OpenAI's ChatGPT 6.1-Sol) each independently proposed candidate path lists, reviewed each other's, and converged.
- 8 topology designs (T0–T7): execution locations were collected from the deployed systems surveyed in Section 2 and generalized: T0 no control; T1–T3 are gateway designs (T2 adds mandatory egress filtering, T3 additionally proxies local transports through the gateway); T4–T6 are in-harness designs (T4 permission chain, T5 permission chain plus gateway, T6 declarative policy engine at the dispatch hook); T7 is kernel-level (Table 2, Figure 1). The authors state that each design generalizes the configuration of at least one deployed system (Table 4).
Deriving coverage results (Section 3.3)
- Three instance-level outcomes: checked means the action channel passes through an enforcement point; bypassed means the channel passes through no enforcement point and has no blocking mechanism; prevented means the channel is blocked by a mechanism outside the policy (such as network egress filtering); conditional additionally means the outcome depends on the specific system or deployment (Figure 2).
- Derived matrix: Table 3 gives the expected outcomes for 15 paths × 8 designs; based on this the authors point out that five paths—P2, P5, P11, P12, P15—bypass all sub-kernel designs, and T7 unconditionally covers 13 of the 15 (except P1; P11 is conditionally covered).
- Trade-off analysis: gateway designs can inspect payload content and metadata but cover few paths; kernel-level designs cover the most paths but can only see syscall-level features, are noisy, and cost more to deploy (Section 3.4).
guardscope static analysis (Section 4.2)
- Inputs and outputs: analyzes the source code of runtime controls, classifying by deployment form, whether it enforces, resolvable action classes, record format, and design it can occupy; for each path it walks a fixed checklist and gives a checkable / not-checkable / conditional / uncertain determination with file-level evidence.
- Executable artifacts: a capability manifest and binding overlays that deploy the control into each design fixture, verified before running.
- Classification of deployed systems: Table 4 assigns the surveyed systems to designs, distinguishing measured verification () from documentation-only derivation (𝑑); based on this the authors point out that most deployed solutions cluster in narrower-coverage designs, gateway designs (T1–T3) are the most numerous, and ActPlane is the only kernel-level (T7) one.
guardscope empirical measurement (Section 4.3)
- Reference workload: 15 action instances implementing 4 tested paths (P2, P4, P6, P7); deterministic replay injection drives the real harness, with no LLM in the loop, and each run executes the same actions.
- Strict readiness gate: the runner first issues an action that the loaded policy must deny, and proceeds only if the enforcement point denies it and the denial record appears in the collection stream; the measurement window is controlled only by the runner.
- Kernel-level ground-truth checking: a Tracee eBPF sensor observes the host kernel throughout and is the only ground truth; decision records are only "claims", and a reconciler checks records against kernel observations to arrive at checked-allowed, checked-denied, bypassed, prevented, or indeterminate; checked-denied must be confirmed by the action's kernel features actually being absent, and a denial that did not take effect is reported rather than trusted.
- Evidence grading: harness-run cells come from the real harness; mock-run cells come from the reference path generator fixture, whose kernel features must be verified by calibration runs to be indistinguishable from the real harness; runs without a kernel sensor enter degraded mode, and all outputs are marked as self-reported and unverified.
Implementation (Section 4.4)
- guardscope is a Python framework based on Docker Compose fixtures, with harness, control, and workload as plugins; each run pins component images by digest and records them in a run manifest, so third parties can rerun with the same pinned values; verified on arm64 (Graviton) and x86_64 hosts, with acceptance runs on both architectures producing the same result counts.
What experiments did the paper conduct?
Running the eight designs as deployments with two harnesses and real controls, the authors report that measurements confirmed the derived results for the tested action classes, and that T7's measured coverage is below the theoretical upper bound for a kernel location.
Experimental setup
- Harness: Pi (v1.0.4, runs all T0–T7 main experiments) and OpenClaw (2026.9.3, runs T1); mock-harness is a reference path generator, not a harness under test, and its kernel signature was verified by a calibration gate to be indistinguishable from a real harness (Table 5).
- Runtime controls under test: agent-os-kernel (the enforcement module of Microsoft agent governance toolkit, 3.7.0, bound to T1–T3, T5), agentgateway (v1.5.0, demonstrated separately at T1, vendor logs translated via a shim), Pi permission chain (T4), Pi in-process policy engine (T6), ActPlane (0.1.8, T7, which the authors state is the only kernel-level system in the taxonomy) (Table 5).
- Ground-truth sensor: Tracee (v0.24.1), one per run, not a plugin, and the components under test cannot select or modify it.
- Workload and adjudication: 15 action instances implement the four paths P2, P4, P6, P7; each instance is classified as checked-allowed, checked-denied, bypassed, prevented, or indeterminate, determined by a reconciler against kernel observations.
- Run protocol: For each design, a shakedown run is followed by an acceptance run, and only acceptance runs are counted; T1 is additionally run with OpenClaw, and T0–T3 are additionally run with the mock harness.
- Host environment: AWS EC2, arm64 using m7g.2xlarge (kernel 7.0.x-aws), x86_64 validation using m7i.2xlarge (kernel 7.0.0-1015-aws); T7 uses c6i.xlarge with the kernel pinned to 6.17.0-1010-aws because ActPlane's eBPF programs failed to load on the other kernels tried (Section 5.2 footnote).
Main results
The paper does not give a summary numerical table; the following are the results reported in the text of Section 5.3:
Design Control Empirical result (per Section 5.3) T1 agent-os-kernel Decision records only for the HTTP tool-call class T2 agent-os-kernel Additionally uses a forced-egress configuration to block 2 direct network actions (confirmed by the kernel) T3 agent-os-kernel Adds inspection of the stdio tool-call class T4–T6 Pi permission chain / in-process engine Two tool-call classes are mediated; the file-write, process-spawn, direct-network, and credential-access classes are bypassed in every run T5 agent-os-kernel + Pi The two tool-call classes are fully mediated, but 8 actions across 4 classes—direct network, process spawn, file write, credential access—take effect without policy evaluation in every run T7 ActPlane 6 checked-allowed, 1 checked-denied, 8 bypassed - The authors state that each design's mediation boundary is consistent with the derived results for the tested action classes, and that T2's prevented result was confirmed by the kernel, showing that the runs demonstrated prevented and not only checked and bypassed.
- In the T6 run, one action was checked and denied by the in-process policy, which the authors state shows that the run covered policy decisions and not only allowed paths; the denied action in T7 is a subprocess spawn, which ActPlane terminated at execution.
- The authors report that result counts are stable across the comparisons that can be supported: the T1 column is identical on Pi and OpenClaw, and arm64 and x86_64 hosts produce identical counts on T0–T3 and T5 (real and mock harness).
Gap between T7 empirical results and the theoretical upper bound for the kernel position
- What was measured: T7 is empirically bound with ActPlane and compared with the theoretical upper bound for the kernel position derived in Section 3 (13 of 15 unconditionally covered).
- Results: The 8 bypassed actions include all 7 tool-call actions, on which basis the authors point out that the kernel position only inspects channels covered by deployed hooks, and that the measured per-hook coverage is below the theoretical bound for the position; another direct network action (a UDP send) was bypassed because of a documented hook gap in ActPlane.
- The authors' interpretation: The authors state that this is a property of the ActPlane hooks under evaluation, not a property of the T7 design itself.
Implications of combinations
- What was measured: An analysis of whether combinations of sub-kernel designs (T1–T6) can cover the 5 fully bypassing paths (P2, P5, P11, P12, P15).
- Results and the authors' interpretation: These 5 paths pass through no enforcement point below the kernel, on which basis the authors point out that no combination of T1–T6 can inspect them; combinations only add inspectable features (as shown in the T5 run) and do not expand coverage; T7 and T4–T6 are complementary at the design level (the only path T7 cannot see, P1, is exactly what T4–T6 can inspect).
Other ablations and analyses
- Distribution of deployed systems: The classification in Table 4 shows that gateway deployments (T1–T3) are the largest group, and that the kernel-level design (T7) has only one entry, ActPlane; the authors state that most deployed solutions are concentrated in designs with narrower coverage.
- agentgateway binding: Demonstrated separately at T1, its vendor logs are translated via a shim rather than emitted natively, which the authors state tests both the ingested-record path and the native path.
- Cross-harness check: OpenClaw's result counts at T1 are identical to Pi's, which the authors use as a harness-independence check.
What are some points that can be further explored?
The authors list three limitations—insufficient validation, safety effectiveness not measured, and kernel ground truth itself being limited—and give corresponding future directions.
Limitations and future directions noted by the authors
- Insufficient validation (Section 6.1): The empirical work covers only the action classes measured by the reference workload, and the workload implements only 4 of the 15 paths (P2, P4, P6, P7); the remaining paths (including all conditional cells) rest on design analysis alone; T7's run coverage is limited by the deployed hooks rather than by the kernel position.
- Safety effectiveness not measured (Section 6.2): checked only means that the action channel passes through an enforcement point, not that the decision is correct or the policy well configured; the T7 run contains only one verified denial, and the authors state that a single denial cannot establish enforcement effectiveness; the authors propose next steps of measuring detection accuracy, policy quality, and attack protection under adversarial workloads.
- Kernel-level ground truth is limited (Section 6.3): The eBPF sensor can only observe kernel-mediated operations, cannot by itself identify which tool or call an operation belongs to, and needs to be matched with enforcement-point records for attribution; it cannot distinguish agent processes from unrelated processes (noise), carries no payload content, and its semantics must be inferred from paths, endpoints, and process history.
- Future work (Section 7): The authors plan to use dedicated workload instances to test conditional paths, evaluate policy effectiveness against adversarial workloads, and extend attribution from kernel events to agent actions.
Scope of experimental coverage
- There are 5 runtime-control bindings under test: agent-os-kernel (T1–T3, T5), agentgateway (T1 only), Pi permission chain (T4), Pi in-process policy engine (T6), ActPlane (T7) (Table 5).
- The harnesses are two real implementations, Pi (all designs) and OpenClaw (T1 only), plus a calibrated mock harness (T0–T3).
- The reference workload is 15 action instances implementing the four paths P2, P4, P6, P7; each design counts one acceptance run, and the paper does not report the number of repeated runs per design.
- Cross-architecture validation covers arm64 (Graviton) and x86_64 hosts, and the authors report identical result counts on both (T0–T3, T5).
- The formation of the path and design base used mutual review by two AI research models (GLM-5.3, ChatGPT 6.1-Sol), and the research process also used models such as Kimi 3 for assistance (Section 8); the paper does not report human agreement metrics for these AI-assisted conclusions.
Summarize the main content of the paper
guardscope turns "which action paths a control can inspect" into a measurement that can be checked against kernel ground truth; the measurements confirm the derived results and find that current deployments lean toward narrow-coverage designs.
A measurement paper on the coverage of runtime control for LLM agents. The problem the authors address is: runtime control can only inspect actions that pass through its deployment point, while an agent can take effect through multiple paths, and previously no work measured this deployment-position-determined coverage against kernel-level ground truth.
- Method highlights: Enumerate 15 action paths (Table 1) and 8 topology designs T0–T7 (Table 2), and derive the checked / bypassed / prevented result for each pair of combinations (Table 3); the static stage of the guardscope framework predicts inspectable paths from control source code, and the empirical stage runs a fixed workload of 15 action instances with deterministic replay and checks the control's decision records against Tracee eBPF kernel records.
- Derived conclusions: Five paths—P2, P5, P11, P12, P15—bypass all sub-kernel designs; T7 unconditionally covers 13 of the 15 but can only see system-call-level features; on this basis the authors point out that coverage, inspectable features, and noise exposure are three independent dimensions, and that combining designs only adds features without expanding coverage.
- Empirical results: Using Pi, OpenClaw, and real controls (Microsoft agent governance toolkit, agentgateway, ActPlane), the eight designs were run as deployments, and the authors report that results for the tested action classes are consistent with the derivations; T7 empirically shows 6 actions checked-allowed, 1 checked-denied, and 8 bypassed, with per-hook coverage below the theoretical upper bound for the kernel position, including 1 UDP send bypassed because of a documented hook gap in ActPlane.
- Stability: The T1 column has identical counts on Pi and OpenClaw, and arm64 and x86_64 hosts have identical counts.
- Deployment status: The classification in Table 4 shows that most deployed solutions are concentrated in gateway designs with narrower coverage, and that the only kernel-level design is ActPlane.
- The authors' conclusions and implications: The authors state that no single control can simultaneously provide broad path coverage and agent-semantic-level inspection, and recommend combining tools whose enforcement points are at different layers (e.g., in-harness enforcement plus kernel-level monitoring); the authors also note three limitations: the workload covers only 4 paths, safety effectiveness was not measured, and kernel ground-truth attribution is limited.