Skip to content

Beyond Patches: Control-Gated Benign Probe Reconstruction of Patch Bypasses from Incomplete Fixes in LLM Agent Frameworks

Beyond the Patch: Reconstructing an Incomplete-Fix Patch Bypass in LLM Agent Frameworks with Control-Gated, Benign Probes

arXiv: 其他 AI 安全论文· arXiv:2611.20514· Chaitanya Garware·Indexed by this site · 2 days ago· Originally publishedAttacks· Its place in the scan record →

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors propose a control-gated benign probe reconstruction method and, on PraisonAI 4.6.31, reproduced the officially recognized CVE-2026-40287 patch bypass (CVE-2026-44334).

Questions
Replaying the original PoC after a CVE patch can only answer whether the trigger is blocked; it cannot prove that the same invariant still holds on other reachable paths; the authors focus on the problem that when a patch is incomplete, the advisory still marks it as "fixed".
Methods
Proposes a control-gated three-state reconstruction method: B0 replays the original trigger, M probes the same invariant through an independent sibling path in the same repository, benign observers record effects, and if the authorized positive control does not pass, the verdict is INCONCLUSIVE.
Experiments and results
On PraisonAI 4.6.31, B0 reports NO_VIOLATION_OBSERVED while M reports VIOLATION, reproducing the officially recognized patch bypass CVE-2026-44334; five before-and-after version reconstructions for Langroid, Agno, and CrewAI were also completed, and all 14 fault-injection scenarios met expectations.
Limitations and future work
The authors state that the same-version B0≠M divergence is demonstrated on only one boundary, PraisonAI; the PraisonAI probe directly calls the loader function rather than the full HTTP chain; the path-restriction cases were run only on Windows; and there is no independent third-party reproduction.
Experimental setupPraisonAI tool loading (CVE-2026-40287 / CVE-2026-44334)、Langroid file tools (CVE-2026-50181) et al. · tri-state verdict (VIOLATION / NO_VIOLATION_OBSERVED / INCONCLUSIVE)
Threat model
The attacker influences inputs that reach the framework boundary (tool names and arguments, evaluated expressions, file paths, surface sender), consistent with indirect prompt injection and untrusted-content scenarios; the attacker cannot modify the framework source code. The researchers measure whether the framework enforces the invariant, not how often the model emits malicious inputs.
Benchmarks
PraisonAI tool loading (CVE-2026-40287 / CVE-2026-44334)Langroid file tools (CVE-2026-50181)Langroid pandas_eval (CVE-2026-54769)Langroid dispatch (CVE-2026-54771)Agno PythonTools (CVE-2026-76832)CrewAI FileWriterTool (CVE-2026-37007)
Metrics
tri-state verdict (VIOLATION / NO_VIOLATION_OBSERVED / INCONCLUSIVE)
AI summaryThe paper proposes a benign, control-gated reconstruction method, conducts a pilot study on four LLM agent frameworks and four types of security invariants, and points out that after patching, merely replaying the vulnerability's original PoC cannot confirm that the security invariant still holds on other code paths. On PraisonAI 4.6.31, the original tool-loading trigger was blocked, but the tool_override path still violated the invariant, reproducing the publicly documented bypass of CVE-2026-40287 by CVE-2026-44334; on 4.6.32, no violation was observed on either of the two tested paths. The authors also completed five pre- and post-fix reconstructions on Langroid, Agno and CrewAI, among which CrewAI showed a violation in the tested version 1.11.0 and the behavior was constrained in 1.11.1, and they do not claim that this is the earliest fixed version. Each run was gated on an authorized control returning positive, and reported one of three verdicts: violation, no violation observed, or uncertain; all 14 scenarios used to validate the testing framework received the expected verdict.

The paper proposes a benign, control-gated reconstruction method, conducts a pilot study on four LLM agent frameworks and four types of security invariants, and points out that after patching, merely replaying the vulnerability's original PoC cannot confirm that the security invariant still holds on other code paths. On PraisonAI 4.6.31, the original tool-loading trigger was blocked, but the tool_override path still violated the invariant, reproducing the publicly documented bypass of CVE-2026-40287 by CVE-2026-44334; on 4.6.32, no violation was observed on either of the two tested paths. The authors also completed five pre- and post-fix reconstructions on Langroid, Agno and CrewAI, among which CrewAI showed a violation in the tested version 1.11.0 and the behavior was constrained in 1.11.1, and they do not claim that this is the earliest fixed version. Each run was gated on an authorized control returning positive, and reported one of three verdicts: violation, no violation observed, or uncertain; all 14 scenarios used to validate the testing framework received the expected verdict.

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Replaying the original PoC after a patch is insufficient to prove that the boundary invariant is restored on all reachable paths; the authors propose a control-gated benign probe reconstruction method to test this.

    Core problem: replaying the original PoC after a vulnerability patch can only answer "whether the reported trigger can still trigger," and cannot prove that the underlying security invariant holds on every code path that reaches the same boundary.

    • Scenario and importance: LLM agent frameworks route model- and attacker-influenced inputs to classic software boundaries such as file access, expression evaluation, and tool dispatch; the authors cite a PRISMA review stating that action-layer vulnerabilities (tool misuse, code injection, sandbox escape) account for only 4.7% of the papers. The patch empirical literature shows that initial patches may not fully eliminate the vulnerability, and the Linux kernel study catalogued thousands of incomplete fixes.
    • Shortcomings of existing practice: the authors argue that advisories record fixed versions and PoCs record triggers, but the trigger no longer triggering does not by itself show that the invariant holds on all paths; PraisonAI is a documented example—the fix for CVE-2026-40287 added an opt-in gate to two tool-loading sinks but missed the equivalent sink in templates/tool_override.py, which was later catalogued as CVE-2026-44334 ("CVE-2026-40287 patch bypass").
    • Core observation: agent framework boundaries are reached via multiple code paths (local tool loader and template override loader, default sanitizing evaluator and trusted-mode evaluator, read tools and write tools), and blocking the reported path may leave sibling paths.
    • What the paper proposes: a small controlled gated reconstruction method plus a pilot study comparing original trigger replay (B0) with sibling-path probes (M), with each run outputting a three-state judgment; benign probes do not execute real payloads.
    • Threat model:
      • Goal: measure whether the framework enforces the boundary invariant when attacker-influenced inputs arrive.
      • Knowledge: the attacker influences inputs that reach the boundary—tool names and arguments, evaluated expressions, file paths, or surface sender; consistent with indirect prompt injection and untrusted-content scenarios.
      • Capability: the attacker cannot modify the framework source code; the authors do not measure how often the model emits such inputs.
      • Victim systems: the tool loading, expression evaluation, file tools, and dispatch boundaries of four frameworks: PraisonAI, Langroid, Agno, and CrewAI.
  2. What related research exists?

    Related work falls into four groups: empirical studies of incomplete patches, systematization of the agent attack surface, classification of defense failures, and threat models at the tool and protocol layers; the authors state that their contribution is orthogonal to these.

    Empirical studies of incomplete patches

    • [1] (arXiv:2607.13206, 2026)—an empirical study of multiple patches for open-source software vulnerabilities, noting that initial patches may not fully eliminate the vulnerability or may introduce new vulnerabilities; it also specifies that a good fix test should check that "the PoC can no longer trigger and normal functionality is preserved."
    • [2] (arXiv:2511.17799, 2025)—a study of the characteristics, root causes, and detection of incomplete security fixes in the Linux kernel, cataloguing thousands of incomplete fixes that each required a follow-up fix.

    Systematization of the agentic LLM attack surface

    • [3] (arXiv:2608.10530, 2026)—a PRISMA review on understanding, identifying, and mitigating agentic LLM vulnerabilities; the authors cite its finding that action-layer vulnerabilities account for only 4.7% of the papers.
    • [4] (arXiv:2603.22928, 2026)—a systematic review of agentic LLM security; the authors cite its view that tool bridges expose classic vulnerabilities (code execution, path traversal, SSRF) when schema/validation is weak.

    Classification of defense failures

    • [5] (Ji et al., arXiv:2511.15203, 2025)—a taxonomy, evaluation, and exploitation of defense frameworks for LLM agents centered on indirect prompt injection; the authors state that it lists incomplete isolation as a recurring flaw.

    Threat models

    • [6] (Ferrag et al., arXiv:2506.23260, 2025) and [7] (arXiv:2601.17548, 2026)—from prompt injection to protocol exploitation, prompt injection attacks against agentic coding assistants; the authors state that the threat models at the tool and protocol layers in these works inspired this paper's attacker model.

    Baselines and benchmarks: this paper has no comparison baseline method; the evaluation targets are six real CVE boundaries (Table I), and the comparison is the judgment of the same probe on the before-and-after versions of the fix (B0 replay vs M probe).

    Authors' positioning: the contribution is orthogonal to new attacks or new defenses—it is a method and corresponding measurements for checking whether a fix restores a boundary invariant; relative to standard security regression testing, the specific elements it relies on are a mandatory positive control (which also serves as reachability evidence) and explicit "independent path vs input variant" labeling, and the authors explicitly state that they do not claim these elements are a new technique.

  3. How does the paper address this problem?

    Control-gated three-state reconstruction: B0 replays the original trigger, M takes an independent sibling path, benign observers record effects, and if the positive control does not pass, the verdict is INCONCLUSIVE.

    Overall approach: for each boundary at a fixed version, run a benign probe—an action that succeeds only if the invariant is violated—together with an authorized positive control on the same boundary; control gating ensures that "the boundary throws an error for all inputs" is not misjudged as fixed.

    Objects and Verdicts

    • Probes and controls: a probe is a benign action (reading a file planted outside the workspace, resolving __import__ inside an evaluated expression), and a positive control is an authorized action on the same boundary that must succeed.
    • Three-state verdict: verdict(b,v) = \begin{cases} \text{INCONCLUSIVE} & \text{control fails} \\ \text{VIOLATION} & \text{control ok} \land \text{effect seen} \\ \text{NO\_VIOLATION\_OBSERVED} & \text{control ok} \land \text{no effect} \end{cases}; that is, a failed control is always INCONCLUSIVE, a passing control with an observed effect is VIOLATION, and a passing control with no effect is NO_VIOLATION_OBSERVED.
    • Semantic constraint: NO_VIOLATION_OBSERVED means only "no violation was observed under this probe, version, and configuration"; it is not a proof of security; the authors avoid using "complete fix", "certifies", or "secure".

    Per-Path Gating and Harness Validation

    • Per-path gating: each path is gated by its own positive control; the PraisonAI harness writes two markers (reached sentinel and effect marker), and if the sink executes but no effect is observed (e.g., the planted code throws an exception), it reports INCONCLUSIVE.
    • Isolation and staleness prevention: each measurement runs in a separate process to avoid module caching and environment variable leakage; observation is token-based—a new token per run must match the marker content, and leftover markers do not count as an effect.
    • Fault-injection suite: the same verdict module was tested under 14 scenarios (2 healthy scenarios and 12 fault scenarios, including control failure, missing entry point, observer failure, two kinds of stale markers, empty record, wrongly typed boolean, wrong version, subprocess failure, etc.), and the authors report that all produced the expected verdicts.

    Distinguishing B0 and M

    • B0: restates the original path of the public PoC.
    • M: reaches the same invariant through an independent sibling path within the repository (a sibling loader or a second tool action), selected by reading the source rather than mutating the PoC string; the authors distinguish "independent code path" (ideally with its own CVE provenance) from "input variant of the same path" and label each case (Table I).

    Benign Effect Observers

    • Three observers: an in-memory flag recording whether a lazy handler was invoked; a uniquely named filesystem sentinel outside the target directory; and a builtins reachability check that resolves __import__ without calling it.
    • Benign constraint: probes plant lazy marker modules or lazy tool objects; loading executes benign Python, but no probe initiates a network connection, spawns a subprocess, or runs a payload.

    Reconstruction Protocol

    1. Read the advisory and locate the fix change in the source.
    2. Fix the last affected and first fixed versions, verified against the package index and version control history.
    3. Install each version into an isolated interpreter, run the probe and positive control, and print the actually imported package version.
    4. Record the verdict and raw output.

    Corpus and Environment

    • Corpus: Table I lists 6 boundaries, 4 frameworks, and 4 invariant types; this is purposive sampling rather than random sampling, with selection criteria being CVEs from 2025–2026 that can be reconstructed offline and have fixable before-and-after version pairs; only PraisonAI (B1) provides an independent second path with its own CVE, while the other five are before-and-after reconstructions of input variants on the same path.
    • Environment: a separate virtual environment per version; the six-boundary corpus was run on Windows under CPython 3.12; the corrected PraisonAI matrix and fault-injection suite were additionally reproduced under Linux (Ubuntu 24.04) CPython 3.13 with identical verdicts, with praisonaiagents pinned to 1.7.23.
  4. What experiments did the paper conduct?

    On PraisonAI 4.6.31, B0 reports fixed while M reports VIOLATION, reproducing the officially recognized patch bypass; there are also five before-and-after version reconstructions, 14 fault-injection scenarios, and one fix provenance investigation.

    Experimental Setup

    • Subjects under test: six security boundaries across four agent frameworks (Table I): PraisonAI tool loading (B1), Langroid file tool (B2), Langroid pandas_eval (B3), Langroid dispatch (B4), Agno PythonTools (B5), CrewAI FileWriterTool (B6); all are framework version pairs rather than models.
    • CVEs and version pairs: B1 is CVE-2026-40287/CVE-2026-44334 (4.5.139, 4.6.31→4.6.32); B2 is CVE-2026-50181 (0.63.0→0.64.0); B3 is CVE-2026-54769 (0.65.1→0.65.2, full_eval=True); B4 is CVE-2026-54771 (0.65.2→0.65.3); B5 is CVE-2026-76832 (2.3.23→2.3.24); B6 is CVE-2026-37007 (1.11.0→1.11.1).
    • Metric: three-state verdict VIOLATION / NO_VIOLATION_OBSERVED / INCONCLUSIVE; NO_VIOLATION_OBSERVED is accepted only when the positive control passes in the same run.
    • Verdict method: benign observers (in-memory flags, filesystem sentinels, builtins reachability checks) plus per-path positive controls; no LLM judging.
    • Environment: the six-boundary corpus was run on Windows/CPython 3.12; the PraisonAI matrix and fault-injection suite were reproduced on Linux (Ubuntu 24.04)/CPython 3.13 with consistent verdicts.
    • Sample size: one measurement per boundary per version (separate process); the paper does not report the number of repetitions.

    Main Results

    According to Figure 2 and §V-A/§V-B:

    Swipe horizontally to read the table

    BoundaryPathPre-fix version: verdictIntermediate version: verdictPost-fix version: verdict
    PraisonAI tool loading (B1)B0 (original CVE-2026-40287 trigger)4.5.139: VIOLATION4.6.31: NO_VIOLATION_OBSERVED4.6.32: NO_VIOLATION_OBSERVED
    PraisonAI tool loading (B1)M (tool_override, CVE-2026-44334)4.5.139: VIOLATION4.6.31: VIOLATION4.6.32: NO_VIOLATION_OBSERVED
    Langroid file tool (B2)input variant0.63.0: VIOLATION—0.64.0: NO_VIOLATION_OBSERVED
    Langroid pandas_eval (B3)input variant (full_eval=True)0.65.1: VIOLATION—0.65.2: NO_VIOLATION_OBSERVED
    Langroid dispatch (B4)input variant0.65.2: VIOLATION—0.65.3: NO_VIOLATION_OBSERVED
    Agno PythonTools (B5)input variant2.3.23: VIOLATION—2.3.24: NO_VIOLATION_OBSERVED
    CrewAI FileWriterTool (B6)input variant1.11.0: VIOLATION—1.11.1: NO_VIOLATION_OBSERVED
    • Authors' interpretation: a PoC-replay perspective would report the PraisonAI boundary as fixed at 4.6.31, whereas an invariant-guided perspective would not; the independent cataloging of CVE-2026-44334 corroborates that the residual is a real, ecosystem-recognized vulnerability.
    • Gate provenance: the authors extracted tool_resolver.py directly from the PyPI wheel, showing that the opt-in guard is absent in 4.5.139 and present in 4.5.140 and 4.6.31 (PRAISONAI_ALLOW_LOCAL_TOOLS text occurrences are 0, 3, and 3 respectively); the authors note that the advisory lineage marks 4.5.139 as the fixed version, one patch release earlier than the version in which the guard actually appears, and call this a metadata/artifact inconsistency.
    • Reachability evidence: with opt-in enabled, every path executes the module and writes two markers; the authors state this confirms the probes reach the real sink.

    End-to-End Reachability Verification (B3, B4)

    • What was tested: for B3 and B4, the full input-parsing pipeline was additionally driven—an attacker-shaped ChatDocument whose content is tool JSON was passed to the agent's normal handle_message, which parses and dispatches it, with no pre-injected objects.
    • Results: end-to-end verdicts are consistent with boundary-level ones (B3: 0.65.1 VIOLATION, 0.65.2 NO_VIOLATION_OBSERVED; B4: 0.65.2 VIOLATION, 0.65.3 NO_VIOLATION_OBSERVED), with positive controls passing throughout.
    • Authors' interpretation: the authors state this establishes that attacker-controlled input can reach the patched sink, rather than only behaving correctly under direct invocation; the injected probes are retained as a faster equivalent.

    Fix Provenance (CrewAI, CVE-2026-37007)

    • What was tested: the CVE record cites only the fix commit (crewAIInc/crewAI 713fa7d) and no fixed package version; a third-party database infers the affected range as "≤1.10.2rc1".
    • Results: in 1.11.0 (a stable release after 1.10.2rc1), FileWriterTool._run still directly concatenates the attacker-controlled filename, and the run is judged VIOLATION; the restriction check first appears in 1.11.1, judged NO_VIOLATION_OBSERVED. The authors state the third-party inference is incorrect, but do not claim that 1.11.1 is the earliest fixed version (not every intermediate pre-release was tested), nor do they claim the official metadata is wrong.
    • Follow-up: the authors plan to notify the maintainers and aggregators; the paper states that no reports had been sent and no confirmation received at submission time.

    Other Ablations and Analyses

    • Fault injection: all 14 scenarios produced the expected verdicts—all genuinely broken conditions reported INCONCLUSIVE, blocked stale markers reported NO_VIOLATION_OBSERVED, and overwrite scenarios reported VIOLATION (§III, §V-A).
    • Agno secondary corroboration: four kinds of broken conditions were artificially created on the real Agno boundary (wrong package import, forced control failure, unreachable sink, fixed-name stale marker); the first three each reported INCONCLUSIVE, and the stale-marker scenario exposed the false VIOLATION that a non-unique sentinel would cause (§VII).
    • Configuration-related observation: the authors observed violations on earlier Langroid versions (0.59.31, 0.59.32) with full_eval=True, but do not count these as incomplete fixes—the earlier advisories (CVE-2025-46724, CVE-2026-25481) target a different configuration, the default-mode sanitizer (§VI).
    • Excluded case: in the LlamaIndex encode_image path traversal advisory (recorded as fixed in llama-index-core 0.12.41), the cited function is byte-identical between 0.12.40 and 0.12.41; the authors state that byte identity cannot prove the patch ineffective (the change may be in the caller, a helper, or packaging), so they record this case as unresolved and exclude it from the corpus (§V-C).
  5. What are some points that can be further explored?

    The authors list limitations including the semantic boundary of non-violation judgments, the single-case divergence, direct invocation, and platform constraints; the experiments cover four frameworks and six boundaries, with no third-party reproduction.

    Limitations and Future Directions Noted by the Authors

    • Meaning of a non-violation verdict (§VII): NO_VIOLATION_OBSERVED means only "no violation was observed under this probe, version, and configuration"; a stronger probe may find a bypass that a weaker probe missed.
    • Only one same-version divergence (§VII): the B0≠M claim is demonstrated on only one boundary, PraisonAI; the other five are before-and-after reconstructions rather than sibling-path divergences, and the authors state that the core claim is a case study plus supporting measurements, not generalization.
    • Direct invocation (§VII): the PraisonAI probe directly calls the documented loader function and plants tools.py as a test setup, establishing boundary behavior under that setup rather than the full HTTP-request-to-server exploit chain described in the advisory; B2, B5, and B6 are also direct tool API calls.
    • Harness guarantees are bounded (§VII): the control gate and the 14-scenario fault suite show that all tested broken conditions report INCONCLUSIVE, but this is evidence, not proof that "no broken condition capable of producing a false non-violation exists".
    • Verdict conservatism (§VII): mapping a failed positive control to INCONCLUSIVE may suppress a directly observed violation when an unrelated dependency fails; the authors accept this conservative bias to avoid false non-violations.
    • Platform and reproduction (§VII): the path-restriction cases (B2, B5, B6) were run only on Windows, and symlink and path semantics differ on other platforms; the authors state that a Linux replication is left for future work (the host machine's WSL Python version is incompatible with the pinned framework versions). All runs were performed by the authors themselves (AI-assisted) in their own two environments, with no independent third-party reproduction.
    • Configuration scope (§VII): B3 was measured under full_eval=True, and the verdict does not transfer to the default mode.
    • Pending notifications (§V-C, §IX): the authors plan to notify maintainers and aggregators about the precision of the CrewAI fixed version, which had not been sent at submission time; there is also an unfixed PraisonAI sibling sink (agents_generator.py) that is publicly documented, for which the authors did not build a probe, stating that any check would go through coordinated disclosure in an isolated environment.

    Experimental Coverage

    • Subjects under test: six boundaries across four frameworks (PraisonAI, Langroid, Agno, CrewAI) and four invariant types (load gating, path restriction, eval-globals, sender verification), all purposively sampled rather than randomly sampled (§IV).
    • Path types: only B1 is a genuinely independent code path with its own CVE provenance; B2–B6 are input-variant probes on the same path (Table I).
    • End-to-end verification: only B3 and B4 were driven through the real input-parsing pipeline; B2, B5, and B6 are direct tool API calls (§VII).
    • Runtime environment: the six-boundary corpus on Windows/CPython 3.12; the PraisonAI matrix and fault suite reproduced on Linux/CPython 3.13 (§IV, §VII).
    • Unreported items: the paper does not report the number of repetitions per measurement, analyst time, or runtime ( §VI explicitly states that no quantitative cost claims are made); runs were not conducted under OS-level network/process monitoring, and "no network/no subprocess" describes how the probes are written rather than a monitored measurement (§IX).
  6. Summarize the main content of the paper

    A methods-plus-case-study paper: the control-gated benign probe reconstruction method reproduced the patch bypass officially recognized by PraisonAI and completed measurements across five before-and-after versions.

    This is a pilot paper of the "method + case study" type, focusing on the verification gap in security patches for LLM agent frameworks.

    • Problem: CVE advisories record fixed versions, and PoC replay only checks whether the reported trigger can still be reproduced; neither can establish that a boundary invariant is restored on every reachable path; the fix for PraisonAI's CVE-2026-40287 missed an equivalent sink in tool_override and was later catalogued as CVE-2026-44334, a documented instance.
    • Method: controlled gated three-state reconstruction—B0 replays the original trigger, M probes the same invariant via an independent sibling path in the repository; benign observers (memory flags, file sentinels, builtins reachability) record effects; if the positive authorization control does not pass, the result is judged INCONCLUSIVE, and NO_VIOLATION_OBSERVED is not treated as a proof of security.
    • Core result: on PraisonAI 4.6.31, B0 reports NO_VIOLATION_OBSERVED while M reports VIOLATION, reproducing the officially recognized "CVE-2026-40287 patch bypass"; both paths are closed in 4.6.32 (Figure 2, 3).
    • Supporting measurements: reconstructions of five before-and-after versions of Langroid (file tools, pandas_eval, dispatch), Agno, and CrewAI all show "VIOLATION before the fix, NO_VIOLATION_OBSERVED after the fix"; B3 and B4 are additionally verified end-to-end through the real input parsing pipeline, with consistent judgments.
    • Fix provenance: for CVE-2026-37007, which has no recorded fixed version, the authors observe VIOLATION in crewai-tools 1.11.0 and a restriction in 1.11.1, and state that the third-party inference of "≤1.10.2rc1" is incorrect.
    • Authors' conclusion: the authors state that the value of the method lies in providing a discriminative check—whether a fix restores the invariant—and in catching residuals missed by PoC replay and advisory labels; they also explicitly state that the same-version divergence demonstrates only one case, does not estimate prevalence, and is not claimed to be a new testing technique.
Read originalarxiv.org