StealthyPoison: programmatic poisoning of third-party skills to manipulate LLM agent behavior

1. 01What problem does this paper attempt to solve?

   How to manipulate LLM agent behavior by poisoning the program rules of third-party skills while maintaining normal execution and evading detection.

   Core question: Can an attacker, by poisoning the decision-critical rules of a third-party skill (such as formulas, thresholds, decision logic), induce an LLM agent to produce incorrect task results while preserving the skill's advertised functionality and normal execution?

   • Scenario and importance: LLM agents increasingly retrieve third-party skills from public registries and follow their procedures to complete tasks. Since public registries accept third-party submissions, attackers can publish poisoned skills for agents to retrieve and execute, which exposes an open attack surface.
   • Shortcomings of existing methods: Prior knowledge corruption attacks (such as PoisonedRAG) craft poisoned content for known target question-answer pairs, and obvious malicious instructions provide a clear detection signal. The authors point out that how to construct procedural poisoning payloads that generalize to unseen requests without access to real user requests and their ground-truth answers remains an unexplored problem.
   • Core observation: The authors believe that by tampering with a skill's procedural rules (such as formulas, thresholds, decision logic), the agent may produce incorrect results while executing normally; this attack mechanism is called procedural poisoning.
   • Paper proposes: StealthyPoison, a stealthy procedural poisoning attack that guides the construction of poisoning payloads through synthesized requests and clean skill execution, and enhances retrieval exposure by predicting request metadata.
   • Threat model:
     • Attacker goal: Cause the poisoned skill to be selected and executed, and to produce incorrect results relative to the ground-truth answers.
     • Attacker knowledge: Has access to public target skills and public registries, but cannot access real user requests and their ground-truth answers.
     • Attacker capability: Can repackage the target skill into a poisoned version (modifying metadata, procedural instructions and bundled artifacts), but cannot modify existing clean skills, benign entries or any component of the retriever in the registry.
     • Victim system: An LLM agent that uses a public registry to retrieve third-party skills.

   Collapse this answer

2. 02What related research exists?

   Related work includes retrieval poisoning and indirect prompt injection, tool retrieval and registry manipulation, poisoned skill attacks, and pre-execution defenses.

   Retrieval poisoning and indirect prompt injection

   • Retrieval poisoning attacks: By inserting or optimizing passages so that they rank high for target queries [41, 47, 55]; optimization methods include HotFlip-style token replacement [14]. PoisonedRAG uses retrieved adversarial evidence to steer generated answers, and AgentPoison targets an agent's memory or RAG knowledge base [6, 10, 59].
   • Indirect prompt injection: Placing malicious instructions in external content such as web pages, documents or tool outputs, causing the agent to perform unintended actions [1, 13, 27, 52, 53].
   • Difference: These attacks manipulate the evidence, memory or external content that the agent reads, whereas this paper studies executable skills competing within a fully benign registry and, once selected, changing the task procedure.

   Tool retrieval and registry manipulation

   • Tool-use benchmarks: Study API discovery and invocation [24, 33, 35], as well as retrieval from large tool or skill collections [25, 40, 42].
   • ToolHijacker: Injects malicious tool documentation to steer the agent toward selecting attacker-controlled tools [39].
   • Attractive Metadata Attack (AMA): Optimizes tool names, descriptions and parameter schemas so that malicious tools are more likely to be selected [30].
   • Difference: ToolHijacker and AMA manipulate metadata to steer the agent toward selecting malicious tools, whereas this paper evaluates the entire process of a poisoned skill from retrieval to execution, and considers procedural poisoning that changes the task itself rather than adding a separate malicious operation.

   Poisoned skill attacks

   • Skill-Inject: Evaluates whether the agent follows obvious or context-relevant malicious instructions embedded in skill files [38].
   • SkillJect: Automates poisoned skill construction by hiding operational payloads in auxiliary artifacts and iteratively optimizing instructions [19].
   • Large-scale study: Finds malicious skills combining adversarial documentation and executable auxiliary scripts in public registries [26].
   • Difference: Prior work assumes the poisoned skill has already been loaded or selected, whereas this paper additionally studies whether it can be retrieved from the registry, selected and executed; procedural poisoning keeps the skill interface and artifacts unchanged but changes how the task is executed.

   Pre-execution defenses

   • Defense methods: Include instruction hierarchy, control-flow isolation, guard agents and LLM-as-a-judge filters [12, 16, 46, 49]. StruQ separates trusted instructions from untrusted data [8], and TRUSTDESC derives tool descriptions from implementations and verifies their behavioral claims [51].
   • Difference: These methods reduce prompt or metadata manipulation but do not directly check whether a skill preserves the intended task procedure.

   Baselines and benchmarks

   • Baseline methods: No Opt., HotFlip [14], GCG [58].
   • Benchmark: SRA-Bench [42], containing a retrieval registry of 5,400 requests and 26,262 skills.

   Collapse this answer

3. 03How does the paper address this problem?

   StealthyPoison constructs procedural poisoning payloads through synthetic requests and pseudo-reference guidance, and uses predicted request metadata to enhance retrieval exposure.

   StealthyPoison is a corpus poisoning attack against skill-augmented agents, divided into two stages: procedural poisoning payload construction and metadata enhancement.

   Problem Setting and Formalization

   • System model: A skill is represented as s = (m, d, A), where m is registry metadata, d is program instructions, and A is optional bundled artifacts. Given a user request q, the retriever ranks candidate skills by similarity to the request, and the agent selects skill ŝ(q) for execution.
   • Attack goal: Given a target skill s, the attacker wants its poisoned version s^⋆ to be selected and executed and to produce incorrect results relative to the ground-truth answers.

   Procedural Poisoning Payload Construction

   • Synthetic queries and pseudo-references: A public LLM generates K synthetic execution queries Q_exec(s) = {q_i}_i=1^K covering the skill's documented functionality and input conditions. A surrogate victim agent executes the clean skill to produce outputs Y = {y_i}_i=1^K as pseudo-references.
   • Manipulation planning: A Planning Agent analyzes the instruction document d, locates correctness-critical elements, and produces a set of candidate manipulation targets P(s) = {p₁, ..., p_r}, where each p_j specifies an element to modify and its intended semantic deviation.
   • Constraints: The feasible set F(s; P(s)) contains three constraints: Ω_locus restricts modifications to only the instruction locations identified by P(s); Ω_task requires preserving the original task and interface; Ω_exec requires that the candidate skill can execute and complete normally.
   • Feedback-guided iteration: An Attack Agent generates candidate poisoned skills, a Surrogate Victim Agent executes them on the synthetic queries, and a Critic Agent compares outputs with the pseudo-references and checks the constraints. Iteration runs for at most T_max = 10 rounds, with early stopping if all K = 10 query answers are changed.

   Metadata Enhancement

   • Synthetic retrieval queries: A public LLM generates 20 synthetic retrieval queries Q_ret(s) representing plausible user requests that should invoke the target skill.
   • Metadata modification: The synthetic queries are appended to the retrieval keyword field w, preserving the original content. The final poisoned skill is s^⋆ = (m[w ← w^⋆], d^⋆, A).

   Success Determination

   • Attack success condition: The poisoned skill is selected, the agent executes it without interface or execution failure, and the result is incorrect relative to the ground-truth answer. Only requests for which the clean target skill produces a correct output are counted.

   Collapse this answer

4. 04What experiments did the paper conduct?

   The experiments evaluate the construction of procedural poisoning payloads, cross-model transferability, end-to-end attack effectiveness, and the detection capability of existing defenses.

   Experimental Setup

   • Models under test: Qwen3-Coder-30B (reference at construction time), Llama-3.3-70B, Mistral-Small-24B, GPT-OSS-20B, Claude Code with Opus 4.8.
   • Dataset: SRA-Bench, containing 5,400 requests, 636 unique gold skills, and a retrieval registry of 26,262 skills; after filtering, 566 eligible skills, 5,244 requests, and 5,985 query-skill pairs were retained.
   • Baselines: No Opt., HotFlip, GCG.
   • Metrics: PoisonHit@k (PHit@k), ASRFlip, ASRpair, E2E-ASR, FPR, DetR.
   • Evaluation method: GPT-5.4-mini as the semantic judge; SkillSpector (static mode), Cisco Skill Scanner (non-LLM analyzer), and VirusTotal as defense detectors.
   • Sample size: 10 synthetic queries generated per skill; end-to-end evaluation uses 720 sampled queries (120 per dataset).

   Main Results

   Swipe horizontally to read the table

   Subset	Method	PHit@1	PHit@20	ASRFlip	Qwen3-Coder-30B	Llama-3.3-70B	Mistral-Small-24B	GPT-OSS-20B
   Overall	No Opt.	15.8	52.3	31.9	6.7	6.0	5.3	6.7
   Overall	HotFlip	26.4	57.5	51.6	10.2	9.1	8.8	9.9
   Overall	GCG	26.4	57.0	50.3	10.2	9.1	8.8	10.0
   Overall	StealthyPoison	30.0	67.6	64.0	9.8	8.7	9.4	9.3

   According to Table 2, StealthyPoison raises PoisonHit@1 from 15.8% (No Opt.) to 30.0% and ASRFlip from 31.9% to 64.0%; E2E-ASR@1 on the four execution models is 8.7%–9.8%, within at most 0.7 percentage points of HotFlip and GCG.

   • Oracle execution ASR: On Qwen3-Coder-30B, the micro-averaged ASR is 32.9%, highest on MedCalc (62.2%) and lowest on BigCode (7.5%) (Table 1).
   • Cross-model transfer: The frozen payloads achieve micro-averaged ASR of 31.0%–31.9% on Llama-3.3-70B, Mistral-Small-24B, and GPT-OSS-20B, close to the surrogate's 32.9% (Table 1).
   • End-to-end effect: In the Claude Code with Opus 4.8 workflow, the poisoned skill enters the top-20 for 73.2% of queries, is selected for 49.4%, and the final E2E-ASR is 10.1% (Table 3).

   Transfer Across Retrievers and Selectors

   • Retriever transfer: StealthyPoison outperforms HotFlip and GCG on all evaluated dense retrievers, including held-out models; it reaches 31.5% on BM25, but 15.5% on SPLADE-v3, below both baselines (Table 5).
   • Selector influence: Compared with the initial PoisonHit@1 of 30.0%, five of seven selectors improve poisoned skill placement; in the Qwen3 listwise family, overall PoisonHit@1 increases with model scale from 36.0% (4B) to 40.6% (32B) (Table 4).

   Defense Evaluation

   • Detection rates: SkillSpector, Cisco Skill Scanner, and VirusTotal detect StealthyPoison at 0.7%, 0.4%, and 0.0%, respectively; GPT-5.4-mini is 45.8% (Table 6).
   • Comparison with SkillJect: GPT-5.4-mini detects SkillJect at 93.1% and SkillSpector at 50.0%, both far higher than their detection rates for StealthyPoison (Figure 3).
   • Error-aware cases: Of 307 missed-detection cases, 59 were explicitly identified as erroneous by the judge but still labeled benign because the malicious intent was unclear.

   Other Ablations and Analyses

   • Synthetic query flip distribution: 89% of the 566 generated payloads flip at least one query, 72% flip some queries, and 17% flip all queries (Figure 2).
   • Task dependence: 40% of payloads on MedCalc flip all queries, whereas 73% of payloads on BigCode flip no queries (Figure 2).

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors note that local testing may differ from real systems, and the experiments cover 566 skills and four defense components.

   Limitations and future directions noted by the authors

   • Local testing limitations: The authors point out that all tests were conducted locally, and the measured attack success rates may differ from real-world systems; real-world systems may have stronger defenses, but may also let agents access sensitive data or important operations (Appendix B).
   • Defense needs: The authors believe that defending against procedural poisoning requires scanners capable of verifying the correctness of decision-critical procedural rules, not merely checking for malicious content or artifacts (Section 4).
   • Behavioral verification: The authors propose behavioral verification that tests candidate skills with reference requests or compares their outputs with trusted implementations, rather than relying only on package inspection (Section 4).

   Experimental coverage

   • Models tested: 5 execution models in total (Qwen3-Coder-30B, Llama-3.3-70B, Mistral-Small-24B, GPT-OSS-20B, Claude Code with Opus 4.8).
   • Retrievers and selectors: 5 retrievers (BGE-M3, BGE-large-en-v1.5, SkillRouter-Embedding-0.6B, Contriever-MSMARCO, BM25, SPLADE-v3) and 7 selectors.
   • Defense components: 4 pre-execution screening components (SkillSpector, Cisco Skill Scanner, VirusTotal, GPT-5.4-mini).
   • Sample scale: 566 eligible skills, 5,244 requests, 5,985 query-skill pairs; the end-to-end evaluation uses 720 sampled queries.
   • Unreported information: The paper does not report the specific number of queries or time required for attack construction.

   Collapse this answer

6. 06Summarize the main content of the paper

   StealthyPoison manipulates LLM agents through program poisoning and metadata augmentation, achieving end-to-end attacks on SRA-Bench and evading existing defense detection.

   StealthyPoison is a stealthy procedural poisoning attack against skill-augmented LLM agents. It induces incorrect task results by tampering with decision-critical rules of third-party skills (such as formulas, thresholds, and decision logic) while preserving the skill's advertised functionality and normal execution.

   • Problem: When LLM agents retrieve third-party skills from public registries, attackers can publish poisoned skills to manipulate agent behavior. Existing research mostly focuses on overt malicious instructions, whereas the ability of procedural poisoning to produce incorrect results while maintaining normal execution has not been explored.
   • Method: StealthyPoison has two stages: (1) procedural poisoning payload construction, which obtains pseudo-references through synthetic requests and clean skill execution and iteratively optimizes modifications to decision-critical rules; (2) metadata enhancement, which independently generates request descriptions and appends them to retrieval metadata to increase the probability that the poisoned skill is retrieved and selected.
   • Key results:
     • On Qwen3-Coder-30B, Oracle execution ASR is 32.9%, highest on MedCalc (62.2%) and lowest on BigCode (7.5%) (Table 1).
     • On the BGE-M3 retriever, PoisonHit@1 rises from 15.8% to 30.0%, and ASRFlip rises from 31.9% to 64.0% (Table 2).
     • In the Claude Code with Opus 4.8 workflow, end-to-end ASR is 10.1% (Table 3).
     • SkillSpector, Cisco Skill Scanner, and VirusTotal detect StealthyPoison at 0.7%, 0.4%, and 0.0%, respectively; GPT-5.4-mini is 45.8% (Table 6).
   • Authors' conclusion: The authors argue that procedural poisoning can induce incorrect task results while being difficult to detect, because the manipulation is embedded in otherwise legitimate task programs. Existing pre-execution defenses can detect malicious skills but often fail to identify procedurally poisoned skills. Effective defenses need to protect both registry routing and program integrity, through stronger metadata governance, retrieval manipulation detection, and independent verification of decision-critical rules.

   Collapse this answer

Related papers