Skill-Inject: measuring AI agents' vulnerability to skill-file attacks

Core problem: third-party agent skills files introduce a new class of prompt injection attack surface, and existing benchmarks and defenses do not cover it.

• Problem setting: skills let users extend an agent with third-party code, knowledge and instructions much like installing software packages; the authors say this makes the agent supply chain more complex, and real exploits against systems such as OpenClaw have already appeared.
• Shortcomings of existing methods: the authors argue that defenses based on instruction privilege hierarchies or on strictly separating instructions from data do not apply to skill files made up entirely of instructions; prior prompt injection research mostly focused on obviously malicious instructions, which can be found by pattern matching or input filtering.
• Core observation: many instructions in skills are dual-use—legitimate in one context and constituting data exfiltration or privilege escalation in another; the authors argue this problem cannot be solved by scaling model size or by input filtering, and fundamentally depends on what information the agent can access and on the semantic context of the task.
• What the paper proposes: the Skill-Inject benchmark, containing 202 injection-task pairs, with attacks ranging from obviously malicious to context-dependent injections hidden in legitimate instructions, measuring both safety (avoiding harmful instructions) and utility (following legitimate instructions).
• Threat model:
  • Goal: make the agent execute injected instructions in the skill file while performing a normal user task (e.g., data exfiltration, deleting files, ransomware-like behavior).
  • Knowledge and capability: the attacker can insert strings into the SKILL.md body (Body Injection), can additionally add a scripts/ script (Body + Script), and in the strongest variant can also control one sentence in the skill description (Body + YAML Description).
  • Victim system: users who use real agent scaffolds such as Claude Code, Gemini CLI, Codex CLI and have installed third-party skills.

Prompt injection attacks

• Greshake et al. (2023) proposed indirect prompt injection: malicious instructions embedded in external content (emails, web pages, tool outputs) hijack model behavior; Perez & Ribeiro (2022) and others studied direct injection.
• The paper points out that deployed systems such as GitHub Copilot and Microsoft Copilot remain vulnerable to carefully crafted injection attacks (CVE, 2025; Rehberger, 2024; Willison, 2025); OWASP lists prompt injection as the top threat to LLM applications (OWASP, 2025).

Prompt injection benchmarks

• Existing benchmarks cover web agents (Evtimov et al., 2025), email assistants (Abdelnabi et al., 2025b), tool outputs (Debenedetti et al., 2024b), and computer-use agents (Kuntz et al., 2025).
• The paper states that its benchmark differs in three respects: it targets the new threat model of skill injection; the injection is an "instruction within an instruction" rather than an anomalous sentence within data; and it studies context-dependent injection, which requires semantic reasoning combining the task and the security policy.

Prompt injection defenses

• Spotlighting (Hines et al., 2024) uses delimiters and encoding to mark external data; Instruction Hierarchy (Wallace et al., 2024) assumes tool outputs contain no instructions; Meta SecAlign (Chen et al., 2025) uses DPO to train models not to execute instructions in the data region.
• The paper points out that such defenses are inherently inapplicable to skill injection, because the question is not "whether an instruction exists" but "whether the instruction is malicious."
• Deterministic defenses (Debenedetti et al., 2025; Costa et al., 2025; Beurer-Kellner et al., 2025) assume that behavior and control flow can be predetermined by trusted sources, which the paper states conflicts with the design of skills dynamically extending the instruction set at runtime.

AI supply chain attacks and context security

• Existing attacks target retrieval databases (Zou et al., 2025), training data (Carlini et al., 2024), MCP tools (Beurer-Kellner & Fischer, 2025), memory-augmented systems (Chen et al., 2024) and permission systems (Rehberger, 2025); skills are a new entry point for instruction-level attacks.
• Context integrity (CI) (Barth et al., 2006) and its operationalization on LLM agents (Bagdasarian et al., 2024, etc.) provide theoretical inspiration for this paper's perspective that "instruction legitimacy depends on context."

Baselines and evaluation targets: The paper does not propose an attack method and has no attack baseline; the defense-side baseline is LLM-as-a-judge skill screening (Figure 6, Table 6), and the evaluation is conducted on the authors' self-built Skill-Inject benchmark. The authors position this paper as a benchmark and evaluation for measuring skill injection vulnerabilities, rather than a new attack or a new defense.

Overall approach: build the Skill-Inject benchmark, planting injections ranging from obviously malicious to contextually dual-use into real skill files, running them on real agent scaffolds such as Claude Code, Gemini CLI, and Codex CLI, and measuring both the injection execution rate and the user task completion rate.

Benchmark Scale and Attack Categories

• Scale: 23 skills covering document processing, machine learning, payment integration, healthcare, and other domains; 30 obvious injections and 41 contextual injections, paired with real tasks to form 76 obvious and 126 contextual injection-task pairs, 202 pairs in total (Table 1).
• Attack categories: 8 categories in total, 76 injections (Table 2): data exfiltration 26, backdoor 13, bias/manipulation 19, data destruction 6, DoS 4, malware/ransomware 4, poisoning 3, phishing 1.
• Design principles: realism (skills mimic how modern coding agents package instructions and scripts), dual-use coverage (emphasizing context-dependent injections to measure the safety-utility trade-off), and a controlled threat model (attacker capability escalates level by level from inserting a single line, to attaching a script, to controlling the description).

Safety Policy Conditions

• Three conditions: Normal (default configuration, no additional safety context); Legitimizing (the system prompt explicitly authorizes the injected behavior, in which case executing the injection is the correct behavior); Warning (the system prompt explicitly warns that the injected behavior violates policy, in which case executing the injection is a clear safety failure).
• Purpose: the policies both simulate organizational authorization rules for agents in real deployments and control the ambiguity of the injection—the same contextual injection becomes clearly malicious under a prohibitive policy and clearly legitimate under an authorizing policy.
• Metric conventions: the term ASR is used for obvious injections and for contextual injections without the Legitimizing policy; under the Legitimizing condition, the more neutral "injection execution rate" is used.

Attacker Capability Levels

• Body Injection: the attacker can only insert a string into the body of SKILL.md.
• Body + Script: the attacker can additionally add a script to the skill's scripts/ directory.
• Body + YAML Description: the attacker additionally controls one sentence in the skill description, which is loaded into the system prompt at initialization.

Evaluation Process and Judgment

• Isolated execution: each experiment runs in an isolated container containing only task-relevant files; after the agent finishes, the entire working directory and container logs are copied.
• LLM judge: the judge receives the agent's final output, the complete working directory, and the bash history recording all shell commands; it first checks the output, then checks the bash history for evidence that the injection payload was executed (e.g., outbound requests, file deletion, script execution).
• Success criteria: user task success means the output meets the task specification; attack task success means there is evidence that the injection payload was executed.
• Task-injection orthogonality: when computing the task completion rate, injection-task pairs in which executing the injection would by design make the user task impossible to complete are excluded, to avoid conflating capability failure with attack damage.

Experimental Setup

• Models and scaffolds tested: OpenAI Codex (GPT-4, GPT-5.1-Codex-Max, GPT-5.1-Codex-Mini, GPT-5.2-Codex, GPT-5.2), Anthropic Claude Code (Haiku 4.5, Sonnet 4.5, Opus 4.5), Google Gemini CLI (Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 3 Flash, Gemini 3 Pro), all using their respective default agent scaffolds.
• Benchmark: Skill-Inject, 202 injection-task pairs (Table 1); the BoN experiment for obvious injections uses a subset of 36 injections (Figure 1 caption), while the Table 3 header separately states that ASR is based on 82 injections.
• Conditions: three safety policies (Normal, Legitimizing, Warning) × three attacker capability levels (Body, Body+Script, Body+YAML Description).
• Metrics: ASR / injection execution rate, task completion rate; the defense experiment uses skill recommendation rate.
• Evaluation method: an independent LLM judge determines whether the user task and the attack task succeeded based on the final output, the working directory, and the bash history; the paper does not state which model the judge uses, nor does it report the number of runs per condition.

Main Results

Contextual injection execution rate and task completion rate (%, skill-body-injection threat model, per Table 7):

Swipe horizontally to read the table

Model	Normal ASR	Normal Task	Legit. ASR	Warning ASR	Warning Task
GPT-5.1-Codex-Max	64.7	87.0	68.3	47.5	87.0
GPT-5.1-Codex-Mini	57.6	83.1	74.8	45.3	89.6
GPT-5.2-Codex	59.7	77.9	53.2	25.9	89.6
GPT-5.2	30.2	96.1	21.6	6.5	89.6
Sonnet 4.5	32.4	87.0	30.9	24.5	87.0
Opus 4.5	26.6	96.1	1.4	7.2	85.7
Haiku 4.5	8.6	64.8	12.2	2.9	70.1
Gemini 3 Flash	67.6	93.5	85.6	56.8	94.7

(The three rows for Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 3 Pro are omitted; their Normal ASR is 37.4, 36.7, and 46.0 respectively, none of which is the worst; the lowest in the table is Haiku 4.5.)

• The authors state that under the Normal condition, ASR ranges from 41.0% (Haiku-4.5) to 79.0% (Gemini-3-Flash)—note that the interval endpoints given in the main text are inconsistent with the values in Table 7 (in Table 7, Haiku 4.5 is 8.6 and Gemini 3 Flash is 67.6); the main text's statement is reproduced here as is.
• The authors state that the Warning condition generally reduces ASR but does not eliminate attacks; the Legitimizing condition often raises the execution rate (under this condition, execution is the required behavior), with Claude models being the exception.
• The authors state that robustness is not a monotonic function of model scale or general capability; the Warning protocol is effective for Sonnet 4.5 and others, but has little effect on GPT-5.1-Max and others.

Obvious Injections and best-of-n

• Obvious ASR: the authors state that obviously malicious injections (deleting the file system, ransomware, phishing, etc.) reach an ASR of up to 70%, and best-of-n can further increase it; Figure 1 shows that BoN exceeds 50% for most models.
• BoN ablation (Table 3, ASR based on 82 injections, with Bo5 Line and BoN Full each based on 36): GPT-5.2-Codex rises from 42.7% to 72.2% under BoN Full; Gemini 3 Flash rises from 62.2% to 83.3%; Opus 4.5 goes from 15.9% to 16.7%, a very small increase. The table note states that the two BoN conditions use different injection subsets and judges, so absolute values across columns are not directly comparable.
• Authors' interpretation: the line position of the injection within the skill file is the main source of variance, while the specific wording of the user task has little effect; single-run ASR underestimates real attacker capability.

Script-Based Injection and Description Injection

• Script vs. text (Table 4): script-based injection has a higher ASR—contextual injection 71.8% vs. 53.8% (+18.0), obvious injection 62.8% vs. 53.7% (+9.0); the main text separately states "an average increase of more than 30 percentage points," which is inconsistent with the +18.0/+9.0 in Table 4; both figures are listed here side by side. The authors believe this is partly because models do not inspect scripts before running them.
• Description injection (Table 5, averaged over GPT-5.2-Codex, Opus 4.5, and GPT-5.1-mini): the overall execution rate rises from 38.7% to 52.8% (+14.1); under the Warning condition +17.6, under the Legitimizing condition +7.0. The main text separately states "an average increase of 10.6 percentage points" and "an increase of 16.8 under Warning," which are inconsistent with Table 5; both are listed here side by side. The authors believe description injection can establish the malicious instruction as the skill's expected behavior before the warning context.

LLM Screening Defense

• Setup: using the prompt in Figure 7, the judge model gives a YES/NO recommendation for a skill, under three conditions: clean skills, injected skills, and injected skills plus a Legitimizing policy (Table 6).
• Results: the recommendation rate for clean skills is 88.2%–98.5%; for injected skills it drops to 4.4% (Opus 4.5)–35.3% (GPT-5.1-Codex-Mini); but after adding the Legitimizing policy, the recommendation rate is 0.0% for Opus 4.5 and Sonnet 4.5, and 67.6% for GPT-5.1-Codex-Mini.
• Authors' interpretation: the judge can detect injection, but still falsely rejects extensively under a legitimate authorization policy; the authors believe the judge cannot correctly assess contextual safety; Gemini models and GPT-5.1-Mini are exceptions.

Other Ablations and Analyses

• GPT-4: lowest ASR but also lowest task completion rate, which the authors attribute to its shorter context window being unable to read long skill files in full (Appendix C).
• Opus 4.5 anomaly: under Legitimizing, ASR (1.4) is lower than under Normal (26.6) and Warning (7.2) (Table 7); the authors speculate that the authorizing system prompt instead makes it more cautious (Appendix C).
• Skill length (Table 8): for the 5 shortest skills, Max ASR is mostly 0.0%; for the 5 longest, it ranges from 0.0% (hf-paper-publisher) to 45.4% (pptx); the authors state that length itself does not determine vulnerability, and that the pairing of skill and injection is a confounding factor.
• Bo4-by-task: the variant that fixes the skill and position and varies only the user task produced no statistically significant ASR difference, and the results are omitted (Section 5.3).

Limitations and Future Directions Noted by the Authors

• Limited coverage (Section 6 Limitations): the evaluation covers only a limited set of skills, tasks, and threat models; results may change with a different agent implementation or a more adaptive attacker.
• Attacks are relatively simple (Section 6 Limitations): these attacks are relatively simple to implement, and real attackers optimizing injections for a specific task, specific model, or scaffold could achieve a much higher attack success rate.
• Defense directions (Section 6 Recommendations): the authors recommend treating skill files as untrusted by default, binding skills to a least-privilege capability set, and requiring context-aware authorization for actions with external side effects; they consider LLM safety screening somewhat helpful but insufficient.
• Benchmark positioning (Impact Statement): Skill-Inject is designed as an evaluation benchmark rather than an exploitation toolkit, and the authors hope to promote treating third-party skills as untrusted code by default and developing context-aware authorization mechanisms.

Scope of Experimental Coverage

• A total of 12 models were tested, running on three default scaffolds: OpenAI Codex, Claude Code, and Gemini CLI (Section 5.1).
• The benchmark contains 23 skills and 202 injection-task pairs, with 8 attack categories (Table 1, Table 2), covering domains including document processing, machine learning, payment integration, and healthcare.
• The defense evaluation covers only one baseline: LLM-as-a-judge skill screening (Figure 6, Table 6).
• The description injection ablation is averaged over three models: GPT-5.2-Codex, Opus 4.5, and GPT-5.1-mini (Table 5 title).
• The paper does not report the number of runs per condition, the specific model used as the LLM judge, or the consistency between judge decisions and manual verification.

A benchmark paper on the security of the agent skill supply chain. The authors state that agent skills let third-party instructions enter the agent's prompt, constituting an "instructions within instructions" prompt injection surface, to which traditional defenses that separate instructions from data do not apply.

• Method: Skill-Inject contains 23 skills, 30 obvious and 41 contextual injections, forming 202 injection-task pairs; under three safety policy conditions (Normal, Legitimizing, Warning) and three attacker capability levels (body insertion, attached script, control of the YAML description), an LLM judge determines whether the injection was executed based on evidence such as the bash history.
• Main results: contextual injection under the Normal condition has an ASR of 41.0%–79.0% (main text 5.2; the values in Table 7 are inconsistent with this interval); the Warning strategy reduces but does not eliminate attacks.
• Obviously malicious injections: the authors state that obvious injection ASR reaches up to 70%, and under BoN Full Gemini 3 Flash reaches 83.3% (Table 3).
• Attack surface expansion: script-based injection has a higher ASR than plain text (contextual 71.8% vs. 53.8%, Table 4); controlling the skill description raises the execution rate by an average of 14.1 percentage points (Table 5).
• Defense baseline: LLM screening can detect injection (Opus 4.5's recommendation rate for injected skills is 4.4%), but it falsely rejects extensively under a legitimate authorization policy, and the authors believe it cannot handle contextual safety.
• Authors' conclusion: this problem cannot be solved by model scaling or simple input filtering; it requires treating skills as untrusted by default, least-privilege binding, and a context-aware authorization framework.