Skip to content

SkillHarm: Automated Attack Construction for the Skill-Use Lifecycle

SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction

arXiv· arXiv:2606.02540· Yuting Ning, Zhehao Zhang, Yash Kumar Lal, Boyu Gou, Junyi Li, Weitong Ruan, Chentao Ye, Rahul Gupta, Diyi Yang, Yu Su, Huan SunThe Ohio State University; Amazon AGI; Stanford UniversityIndexed by this site · 19 days ago· Originally publishedSelected AttacksSelectedAttention39

Quick read

Evaluation and benchmarks

Based on the paper PDF (AI-generated); refer to the original

The authors construct the SkillHarm benchmark to evaluate agent skill poisoning attacks; across 8 model-harness configurations, FPP ASR is at most 86.3% and SMP at most 69.3%.

Questions
Third-party agent skills are trusted and executed by agents by default, making them a vulnerable surface; existing skill attack benchmarks cover only single task execution and have scattered risk lists, lacking evaluation of cross-session reuse and systematic risk classification.
Methods
Propose the SkillHarm benchmark, defining two attack scenarios, FPP (fixed payload) and SMP (self-mutating), and 12 risk categories; use the AutoSkillHarm pipeline to let a coding agent automatically complete target selection, attack design, and quality filtering, driven by a natural-language harness.
Experiments and results
Across 8 model-harness configurations, FPP ASR is at most 86.3% (Codex GPT-5.4) and SMP at most 69.3%; the authors state that many failures stem from the agent not touching the poisoned file, and after exposure cASR rises; neither skill scanners nor defensive system prompts can reliably reduce ASR.
Limitations and future work
The authors note: the tasks are based on SkillsBench, mainly programming and data processing, and do not cover domains such as web browsing or multi-agent collaboration; SMP has only 12 task pairs and uses only one delayed mutation mechanism, atexit. DSP defenses are evaluated only on the high-ASR subset, and guardrails are evaluated only on a subset of 200 trajectories.
Experimental setupSonnet 4.6、Opus 4.7 et al. · SkillHarm、SkillsBench · ASR、cASR et al.
Threat model
The attacker is a skill publisher who publicly releases a poisoned skill package, controls all files within the package (SKILL.md, reference documents, executable scripts, etc.), and knows the skill's claimed purpose; however, the attacker has no post-installation access, cannot observe victim execution, cannot adjust the payload online, and does not know the specific downstream prompts or system configuration. The victim installs the skill following the standard procedure, and the agent loads and executes the skill content when completing related tasks.
Evaluated models
Sonnet 4.6Opus 4.7GPT-5.4GPT-5.5Gemini 3 FlashQwen-3.6 27BGLM-5.2Kimi K2.6GPT-5.6 TerraSonnet 5
Benchmarks
SkillHarmSkillsBench
Metrics
ASRcASRARRTask Success RateDetection RateFNRFPR
AI summaryand selection rationaleThe researchers propose the SkillHarm benchmark to evaluate the safety of agents against skill poisoning attacks during the skill-use lifecycle, along with AutoSkillHarm for automatically constructing attack samples. The benchmark contains 879 attack samples, covering 71 skills and 12 risk categories, and distinguishes two scenarios: fixed-payload poisoning (FPP) and self-mutating poisoning (SMP); tested on eight model combinations of Claude Code, Codex, Gemini CLI and OpenCode, Codex with GPT-5.4 achieves the highest FPP attack success rate of 86.3% and the highest SMP rate of 69.3%. Conditional attack success rates show that some failures stem from the agent not touching the poisoned file; after contact, the success rate increases by up to 32.1 percentage points, corresponding to SMP with Claude Code and Opus 4.7. The strongest scanner configuration detects only 55.6% of FPP injections and 68.8% of SMP injections, and under defensive system prompts the ASR of most configurations remains above 70%.

The researchers propose the SkillHarm benchmark to evaluate the safety of agents against skill poisoning attacks during the skill-use lifecycle, along with AutoSkillHarm for automatically constructing attack samples. The benchmark contains 879 attack samples, covering 71 skills and 12 risk categories, and distinguishes two scenarios: fixed-payload poisoning (FPP) and self-mutating poisoning (SMP); tested on eight model combinations of Claude Code, Codex, Gemini CLI and OpenCode, Codex with GPT-5.4 achieves the highest FPP attack success rate of 86.3% and the highest SMP rate of 69.3%. Conditional attack success rates show that some failures stem from the agent not touching the poisoned file; after contact, the success rate increases by up to 32.1 percentage points, corresponding to SMP with Claude Code and Opus 4.7. The strongest scanner configuration detects only 55.6% of FPP injections and 68.8% of SMP injections, and under defensive system prompts the ASR of most configurations remains above 70%.

Why it was selectedThe paper presents an evaluation set for agent skill poisoning covering single-use and cross-session reuse, and reports attack success rates and refusal differences across multiple agents, which can be used to compare skill supply chain risks.

Related events and news

1 events · 1 news mentions

News that mentions this paper

北大与微信安全团队发布通用智能体全生命周期安全综述

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Existing skill attack benchmarks only test single executions and have scattered risk lists; the authors propose the SkillHarm benchmark covering the full lifecycle of skill use.

    Core question: how to systematically evaluate agent vulnerability to third-party poisoned skills, covering the full lifecycle of cross-session skill reuse and a systematic risk taxonomy.

    • Problem setting: Skills are reusable components packaging natural language instructions, reference materials, and executable scripts; agents are expected to follow and execute them and occupy a trusted privileged position; the authors state that this makes third-party skills an easily attacked surface, where malicious content can be quietly executed through the agent's permissions over files, shell, credentials, and tools.
    • Existing gaps: The authors state that existing skill attack benchmarks abstract skill use as a single task execution and judge success or failure within that run, missing the lifecycle in which skills persist across tasks, are reused, and are modified; and risk coverage is mostly benchmark-specific risk lists, lacking systematic organization.
    • Core observation: Skills are persistent procedural artifacts, and harm can be realized at different lifecycle points—triggered directly within the same execution, or by first modifying persistent skill content and triggering only upon later reuse; single-execution evaluation cannot see the latter failure mode.
    • Paper output: Proposes the SkillHarm benchmark (879 attack samples, 71 skills, 12 risk categories) and the automated construction pipeline AutoSkillHarm, and defines a three-category risk taxonomy organized by workflow component.
    • Threat model (§3.1):
      • Goal: Through publicly distributed poisoned skill packages, make the victim agent achieve a harmful final state during normal skill use.
      • Knowledge: Knows the skill's claimed purpose and the task categories it supports.
      • Capability: Controls all files in the package (SKILL.md, reference documents, scripts, auxiliary resources); has no post-install access, cannot observe victim execution, cannot adjust payloads online, and does not know the specific downstream prompts, files, or system configuration.
      • Victim system: After the user installs the skill, the agent loads the skill instructions, references, and scripts in the standard workflow when completing related tasks.
  2. What related research exists?

    Related work falls into two categories, skill attacks and automated attack construction; the authors state that this paper covers both cross-session reuse and systematic risk taxonomy, distinguishing it from all existing benchmarks.

    Agent Prompt Injection and Skill Attacks

    • Prompt injection attack surfaces: Research tools such as InjecAgent (Zhan et al., 2024) and AgentDojo (Debenedetti et al., 2024), and attack surfaces such as web pages, MCP servers, memory, and the OS environment; the authors state that skills are a distinct attack surface because agents not only read skills but also follow their instructions and execute their scripts.
    • Skill attack benchmarks: SKILL-INJECT (Schmotz et al., 2026, malicious skill files), SkillJect (Jia et al., 2026, trajectory-optimized injection), PoisonedSkills (Qu et al., 2026, document-driven payloads), BadSkill (Tie et al., 2026, model-in-skill threat), SkillSafetyBench (Jin et al., 2026, poisoned skill artifacts), SkillAttack (Duan et al., 2026, iterative path refinement), SkillTrojan (Feng et al., 2026, backdoored skills). The authors state that none of these benchmarks simultaneously covers cross-session skill reuse and systematic risk organization.
    • Ecosystem evidence: Empirical studies and security reports by Liu et al. (2026a,b) and OWASP Foundation (2026); a paper footnote cites Liu et al. (2026b) stating that 26.1% of public skills contain at least one vulnerability.

    Automated Attack Construction

    • Template/seed mutation: SkillTrojan and BadSkill reuse a small number of templates or fixed rules, which the authors state limits attack diversity; Appendix A notes that SkillTrojan's 3,000+ samples share the same encrypted payload fragmentation strategy, so a single defense may neutralize the entire batch.
    • Fixed LLM workflows: PoisonedSkills uses hard-coded schemas and orchestration logic, which the authors state is fragile under heterogeneous settings.
    • Victim-in-the-loop optimization: AgentVigil (Wang et al., 2025b), SkillJect, SkillAttack, and others require repeated victim rollouts, which the authors state is costly and may overfit to the assumed victim, inconsistent with public release scenarios.

    Baselines and Comparison Setup

    • Comparison benchmarks: Table 1 compares with the above 7 skill attack benchmarks on number of samples, number of skills, number of risk types, whether a risk taxonomy is present, whether cross-session is supported, whether deterministic evaluation is used, and construction method.
    • Defense baselines: Two skill scanners, Skill Scanner (Cisco AI Defense, 2026) and Agent Scan (Snyk, 2026), as well as defensive system prompt (DSP) and prompt-based output guardrails.

    Author positioning: SkillHarm extends evaluation along two dimensions, the cross-session lifecycle and 12 categories of workflow risk, and replaces hard-coded pipelines with a coding agent driven by a natural language harness.

  3. How does the paper address this problem?

    AutoSkillHarm uses a natural language harness to drive a coding agent, automatically constructing attack samples for two scenarios through three stages: target selection, attack design, and quality filtering.

    Overall approach: formalize skill attacks as a design space of two lifecycle scenarios (FPP, SMP) × 12 risk categories, then use the AutoSkillHarm pipeline to let a coding agent automatically construct attack samples driven by a natural-language harness, with each sample carrying a deterministic attack-success evaluator.

    Problem setting and formalization

    • FPP (Fixed-Payload Poisoning): The payload is present in the package at installation time and remains fixed; it can appear in instructions, reference documents, scripts, or newly added artifacts; any task execution that reaches the payload realizes harm within the same session.
    • SMP (Self-Mutating Poisoning): The skill contains a mutation hook rather than a direct payload; the first session behaves normally but quietly modifies persistent skill content, and harm is delayed until a later session reuses the modified skill.
    • Risk taxonomy: 12 risk categories are divided into three classes by the workflow component in which harm occurs (Table 2): data pipeline exploitation (3 categories), system environment exploitation (6 categories), and agent autonomy exploitation (3 categories).

    Attack target selection (§4.1)

    • FPP: The target is an injection point (T, f), i.e., a task T and a skill file f that the agent may read or execute; the exposure rate is rT,f = nT,f/NT, i.e., the proportion of benign baseline trajectories that reach f; files with r ≥ 0.30 are retained, with at most 3 documents and 3 scripts per task, yielding 142 injection points (36 scripts, 106 documents).
    • SMP: Select task pairs (TA, TB) that reuse the same skill; three selection agents (Codex GPT-5.2, Codex GPT-5.4, Claude Code Opus 4.6) independently propose based on feasibility and impact (each scored 1–10), aggregated by score = votes × feasibility × impact, yielding 12 task pairs.
    • Task source: Based on runnable task environments, skill packages, and benign baseline trajectories from SkillsBench.

    Attack design (§4.2)

    • Design process: The main designer is Claude Code (Opus 4.7); it first explores the skill package to find attack vectors, instantiates abstract risks into harmful end states that can be checked deterministically, then modifies skill files to implement the payload, and finally writes a deterministic evaluator that gives a binary success signal.
    • Payload forms: For FPP document targets, 4 implementations (in-place rewriting, routing to a newly added .md, newly added scripts, newly added PDFs); for script targets, 2 (in-place code insertion, importing a newly added module); SMP uses an atexit callback to rewrite files in the same package when the script exits, and covers the skill directories of each framework.
    • Iterative self-refinement: The designer checks whether the injected content blends in naturally and may invoke an LLM-based detector; when flagged, it modifies the presentation or placement of the payload while keeping the harmful end state unchanged, until there are no new severe flags.
    • Environment grounding: The designer modifies the task Dockerfile to preset attack prerequisites (e.g., preset target files, directories, environment variables, simulated HTTP endpoints).

    Quality filtering and success determination (§4.3, §5.1)

    • Quality filtering: First, execute end-to-end on two representative victim agents (Claude Code Sonnet 4.6, Codex GPT-5.4) and record trajectories; a judge agent (Claude Code Opus 4.7) performs a trajectory-grounded review with three groups of checks—target validity, evaluator faithfulness, and payload quality—and candidates with construction defects are discarded; 70% are retained overall.
    • Success determination: The main metric ASR is computed by a deterministic evaluator on the environment's end state (no LLM involved); cASR is ASR conditioned on the agent touching the target payload file, and ARR is the explicit refusal rate; the latter two are determined by GPT-5.4 as an LLM judge from trajectories.
  4. What experiments did the paper conduct?

    Across 8 model-harness configurations, FPP ASR is at most 86.3% and SMP at most 69.3%; the authors state that neither scanners nor defensive prompts can reliably mitigate.

    Experimental Setup

    • Configurations under test: 4 agent harnesses × 8 models: Claude Code (Sonnet 4.6, Opus 4.7), Codex (GPT-5.4, GPT-5.5), Gemini CLI (Gemini 3 Flash), OpenCode (Qwen-3.6 27B, GLM-5.2, Kimi K2.6).
    • Benchmark: SkillHarm, 879 samples (687 FPP, 192 SMP), 57 user tasks, 71 skills, 12 risk categories (Table 3); SMP covers 12 task pairs and 6 skills.
    • Execution protocol: FPP runs a single task with the poisoned skill and then checks the environment's final state; SMP first runs TA, carries the skill package snapshot into the TB environment, then runs TB, and judges after TB. Single run per configuration (Appendix D.1).
    • Metrics: ASR (deterministic evaluator), cASR (under exposure condition), ARR (explicit refusal rate); exposure and refusal are judged by GPT-5.4 as LLM judge.
    • Judge reliability: GPT-5.4 and human annotation agree on exposure judgment at 97% and refusal at 99% on 50 sampled trajectories, with Cohen's κ of 0.91 and 0.96 respectively (Appendix D.2).
    • Construction validation: On 40 sampled candidates, human and reviewer agent retention/filtering decisions agree at 87.5% (Appendix B.4).

    Main Results

    According to Table 4 (ASR / cASR / ARR, unit %):

    Swipe horizontally to read the table

    HarnessModelFPP ASRFPP cASRFPP ARRSMP ASRSMP cASRSMP ARR
    Claude CodeSonnet 4.652.462.625.351.670.22.1
    Claude CodeOpus 4.727.441.337.89.441.514.1
    CodexGPT-5.486.390.72.869.377.41.0
    CodexGPT-5.581.485.64.665.672.73.1
    Gemini CLIGemini 3 Flash63.881.70.645.861.40.5
    OpenCodeQwen-3.6 27B53.968.45.851.665.00.0
    OpenCodeGLM-5.252.062.93.242.458.83.6
    OpenCodeKimi K2.648.862.020.032.858.00.0
    • Authors' interpretation: The two Codex configurations have the highest ASR in both scenarios, and Claude Code (Opus 4.7) the lowest; many apparent failures are because the agent did not touch the poisoned file, and after exposure cASR rises, with the largest being +32.1 percentage points for Opus 4.7 on SMP.
    • Refusal behavior: The authors state that the Claude family refuses most, and ARR decreases in SMP (Sonnet 4.6 from 25.3% to 2.1%, Kimi K2.6 from 20.0% to 0.0%); the authors believe that the temporal decoupling of preparatory actions from harmful outcomes reduces the visibility of malicious intent.
    • Risk and payload form: Appendix D.3 shows that the data pipeline exploitation category has the lowest average ASR among the three categories; the authors state that script-borne payloads have higher ASR than document-borne ones, because agents often execute auxiliary scripts without inspecting their content.

    Defense Evaluation

    • Skill scanners (Table 5): Skill Scanner with GPT-5.5 / Opus 4.7 as backbone has injection detection rates of 21.5% / 55.6% on FPP and 5.2% / 68.8% on SMP; Agent Scan is 7.4% (FPP) and 9.9% (SMP). The authors state that contextualized injections often bypass detection.
    • Defensive system prompt (DSP): Evaluated on a risk-stratified high-ASR subset (top 30% by average ASR per risk type); Figure 3 shows that DSP reduces ASR in some configurations, but in the figure almost all configurations remain above 70% in both scenarios, and the authors state that prompt-level warnings are unreliable.
    • Output guardrails (Appendix E.2, subset of 200 trajectories): GPT-5.6 Terra has FNR of 4% (FPP) / 12% (SMP) and FPR of 42% / 40%; Sonnet 5 has FNR of 36% / 44% and FPR of 26% / 14%. The authors state that a detection–interference trade-off exists.

    Task Utility and Joint Analysis

    • Task success rate (Table 6): Under most configurations, poisoned skills maintain similar task success rates; the exception is Claude Code (Opus 4.7), which drops from 61.4% to 47.6% (-13.8 percentage points), which the authors state is consistent with its stronger refusal behavior.
    • Joint analysis (Table E.2, average over six representative agents): Task and attack both succeeding accounts for 30.1% (FPP) and 19.6% (SMP); conditional on attack success, the proportion where the task also succeeds averages 48.1% (FPP) and 37.9% (SMP). The authors state that many attacks quietly achieve harm while the user receives seemingly normal results.

    Other Ablations and Analyses

    • Harness–model decoupling (Table E.3): After switching GPT-5.4 from Codex to OpenCode, FPP ASR drops from 86.3% to 76.1%; Opus 4.7 under OpenCode has SMP ASR of 39.5%, higher than 9.4% under Claude Code; the authors state that vulnerability is a property of the model-harness whole.
    • Designer ablation (Table E.4): FPP is insensitive to designer choice (attacks designed by Codex GPT-5.4 and Claude Code Opus 4.7 have the same average ASR); in SMP, attacks designed by Claude Code (Opus 4.7) are stronger (55.0% vs 44.4% against Codex GPT-5.4 victims).
    • Refinement dynamics (Table F.1): The designer calls the detector on average 3.9 times per design (FPP) / 3.7 times (SMP), and 12% (FPP) and 14% (SMP) of designs made evasive modifications after severe flags; the main text states that 87% of attacks pass the detector within two attempts.
  5. What are some points that can be further explored?

    The authors note that task domain and SMP mechanism coverage are limited; experiments cover 8 configurations and two scenarios, and some defenses are evaluated only on small subsets.

    Limitations Noted by the Authors and Future Directions

    • Task and skill coverage (Limitations): SkillHarm is based on SkillsBench and is mainly programming and data processing workflows, spanning eight application domains; domains such as web browsing, multi-agent collaboration, and physical-world tool use are not yet covered.
    • SMP scale (Limitations): The cross-session scenario is limited to 12 task pairs, because the two tasks must share executable skill content; the authors state that these constraints come from the underlying task substrate rather than the construction method, and the pipeline can scale to larger skill libraries.
    • SMP mechanism (Limitations): SMP samples use only one controlled delayed mutation mechanism (atexit mutation) as the persistence mechanism; the authors point out that the persistence mechanism is independent of the harmful effects it produces, and SMP still covers all 12 risk categories.
    • Defense directions (§5.3): The authors state that existing defenses provide insufficient protection, and effective mitigation may require mechanisms beyond prompt-level intervention, such as effective runtime guardrails, runtime access and permission control, or safety post-training.

    Scope of Experimental Coverage

    • Subjects under test: 4 harnesses (Claude Code, Codex, Gemini CLI, OpenCode) × 8 models, 8 configurations in total; the harness–model decoupling experiment additionally tests Opus 4.7 and GPT-5.4 under OpenCode (Appendix E.4).
    • Benchmark scale: 879 samples (687 FPP, 192 SMP), 57 tasks, 71 skills, 12 risk categories; SMP covers 12 task pairs and 6 skills (Table 3, Appendix C).
    • Defense evaluation: Two skill scanners (Skill Scanner with two backbones, Agent Scan), DSP (on the high-ASR subset in the top 30% by average ASR per risk type), output guardrails (balanced subset of 200 trajectories, Appendix E.2).
    • Number of runs: Appendix D.1 states that ASR is computed from a single run over the full sample set; the paper does not report multiple repetitions or variance.
    • Human validation: Human validation of construction filtering is 40 candidates (Appendix B.4); human agreement validation of the LLM judge is 50 trajectories (Appendix D.2).
  6. Summarize the main content of the paper

    SkillHarm uses an automated pipeline to construct 879 skill attack samples covering two lifecycle scenarios; the authors state that existing agents are generally vulnerable and defenses are insufficient.

    SkillHarm is a benchmark for evaluating agent skill poisoning attacks, accompanied by the automated construction pipeline AutoSkillHarm.

    • Problem addressed: Third-party skills are trusted and executed by agents by default and are an easily attacked surface; the authors state that existing benchmarks only test single task execution and have scattered risk lists, and cannot answer whether agents are safe under skill reuse and different attack outcomes.
    • Method highlights: Defines two scenarios, FPP (fixed payload, direct trigger within a single session) and SMP (atexit hook self-mutation, delayed cross-session trigger), with 12 risk categories organized into three groups: data pipeline, system environment, and agent autonomy; AutoSkillHarm has a coding agent driven by a natural language harness complete target selection (exposure rate threshold 0.30), attack design (Claude Code Opus 4.7 as main designer, LLM detector-assisted self-refinement), and quality filtering (retain 70%), with each sample carrying a deterministic evaluator.
    • Main results: Across 8 model-harness configurations, FPP ASR is at most 86.3% (Codex GPT-5.4) and SMP at most 69.3% (same configuration, Table 4); cASR conditioned on touching the poisoned file rises, by up to +32.1 percentage points (Claude Code Opus 4.7, SMP); refusal rate decreases in SMP (Sonnet 4.6 from 25.3% to 2.1%); the strongest scanner configuration detects only 55.6% (FPP) and 68.8% (SMP) of injections (Table 5); on the DSP subset, almost all configurations still have ASR above 70% (Figure 3).
    • Authors' conclusion: The authors state that many apparent attack failures are merely the agent not touching the poisoned file rather than genuine resistance, cross-session temporal decoupling further suppresses refusal, and neither scanners nor defensive prompts can reliably mitigate; the authors consider skill attacks a unique and understudied area of agent safety and release SkillHarm to support defense research.

Related papers

Show 2 more papers
Read originalarxiv.org