How LLM Agent Skills Leak User Credentials: An Empirical Study

Core problem: third-party agent skills handle sensitive credentials in privileged execution environments, but how credentials leak through the skill ecosystem still lacks systematic study.

• Problem scenario and importance: agent skills are hybrid artifacts of NL descriptions plus executable code, spreading rapidly in frameworks such as ClawHub; the authors state that many frameworks run skills directly on the user's local machine, giving them ambient authority over environment variables, configuration files, and the host network, making credential leaks both a supply chain and a local problem.
• Insufficient existing research: the authors state that existing work (such as vulnerability analysis of 31,132 skills and a behavioral study confirming 157 malicious skills) characterizes the overall vulnerability surface and attacker strategies, but to their knowledge no work has systematically studied how credentials leak through the agent skill ecosystem.
• Core observation: the NL of skills enters the LLM reasoning context, and the PL inherits local credential storage; the two modalities may expose credentials in ways that neither analysis alone can discover; the authors consider this cross-modal interaction the core risk distinguishing agent skills from traditional single-modality software packages.
• Paper outputs: a four-stage empirical analysis of 17,022 skills sampled from SkillsMP, a dataset containing 437 vulnerable and 83 malicious skills, a taxonomy of 10 leak patterns, and responsible disclosure.
• Threat model (the leak scenarios characterized in this study):
  • Target: authentication materials handled by the skill (9 categories such as API keys, OAuth tokens, and private keys, Table 1) exposed to undeclared recipients or channels.
  • Attacker: a developer publishing a malicious skill, using NL/PL decoupling to present a benign interface while executing multi-objective payloads; or embedding indirect prompt injection in external content consumed by the skill.
  • Victim system: an LLM agent running the skill locally with the full privileges of the host process (with Claude Code as the runtime in the sandbox) and its users.

Secret detection in software repositories

• Feng et al. (2022) used a deep neural network to detect 142,479 passwords in 64,045 GitHub repositories; Shi et al. (2025) found 84,491 credential leaks in 413,775 mini-programs; Krause et al. (2023) reported that 30.3% of surveyed developers had encountered a secret leak incident.
• Other leak studies target Android app logs, IoT firmware, WeChat mini-program data flows, browser extensions, and Web APIs.
• The paper notes that these methods target single-language codebases and assume secrets appear as string literals discoverable by regex or neural pattern matching; agent skills break this assumption, as credentials can be exposed through cross-modal channels such as NL instructions and stdout entering the LLM context, which existing detectors do not cover.

Software supply chain security

• Zimmermann et al. (2019) analyzed systemic risks in the npm dependency graph, subsequent work identified attack vectors for Maven and Go, and Torres-Arias et al. (2024) proposed a framework for analyzing supply chain security properties.
• The paper notes that these works share the same threat model—injecting malicious code through dependency resolution in single-language registries; skill registries combine NL and PL and have attack vectors that traditional packages do not.
• Liu et al. (2026) confirmed 157 malicious skills among 98,380 registry entries through dynamic sandboxing; Hu et al. (2026) state that the synthesis cost of malicious tools is as low as $0.013 each. The paper notes that these studies focus on malicious behavior, and whether the skill supply chain facilitates credential theft remains unstudied.

LLM agent ecosystem security

• Prompt injection can manipulate agent decisions (AgentDojo, InjecAgent, etc.), and unconstrained autonomy can cause system damage; Deng et al. (2026) proposed a five-layer lifecycle threat framework, Shen et al. (2025) cataloged 221 vulnerabilities in 50 agent applications, Maloyan et al. (2026) systematized 42 attack techniques against coding assistants, and Jiang et al. (2026) surveyed the full skill lifecycle.
• Defense work includes permission control frameworks, sandbox isolation, reasoning-based guardrails, formal capability analysis, and MCP-specific vulnerability benchmarks.

Positioning of this paper: the authors state that none of the above work examines how the NL+PL skill architecture inherently facilitates credential leaks—existing secret detectors miss cross-modal channels, and supply chain analyses focus on malicious behavior rather than data exfiltration; this paper's empirical study fills this gap. The paper does not set up comparative baseline methods or reuse existing benchmarks.

The overall approach is a four-stage empirical study of 17,022 skills sampled from SkillsMP: dataset collection, static filtering, dynamic validation, and manual classification (Figure 2). The authors state that the first three stages are evidence-collection infrastructure, and the final labels are determined through joint interpretation in the manual classification stage.

Dataset collection

• Source and scale: a snapshot of all active and historical skills on the SkillsMP platform as of February 12, 2026, totaling 170,226; a random 10% sample (17,022) was taken for in-depth analysis.
• Sampling rationale: the authors state that according to the Cochran formula (finite population correction, p = 0.5), 17,022 exceeds the minimum sample size required for a 99% confidence level and 1% margin of error; full analysis is infeasible because dynamic validation requires executing each one in a dedicated sandbox.
• Data composition: two types of artifacts were collected for each skill—source code files (37,409 in total) and NL descriptions (17,022 in total, including prompts, manifests, SKILL.md, etc.).

Static credential filtering

• Keyword matching: using an existing credential leak taxonomy, three authors with security backgrounds first conducted a pilot study on 150 random skills to calibrate the keyword list (Table 1, 9 credential categories); 4,127 skills were flagged in the NL stream and 6,893 in the source code stream, with 2,341 flagged by both paths.
• NL semantic constraint analysis: within a three-sentence sliding window at keyword hits, three types of constraints were checked—co-occurrence of credential words and action verbs, prompt injection patterns, and social engineering phrasing; 309/4,127 were retained.
• Source code AST analysis: non-executable regions such as comments, docstrings, and example placeholder blocks were first removed, then AST parsing was performed with tree-sitter (supporting Python and JavaScript), with function-scope resolution and sensitive sink detection (three types: network transmission, logging, and file I/O, ranked by severity, with network transmission highest); 2,958/6,893 were retained. Merging the two paths yielded 3,156 candidate skills.

Dynamic validation

• Execution environment: each candidate skill was run by a Claude Code agent instance with default settings in an isolated Docker container (Ubuntu 22.04, Python 3.11, Node.js 20), equipped with tcpdump, strace, auditd, and stdout/stderr capture, limited to 2 GB memory and a 120-second timeout.
• Mock credentials: mock credentials with unique markers were injected via three channels following real deployment practices—environment variables, configuration files referenced by the skill manifest, and authentication parameters declared in documentation—so that leaks could be attributed to a specific skill and execution round.
• Dual-condition differential testing: the benign condition (B) executes according to the workflow declared by the skill, and the adversarial condition (A) embeds an adversarial payload in external content consumed by the skill; 3 rounds are executed per condition. The decision threshold is B ≥ 2 or A ≥ 1; the authors state that the stricter benign threshold filters nondeterministic artifacts, while the looser adversarial threshold reflects that a single successful exploitation constitutes a real vulnerability. This stage retained 1,427 skills.

Manual classification and taxonomy generation

• Classification protocol: three authors independently reviewed all 1,427 skills, first reading the documentation to understand the declared functionality, then assigning Benign / Vulnerable / Malicious labels against the (B, A) execution profiles; the three behavioral profiles were structurally classified: attack-induced (B ≤ 1 ∧ A ≥ 1, all 237 judged Vulnerable), dual-trigger (B ≥ 2 ∧ A ≥ 1, yielding 19 Vulnerable and 72 Malicious), and baseline-only (B ≥ 2 ∧ A = 0, judged by intent matching as 907 Benign, 181 Vulnerable, 11 Malicious).
• Agreement: pairwise average Cohen's κ = 0.88 (N = 1,427), with disagreements resolved through discussion with a third person; the final confirmation was 437 Vulnerable, 83 Malicious, and 907 Benign.
• Taxonomy generation: deductively anchored to CWE-798 and CWE-200, and inductively supplemented with agent-specific attack vectors through open coding; after three rounds of axial coding the codebook converged (weighted κ = 0.82), yielding 4 developer negligence patterns and 6 malicious attack patterns.

Experimental setup

• Study subjects: 17,022 skills from a stratified random sample of the 170,226 skills on the SkillsMP platform (including 37,409 source code files and 17,022 NL descriptions), organized around RQ1 popularity, RQ2 leak patterns, and RQ3 exploitability.
• Execution environment: each candidate skill is run by a Claude Code agent with default settings in an isolated Docker container, monitoring network I/O, system calls, file auditing, and stdout/stderr; 2 GB memory, 120-second timeout.
• Test protocol: benign and adversarial dual conditions, 3 rounds per condition; the leak metric threshold is B ≥ 2 or A ≥ 1.
• Review method: three authors with security research experience manually classified all 1,427 dynamically flagged skills, with an average Cohen's κ = 0.88; taxonomy coding converged after three rounds, with weighted κ = 0.82.
• Metrics: number and proportion of affected skills, number of security issues, distribution by pattern/channel/lifecycle, exploitability proportion; the paper does not set up baseline methods for comparative experiments.

Main results

RQ1 overall results (according to Section 4.1 and Table 2):

Dimension	Value
Skills containing credential leaks	520 / 17,022 (3.1%)
Total security issues	1,708 (median 2 per skill, mean 3.28)
Developer negligence (Vulnerable)	437 (84.0%)
Deliberately malicious (Malicious)	83 (16.0%)
Functional categories with most leaks	Web Scraping 17.1%, Data Processing 14.6%, API Integration 13.1%
Language distribution	Python 60.0%, JavaScript/Node.js 27.5%, TypeScript 7.9%
Attack surface	Code+NL 76.3%, Code only 20.6%, NL only 3.1%

• The authors state that 76.3% of cases can only be discovered by jointly analyzing the NL description and the program logic, that neither modality alone can reveal the exposure, and that 3.1% are exploited via pure natural language prompt injection without executable code.
• The authors state that 72% of hardcoded credential cases carry AI-assisted development signatures (references to Copilot, Claude, or ChatGPT in GitHub commit messages), and believe code generation tools are spreading insecure patterns at scale.

RQ2 leak pattern classification (according to Table 3):

Swipe horizontally to read the table

Category	Pattern	Number of issues (within-group proportion)	Skills involved
Vulnerable (1,371 issues)	Information Exposure	1,007 (73.5%)	352
	Hardcoded Credentials	249 (18.2%)	107
	Insecure Storage	110 (8.0%)	77
	Artifact Leakage	5 (0.4%)	5
Malicious (337 issues)	Remote Exploitation	176 (52.2%)	55
	Defense Evasion	116 (34.4%)	34
	Credential Compromise	28 (8.3%)	16
	Data Exfiltration	12 (3.6%)	10
	Resource Hijacking	4 (1.2%)	4
	Persistence	1 (0.3%)	1

• The authors state that Information Exposure arises from the agent framework injecting stdout into the LLM context, making credentials output by print/console.log retrievable via natural language queries; the distribution of hardcoded credential types is API keys 29.7%, tokens 24.5%, passwords/secrets 19.7%, OAuth 8.8% (Figure 3).
• The authors state that 37.3% of malicious skills combine multiple attack patterns, and the most common chain is first using Defense Evasion such as Base64 to bypass static scanning, then triggering Remote Exploitation to establish a reverse shell or RCE backdoor.

RQ3: Exploitability

• What was measured: the runtime leak channels and lifecycle stages of the 520 affected skills in the sandbox.
• Results: 466/520 (89.6%) leak via at least one runtime channel during regular execution without requiring elevated privileges; the remaining 54 contain hardcoded credentials but do not leak at runtime. Channel distribution (Figure 4): stdout leaks 394 (75.8%), file exposure 97 (18.7%, static source code 54 plus unprotected storage 43), active exfiltration to attacker endpoints 68 (13.1%). 92.5% (481/520) become exploitable at the execute stage.
• Authors' interpretation: credential leaks persist across the fork network—after 107 upstream repositories deleted hardcoded credentials, the same credentials remained active in 50+ independent forks; after bybit-trading was taken down, its payload continued to propagate through the fork network, and the authors state that a single C2 payload had spread to 330 files in 70 repositories using 15 different skill names.

Responsible disclosure and remediation

• What was measured: the handling results after reporting all 520 affected skills to the SkillsMP platform.
• Results: the platform confirmed and initiated remediation within 48 hours; all 83 malicious skills were permanently removed, and 91.6% of hardcoded credential cases have been fixed by developers.

Other ablations and analyses

• Static filtering retention: NL semantic constraint analysis retained 309/4,127; AST analysis retained 2,958/6,893; after merging, 3,156 candidates (198 NL only, 2,847 code only, 111 both paths).
• Dynamic validation flagging rate: 1,427 of the 3,156 candidates showed at least one suspicious runtime indicator.
• Manual classification output: of the 1,427, 437 Vulnerable, 83 Malicious, 907 Benign.
• Window size choice: the authors state that a two-sentence window would miss cross-sentence credential-action references, and larger windows introduce spurious co-occurrence with irrelevant context, so three sentences were used.

Limitations and future directions noted by the authors

• Subjectivity of intent classification (Section 7, Construct Validity): judging whether a credential leak is accidental or deliberate is inherently subjective; although three people independently labeled and resolved disagreements through discussion (κ = 0.88), the taxonomy may still miss patterns beyond the CWE/CAPEC anchors.
• Static analysis coverage (Section 7, Internal Validity): AST credential-flow analysis covers only the intraprocedural level for Python and JavaScript; other languages (such as Bash and Ruby) degrade to keyword matching and do not track propagation across variables.
• Dynamic analysis coverage (Section 7, Internal Validity): LLM nondeterminism and dormant logic may prevent the sandbox from reaching every conditional path; the dual-condition testing with 3 rounds per condition can only narrow, not eliminate, this gap.
• External validity (Section 7, External Validity): all data come from the single SkillsMP platform, and popularity on other platforms (such as ClawHub and Skills.sh) may differ; the authors also believe the reported vulnerabilities stem from agent framework design rather than a single marketplace.
• Open problems (Section 8, Conclusion): the authors note two open problems—credential redaction before stdout enters the context, and automated detection that jointly analyzes natural language and code.

Scope of experimental coverage

• The data source is a 10% sample (17,022) of 170,226 skills from the single SkillsMP platform, with a snapshot date of February 12, 2026 (Section 3.1).
• The runtime for dynamic validation is a Claude Code agent instance with default settings, with containers limited to 2 GB of memory and a 120-second timeout, executing 3 rounds per condition (Section 3.3).
• The adversarial condition tests only indirect prompt injection embedded in external content consumed by the skill; the authors explicitly did not cover skills with operator prompts that directly force exfiltration (Section 3.3).
• The scale of manual classification is 1,427 skills and three reviewers, with a pairwise average κ = 0.88; the paper does not report agreement between reviewers and external annotators.
• The paper does not report the time or computational cost of each stage of static filtering and dynamic validation.

A large-scale empirical study of credential leaks in LLM agent skills (ASE '26), which the authors describe as the first systematic study in this direction.

• Problem: skills are hybrid artifacts of NL descriptions plus executable code, running locally with the full privileges of the host process; how credentials leak through this ecosystem had not been systematically studied before.
• Method: sampled 17,022 of the 170,226 skills on SkillsMP, sequentially performing keyword and AST static filtering (3,156 candidates), dual-condition dynamic testing in a Claude Code sandbox injected with mock credentials (1,427 flagged), and three-person manual classification (κ = 0.88), deriving a taxonomy of 4 negligence patterns plus 6 malicious patterns.
• Popularity: 520 skills (3.1%) contain 1,708 security issues; 84.0% stem from developer negligence, 16.0% are deliberately malicious.
• Cross-modality: 76.3% of cases require joint analysis of NL and PL to discover, and 3.1% are exploited via pure natural language prompt injection; the authors state that existing static analyzers do not handle this characteristic.
• Dominant channel: debug logging (print/console.log) accounts for 73.5% of vulnerability issues, which the authors attribute to agent frameworks injecting stdout into the LLM context; 72% of hardcoded cases carry AI-assisted development signatures.
• Exploitability and persistence: 89.6% (466/520) are exploitable during regular execution without requiring elevated privileges, and 92.5% are concentrated at the execute stage; credentials deleted upstream remain active in 50+ forks. After disclosure, all 83 malicious skills were taken down, and 91.6% of hardcoded cases were fixed. The authors note two open problems: credential redaction in the stdout pipeline and cross-modal automated detection.