When skills tell agents not to ask: a large-scale study of oversight instructions in agent skills

Core problem: agent skills contain directives that let agents bypass human oversight (oversight-loosening directives), but their scale, distribution, spread, and detectability have not been studied before.

• Setting and importance: skills are reusable instruction packages centered on SKILL.md, widely shared and copied; the authors note that a single line of instruction can change the oversight settings of every agent that loads it, and an agent told "don't ask" will also not stop to ask when facing a misread task or a malicious webpage.
• Gaps in existing work: the authors state that prior security research focuses on malicious skills, vulnerabilities, and credential leaks, and scanners ask "does it contain an attack", whereas loosening directives contain no attack content; studies of instruction files such as AGENTS.md and Cursor rules classify by topic and do not ask about the effect of instructions on oversight.
• Core observation: oversight language has polarity—the same phrase can loosen or tighten oversight in different positions, and simple keyword matching cannot distinguish them; loosening directives are not necessarily malicious themselves, and most merely keep workflows from being interrupted.
• The paper proposes: the concept of loosening/tightening oversight directives, a codebook classifying by polarity, mechanism, and scope, 2,097 manually annotated units, a detector, and a measurement of 1.53M English skills and an audit of two open-source scanners.
• Research questions: RQ1 prevalence and mechanisms; RQ2 which types of skills are more likely to contain loosening directives; RQ3 how loosening directives spread through copying; RQ4 how well existing scanners detect them.

Supervision mechanisms in agents

• Parasuraman et al. (2000) and Sheridan and Verplank (1978) — treat automation as a continuum of human-machine control; Feng et al. (2025) define agent autonomy levels by user role.
• Shavit et al. (2023) — recommend requiring human approval for high-impact or irreversible actions; coding agents implement these practices with approval prompts, permission modes, allow/deny lists, and flags that suppress prompts (Liu et al. 2026 analyze the Claude Code design space).

Safety analysis of agent skills

• Liu et al. (2026) — 26.1% of 31,132 marketplace skills contain at least one potential vulnerability pattern.
• Liu et al. (2026, another paper) — confirmed 157 malicious skills among 98,380 registered skills.
• Holzbauer et al. (2026) — ran multiple scanners on 238,180 skills and found scanners rarely agree with each other, and most flags do not hold up after contextual review; open-source scanners combine static rules with optional LLM analysis (Cisco skill-scanner, NVIDIA SkillSpector).

Empirical studies of agent instruction files

• Jiang and Nam (2026) — classified Cursor rules into conventions, guidelines, project information, LLM instructions, and examples, and found 28.7% of rule lines are duplicated.
• Chatlatanagulchai et al. (2026) — classified instructions in 2,303 context files such as CLAUDE.md and AGENTS.md.
• The authors note that these studies classify instructions by topic, and none asks whether instructions relax or tighten human supervision.

Baselines and comparison objects: The baselines for RQ4 are keyword lexicon matching, GPT-5.4 with this paper's codebook, and this paper's detector; the audited scanners are Cisco AI Defense skill-scanner 2.2.1 and NVIDIA SkillSpector 2.12.0 (each run in offline and LLM modes). The dataset is GitSkills (Destefanis et al. 2026).

The authors position this paper as: proposing a class of risk distinct from malicious content (supervision-relaxing instructions) and providing the first large-scale measurement (the authors call it "the first large-scale study").

Overall approach: screen English skills from the GitSkills dataset, split skill bodies into units, retrieve candidate units with a regex vocabulary, develop an oversight-directive codebook and manually annotate, train a detector to annotate all 1.37 million candidate units, supporting four research questions respectively.

Data and candidate units

• Analysis population: GitSkills contains 3,797,117 SKILL.md files, deduplicated by content hash to 1,877,981; after four filters (is a skill, not earlier than October 2025, not a symlink placeholder, English) 1,527,281 distinct skills and 3,301,422 occurrences remain.
• Unit splitting: skill bodies are split into sentences, list items, table cells, or code lines, and each unit is read with context (section heading and preceding/following units).
• Vocabulary retrieval: 36 regex patterns (e.g., "without asking", --no-verify) are used as a filter; a match makes a candidate unit, 1.37 million in total; the vocabulary only retrieves and does not judge.

Codebook and manual annotation

• Four-level labels: type (directive D / mention M / neither N), polarity (loosening L / tightening T), mechanism (nine categories, Table 1: PERM, CONF, PERSIST, SAFE, AUTO, TRANS, ACT, INVOKE, PREC), scope (loosening directives only: conditional or unscoped). The criterion is whether deleting the unit would change human control over the agent.
• Development process: the codebook went through three versions v0–v2; v2 requires annotators to answer three questions first before applying the detailed rules; 12 additional decision rules handle disputed cases.
• Annotation sets: six sets totaling 2,097 distinct units; 1,600 stratified-sampled units form the prevalence sample, and the 500 most-copied units form the census. The first author annotated or reviewed all units, Claude Opus 5.5 as an independent second annotator annotated 1,000 of them, and all disagreements were adjudicated by the first author; on the gold set the three-class label κ=0.82 (Table 2).

Detector

• Combining two classifiers: fine-tuned RoBERTa-base (two heads: three-class label + mechanism, averaged over three random seeds) and Jev 1.13 (no fine-tuning, two multiple-choice questions, probabilities reweighted as neither 1.0 / loosening 0.25 / tightening 0.75), averaging the two probabilities, with mechanism taken from Jev.
• Performance: on 400 gold units, accuracy 0.80, κ=0.65, loosening F1 0.65 (precision 0.70, recall 0.61) (Table 3); the authors state it is comparable to GPT-5.4 with the codebook but more than ten times cheaper.

Analysis methods for the four RQs

• RQ1: weighted estimation of unit-level prevalence from the prevalence sample (95% bootstrap CI), with the detector giving skill-level estimates; the census supports copy-count-weighted estimates.
• RQ2: three logistic regressions (M1–M3) fitted on 512,876 candidate skills, logit P(Y_s=1)=α+β^⊤ c_s+γ^⊤ r_s+δ^⊤ z_s, where Y_s is whether the skill contains loosening (M1, M2) or tightening (M3) directives, c is nine capability features (shell code blocks, package installation, sudo, curl | sh, etc.), r is repository context (stars, whether copied), z is control variables (log of length and candidate unit count), and standard errors are clustered by repository.
• RQ3: MinHash estimates Jaccard similarity of 5-word sequences ≥0.7 to link near-duplicate skill families (118,076 families), measuring loosening concentration; the earliest committer is treated as the ancestor, comparing the share of loosening among units added and removed in variants.
• RQ4: on 400 gold units (61 loosening directives), two scanners (offline/LLM modes) and three baselines are evaluated, with metrics precision, recall (95% CI), F1, and the proportion of falsely flagged tightening directives (%T).

Experimental setup

• Data: 1,527,281 distinct English skills in GitSkills (3,301,422 occurrences); 1.37 million candidate units.
• Manual annotation: 2,097 units; 1,600-unit prevalence sample + 500-unit census; a 400-unit gold set (61 loosening directives) used to evaluate the detector and scanners.
• Models: Claude Opus 5.5 as second annotator; the detector is a RoBERTa-base + Jev 1.13 combination; GPT-5.4 and GPT-5.4-mini as LLM classifier comparisons; scanner LLM mode integrated via GPT-5.4.
• Audited scanners: Cisco AI Defense skill-scanner 2.2.1 and NVIDIA SkillSpector 2.12.0, each run in offline and LLM modes; a unit is counted as flagged if the line has a finding of medium severity or above or a finding references that unit.
• Metrics: prevalence (95% bootstrap CI, 2,000 resamples), odds ratio (95% CI clustered by repository), precision/recall/F1, Cohen's κ, McNemar test.
• RQ2 sample: 512,876 candidate skills, of which 17.2% contain loosening directives and 50.4% contain tightening directives (detector reference).

Main results

RQ1 prevalence (Table 4, candidate-unit level) and RQ4 scanner performance (Table 6, gold set of 400 units, 61 loosening):

Metric	Value	Source
Candidate units that are directives (manual)	49.7% [46.8, 52.6]	Table 4
Share of loosening directives (manual / detector)	13.3% [11.7, 15.0] / 10.3%	Table 4
Share of tightening directives (manual / detector)	36.4% [33.7, 39.2] / 39.3%	Table 4
Share of loosening that is unscoped	70.1% [63.9, 75.8]	Table 4
Skills with ≥1 loosening directive	5.8% (about 88,000)	§4.1 Finding 1.5
Skills with ≥1 tightening directive	16.9%	§4.1 Finding 1.5

Swipe horizontally to read the table

RQ4 method	Prec.	Recall	F1	%T
Keyword vocabulary	0.20	0.82	0.33	24%
Cisco offline	0.14	0.02	0.03	29%
Cisco + LLM	0.39	0.15	0.21	13%
SkillSpector offline	0.21	0.21	0.21	29%
SkillSpector + LLM	0.27	0.46	0.34	23%
Either + LLM	0.28	0.51	0.36	23%
GPT-5.4 + this paper's codebook	0.71	0.61	0.65	4%
This paper's detector	0.70	0.61	0.65	4%

(According to Table 6; 95% CIs for recall are in the original table)

• The authors state that tightening is 2.7 times loosening, but loosening is not rare; loosening is mainly achieved through three mechanism categories: permission flags (33.9%), skipping confirmation (27.4%), and not stopping (16.0%) (Table 1).
• The authors note that the detector's loosening recall (0.61) is lower than its precision (0.70), and 5.8% is more likely an underestimate.
• The authors state that scanners mistake guardrails for risk: 23%–29% of flags fall on tightening directives, whereas this paper's detector has only 4%; the same LLM asked oversight questions reaches F1 0.65, statistically indistinguishable from the detector (p=1.0).

RQ2: which types of skills are more likely to contain loosening directives

• What was measured: the association of nine capability features with containing loosening/tightening directives, controlling for skill length and candidate unit count (n=512,876, Table 5).
• Results: the loosening ORs for shell code blocks, package installation, and sudo are 1.74–1.82, curl | sh is 1.31, network calls 1.19; skills referencing secrets have OR 0.77 (lower); destructive commands show no association (OR 0.98). On the tightening side: package installation, sudo, and curl | sh have ORs of 0.59–0.77 (less tightening), with git write as an exception (OR 2.01, more tightening).
• Authors' interpretation: under equal amounts of oversight language, execution-type skills are more likely to loosen oversight and do not compensate by increasing tightening; stars are unrelated to loosening, and skills copied to ≥2 repositories have loosening OR 1.13; 12,484 skills (0.82%) contain both loosening directives and destructive commands/curl | sh/sudo.

RQ3: how loosening spreads

• What was measured: whether copying amplifies loosening, the concentration of loosening in skill families and phrasings, and variants' edits to oversight text (MinHash near-duplicate families, threshold 0.7, repeated at 0.6/0.8).
• Results: copying does not amplify loosening (5.8% of distinct skills vs 5.4% of occurrences, ratio 0.94); the top 100 families carrying the most loosening account for 23% of occurrences of loosening-carrying skills and only 1.4% of all skill occurrences; 960 distinct loosening units (1.1% of 86,048) account for half of all loosening occurrences; 93.4% of variants do not modify the ancestor's oversight directives (92–97% across settings); among units added in variants, loosening accounts for 20.3%, and among removed, 20.2% (+0.1 percentage points [−5.3, +4.7]).
• Authors' interpretation: loosening mainly spreads through a few upstream skills, and adopters rarely modify oversight text; when the most-copied skills are treated as ancestors, variants add slightly more loosening (+1.4 to +5.5 percentage points, significant in two of five settings), which the authors describe as at most a small shift toward loosening.

Other ablations and analyses

• Vocabulary recall audit: among 250 random skills, the vocabulary is estimated to capture 32% [14%, 75%] of loosening directives and 16% [10%, 25%] of tightening directives (§6), so the authors state that prevalence estimates are lower bounds and the 2.7:1 ratio is conservative.
• Persistence mechanism definition: when counting only "not stopping" that skips human check-ins, loosening prevalence is 12.7% [11.0, 14.4] rather than 13.3% (§6).
• RQ2 robustness: across five refits, shell, package installation, and sudo remain significant, and curl | sh is significant except in the high-confidence setting; on the full 1.53M skills, network calls lose significance and the copying direction reverses (OR 0.85).
• Reproducing RQ2 on the prevalence sample: shell, network, package installation, and git write have consistent directions, while secrets, sudo, and curl | sh are inconsistent (the latter two are based on only 39 and 42 units).
• The detector combination on the gold set corrects 43 RoBERTa errors and adds 20 (McNemar p=0.005); units with confidence ≥0.9 (31%) are 97% correct.
• After excluding security-tool repositories and test directories, the loosening share changes only from 13.2% to 13.3%; estimates by annotation set range from 12.5% to 15.6%.

Limitations and future directions noted by the authors

• The second annotator is an LLM (§6): there is no second human annotator, so agreement reflects human vs. model rather than two humans; the pre-annotation set may be anchored by the proposal and is not counted toward reliability.
• Limited vocabulary recall (§6): the vocabulary is estimated to capture only 32% of loosening directives and 16% of tightening directives, prevalence estimates are conditional on candidate units, and absolute counts are lower bounds; RQ2 assumes misses are independent of skill capability, but the recall sample is too small to test this.
• Detector error (§6): skill-level analyses rely on a detector with loosening F1 0.65; RQ3 ancestors are inferred from commit time or copy count rather than observed, and the two ancestor rules disagree on the small shift.
• External validity (§6): GitSkills covers only public GitHub repositories, and private and marketplace skills may differ; only English skills are analyzed; LLM-mode scanning is evaluated only on 400 gold units with a single model, GPT-5.4, and confidence intervals are wide.
• Text only, not behavior (§5): the authors encourage future work to test whether agents follow oversight directives and whether scope clauses change their behavior.
• Future directions (§5): improve vocabulary recall, extend the analysis to other instruction files such as AGENTS.md, and track how loosening directives change over time.

Scope of the experiments

• The data are 1,527,281 distinct English skills from public GitHub repositories in GitSkills; non-English skills are excluded (English accounts for 85.6% of distinct content).
• The audited scanners are two, Cisco skill-scanner 2.2.1 and SkillSpector 2.12.0, selected for being open-source, running locally, including both rule-based and LLM analysis, and reporting by line; other scanners are not included.
• Scanner LLM mode is evaluated only via GPT-5.4 integration; the gold set is 400 units, of which 61 are loosening directives.
• Manual annotation covers 2,097 units in total, all annotated or reviewed by the first author; the paper reports no second human annotator.
• The paper reports no experiment on the actual behavior of agents after loading skills containing loosening directives.

This is a large-scale measurement study of agent skills security, proposing "oversight-loosening directives" as a risk concept distinct from malicious content.

• Problem: Some instructions in skills make agents act without confirmation, skip checks with --no-verify, or not stop to report; they remove human oversight checkpoints, and no prior work has studied their scale and spread.
• Method: Screened 1,527,281 distinct English skills from GitSkills, used 36 regex patterns to retrieve 1.37 million candidate units, developed a codebook classifying by polarity, mechanism, and scope, manually annotated 2,097 units (Claude Opus 5.5 as second annotator, gold set κ=0.82), trained a RoBERTa+Jev detector (loosening F1 0.65) to annotate the full set.
• Main results: Among candidate units, 13.3% loosen oversight and 36.4% tighten oversight; 5.8% of English skills (about 88,000) contain at least one loosening directive, 70.1% apply unconditionally; loosening is mainly achieved via permission flags (33.9%), skipping confirmation (27.4%), and not stopping (16.0%).
• Risk context: Under equal amounts of oversight language, skills containing shell code blocks, package installation, or sudo have 1.7–1.8 times the odds of containing loosening directives, and do not compensate by increasing tightening; 12,484 skills contain both loosening directives and destructive commands, curl | sh, or sudo.
• Spread: Copying does not amplify loosening; 2.4% of loosening-carrying skill families account for half of the copy volume of loosening-carrying skills, and 1.1% of loosening phrasings account for half of loosening occurrences; 93.4% of variants do not modify oversight text.
• Detection: Two open-source scanners recall 2%–28% offline, 15%–46% with an LLM, with precision below 0.4, and 23%–29% of flags falsely hit tightening directives; the same GPT-5.4 asked with this paper's codebook reaches F1 0.65.
• Authors' conclusion: Scanner misses are not because the model cannot recognize it, but because it was not asked the right question; the authors recommend that skill authors add scope clauses, vendors warn at install time about unconditional loosening, fixes target the few upstream skills, and provide the codebook, annotations, and detector for community use.