Skip to content

SkillScreen: high-recall screening of malicious Agent skills with a decision model

SkillScreen: High-Recall Screening of Malicious Agent Skills with Decision Models

Paper Tracker· arXiv:2610.17727· Jerrold Huang, Junwei Quan, Saiyue Lyu, Zihan Guo, Yingqiang GeUniversity of Toronto; University of British Columbia; Rutgers UniversityIndexed by this site · 2026-10-20· Originally publishedDefenses

Quick read

Defenses

Based on the paper PDF (AI-generated); refer to the original

SkillScreen uses a 2B model to screen skills; on unseen attack families in MSD, the FPR at 99% test recall is 22.6%, versus 45.6% for Word-SVM.

Threat modelAn attacker publishes a skill for users to install. SkillScreen targets attacks expressed directly in SKILL.md, or through the initialization steps it specifies; behavior that exists only in bundled code requires package-level analysis. The detector scores the Markdown before the skill is used, and high scores go to human review. Evaluation uses fixed rewrites rather than adaptive attacks.

Questions
Agent skills in public registries turn instructions into actions, and legitimate automation and malicious behavior look similar. Screening should miss as few as possible while keeping false positives within a reviewable range. The authors state that static scanning has low recall, and that prompt review mislabels at least 87% of benign skills at 99% recall.
Methods
SkillScreen treats SKILL.md as the premise, pairs it with a fixed malicious hypothesis, fine-tunes OpenJev-2B with LoRA, and outputs a continuous maliciousness score in a single forward pass. On the validation set, it selects the highest threshold that reaches 99% recall; before testing, the checkpoint, calibration and threshold are frozen.
Experiments and results
On unseen attack families, SkillScreen has a default recall of 95.6% and FPR 3.3%, and at 99% recall an FPR of 22.6%, lower than Word-SVM's 45.6%. After switching to unseen sources, that FPR is 50.5%, and ModernBERT-large's is 74.7%. The authors say unfamiliar legitimate sources raise false positives.
Limitations and future work
The authors state that it reads only SKILL.md and uses fixed rewrites rather than adaptive attacks; after commands are moved out of the body, the frozen gate's recall drops from 99.8% to 87.0%. False positives rise when the legitimate test sources differ from the training sources. At about 21% FPR, precision is 34% when the malicious share is 10% and 4.5% when it is 1%.
Experimental setupSkillScreen · MaliciousSkillBench、Malicious-Structural-Disjoint (MSD) et al. · Recall、FPR et al.
Evaluated models
SkillScreen
Benchmarks
MaliciousSkillBenchMalicious-Structural-Disjoint (MSD)Source-Disjoint (SD)Source-Balanced Random (SBR)
Metrics
RecallFPRAUROCAPFPR at 99% recall
AI summarySkillScreen is a compact decision model that directly outputs a maliciousness score for a skill from a single forward pass over SKILL.md, achieving 99% recall on unseen attack families on MaliciousSkillBench. At this recall level, it mislabels 20-25% of benign skills, fewer than the word-level classifiers and prompting LLM judges tested in the paper. Under the same training protocol, it also generalizes better to unseen sources than a fully fine-tuned encoder. But against unfamiliar benign sources, false positives rise markedly, and the authors accordingly note that identifying new attacks does not equal reliably screening new skill sets. The paper is 21 pages and classified under cs.CR.

SkillScreen is a compact decision model that directly outputs a maliciousness score for a skill from a single forward pass over SKILL.md, achieving 99% recall on unseen attack families on MaliciousSkillBench. At this recall level, it mislabels 20-25% of benign skills, fewer than the word-level classifiers and prompting LLM judges tested in the paper. Under the same training protocol, it also generalizes better to unseen sources than a fully fine-tuned encoder. But against unfamiliar benign sources, false positives rise markedly, and the authors accordingly note that identifying new attacks does not equal reliably screening new skill sets. The paper is 21 pages and classified under cs.CR.

Related events and news

1 events · 1 news mentions

News that mentions this paper

RL/RLHF 训练条件可诱发模型智能体提权逃逸

Paper analysis

6 questions; open one to read the full answer

  1. What problem does this paper attempt to solve?

    Legitimate and malicious skills look similar, so screening needs high recall and low false positives.

    Agent skills in public registries should be screened before installation: miss as few malicious skills as possible while sending few legitimate skills to human review.

    • Scenario: skills use SKILL.md to turn instructions into file, code or service operations, forming a supply chain through public registries. The authors say existing skills steal credentials, execute remotely or manipulate agents, and controlled evaluations also see data exfiltration and destructive actions, so a single missed detection could affect users.
    • Gap in existing approaches: the authors say the static scanner SkillSpector detects only 10.5% of malicious skills on MaliciousSkillBench; prompt judges are expensive to query, conservative, and their scores are too coarse. The introduction says that on the main split, the evaluated judges mislabel at least 87% of legitimate skills at 99% recall, and in Table 1 these judges' FPR is 86.8%–100.0%.
    • The authors' observation: legitimate development tools also run shell, read credentials and access remote services. The authors say the distinguishing point is how these capabilities are used in context; lowering the threshold to catch more attacks also flags more legitimate skills.
    • Proposed screener: SkillScreen uses a small decision model to output a continuous maliciousness score from SKILL.md in a single forward pass, and calibrates a 99% recall threshold on the validation set; it runs locally, and per-item billed APIs are not its intended path.
    • Threat model:
      • Goal: an attacker publishes a skill for users to install. Screening targets attacks expressed directly in the Markdown, or through the initialization steps it specifies.
      • Evaluation method: Limitations state that it targets an attacker who publishes skills, using fixed rewrites rather than adaptive attacks. The paper does not separately describe how the attacker accesses the detector weights.
      • Victim: after the agent loads the skill, it executes instructions with the tools and permissions already granted. Scoring happens before the skill is used.
      • Handling: high scores go to human review. Behavior that exists only in bundled code and is not expressed in the main text, the authors say, requires package-level analysis.
  2. What related research exists?

    Related work falls into four groups: agent attacks, malicious skills, software packages, and decision models.

    Attacks against agents

    • Greshake et al. (2023), Perez and Ribeiro (2022), Liu et al. (2025) study indirect prompt injection: adversarial instructions embedded in content consumed by LLM applications.
    • InjecAgent, AgentDojo and Agent Security Bench (Zhan et al., 2024; Debenedetti et al., 2024; Zhang et al., 2025) test such attacks and defenses through agent interactions.
    • Radosevich and Halloran (2025) discuss similar risks in MCP tool descriptions. The paper here only states that it screens before agents use skills.

    Malicious skills and detection

    • Liu et al. (2026) document credential theft and agent manipulation in public skill registries.
    • SkillInject (Schmotz et al., 2026) evaluates attacks embedded in skill files. MalSkillBench (Guo et al., 2026) verifies malicious behavior at runtime and evaluates detectors across code and instruction attacks.
    • MaliciousSkillBench (Wang et al., 2026) aggregates public sources and provides structurally disjoint and source-disjoint protocols. The authors compare small trained scorers on these protocols.

    Malicious packages and static rules

    • Ohm et al. (2020) survey supply chain attacks distributed through third-party packages; Sejfia and Schäfer (2022) classify malicious packages using metadata and code. The authors state that agent skills also contain instructions interpreted directly by an LLM.
    • SkillSpector (NVIDIA, 2026) uses static rules on skill packages. The authors compare in Table 1: SkillScreen detects 95.6% of malicious skills, static rules detect 10.5%.

    Decision models, NLI and review

    • TypeSafe (2026), Ibrahim and Zaki (2026) discuss decision models that output probabilities for predetermined outcomes. Llama Guard and ShieldGemma use language models for safety screening (Inan et al., 2023; Zeng et al., 2024).
    • Yin et al. (2019), Wang et al. (2021) express labels with hypotheses. SkillScreen uses LoRA (Hu et al., 2022) to apply this interface to skill detection.
    • Zheng et al. (2023), Wang et al. (2024) study prompt review and its biases; Greenblatt et al. (2024), Baker et al. (2025) and others apply similar judgments to oversight. The authors state that, on the same visible skill text, task-trained scoring reaches high attack recall at lower false-positive cost.

    Baselines and benchmark: the comparison baselines are fully fine-tuned ModernBERT-large, word-level TF-IDF linear SVM, un-fine-tuned OpenJev-2B, TypeSafe Jev 1.13, SkillSpector, and GPT-5.5, Grok 4.3, Claude Haiku 4.5, GPT-5.4 mini. The benchmark is MaliciousSkillBench, with protocols MSD, SD, SBR.

    The authors place this work in pre-installation screening: training a small scorer on labeled skills rather than relying only on static rules or prompt review, and comparing false positives at high recall under attack-structure shift and source shift.

  3. How does the paper address this problem?

    SkillScreen fine-tunes OpenJev-2B with LoRA, turning SKILL.md into a maliciousness score with a calibratable threshold.

    SkillScreen formulates detection as natural language inference: SKILL.md is the premise, a fixed hypothesis states whether the skill is malicious, the classification head outputs a continuous score, and a validation-set threshold decides whether it is sent for review. In Figure 1, training and inference use the same skill, hypothesis and model, and the model, calibration and threshold are frozen before testing.

    Data splits

    • MSD (main protocol): malicious structural families do not cross splits. Train 6,814, validation 974, test 1,947 (malicious 1,500, benign 447). The 2,458 training families and 1,364 test families are disjoint, and after whitespace normalization the texts do not repeat across splits either.
    • SD: whole-source holdout. Train 7,510, validation 833, test 1,384 (malicious 839, benign 545), and the test comes from three sources not in training. Thirty malicious families, accounting for 38% of malicious test skills, also appear in training.
    • SBR: each source appears in both training and test. Train 6,812, validation 973, test 1,950. Half of the malicious test skills belong to families seen in training.
    • The benchmark contains eleven sources and 9,740 labeled skills. The experiments exclude five records lacking a public original source, and keep the official labels and splits; SD has eight fewer (Section 2).

    LoRA fine-tuning

    • Initialized from a fixed OpenJev-2B. This checkpoint adapts Qwen3.5-2B to NLI.
    • The backbone is frozen, adapters with rank 8 and scaling factor α=16 are trained on the attention projections, and the classification head initialized from pretrained NLI weights is fully fine-tuned. Trainable parameters 2.19M, and the adapter can be distributed separately from the backbone.
    • One epoch of training on the labeled training set, on one 10 GB GPU. Appendix A gives AdamW, learning rate 2×10−4, weight decay 0.01, gradient clipping 1.0; 426 updates for MSD and SBR, 470 for SD. The authors state that this configuration was not tuned further.
    • Validation scores are checked mid-training and at the end, and the checkpoint with the lowest FPR at 99% validation recall is selected; ties are broken first by higher AP, then by lower loss. In the text, all fifteen current code runs selected the final checkpoint.

    NLI and scores

    • Markdown is the premise, and the fixed hypothesis is This AI agent skill is malicious. The input is the premise followed by the hypothesis, capped at 8,192 tokens; when too long, only the premise is truncated and the hypothesis is kept intact.
    • The head outputs three logits: contradiction, entailment, neutral. Cross-entropy labels benign skills as contradiction and malicious as entailment; no sample targets neutral. The un-fine-tuned checkpoint scores directly with the same input as a zero-shot reference.
    • The detection score is the weight of entailment after softmax over the three logits: s(x)=exp(ze)/(exp(zc)+exp(ze)+exp(zn)). zc, ze, zn are the logits for contradiction, entailment, neutral respectively. The score is continuous, so a threshold can be set for a target recall.
    • The authors state that writing the task into the hypothesis helps at 99% recall (Section 5.4); when only 25%–50% of labels remain, NLI initialization plus a hypothesis naming the task roughly halves false positives (Section 5.5).

    Threshold and review

    • Results are reported both for the raw score at 0.5 and for the separate high-recall threshold. A sigmoid with positive slope is fitted to the validation-set log-odds, and the highest threshold reaching 99% validation recall is taken.
    • The checkpoint, calibration and threshold are frozen before testing. Skills at or above that threshold are sent for human review.
    • A separate default decision rule is used to align with baselines: probability ≥ 0.5, SVM margin ≥ 0, scanner risk > 50.
    • The comparison encoder ModernBERT-large is fully fine-tuned for one epoch, learning rate 2×10−5, input likewise truncated to 8,192 tokens, but with its own tokenizer. The word-level SVM uses 120,000-dimensional word uni/bigram TF-IDF, and margin ≥ 0 is judged malicious.
  4. What experiments did the paper conduct?

    On MSD at 99% test recall, SkillScreen's FPR is 22.6%, lower than Word-SVM and LLM reviewers.

    Experimental Setup

    • Detectors: SkillScreen (OpenJev-2B + LoRA); fully fine-tuned ModernBERT-large (395.8M); Word-SVM; untuned OpenJev-2B; TypeSafe Jev 1.13; SkillSpector 2.12.0. LLM judges are GPT-5.5, GPT-5.4 mini, Claude Haiku 4.5, Grok 4.3. Ablations additionally train from Qwen3.5-2B and OpenJev-0.8B.
    • Data: MaliciousSkillBench. MSD test 1,947 items (malicious 1,500, legitimate 447), test families disjoint from training, of which 90% are singleton families. SD test 1,384 items (839 / 545). SBR test 1,950 items. Source counts in Table 3.
    • Metrics: malicious recall, legitimate FPR, AUROC, AP. The shaded columns in Table 1 are the FPR at which each method reaches 99% recall on test scores. The high-recall threshold actually used for decisions is selected on the validation set and then frozen; the two are not the same operating point.
    • Labels and judging: official labels are used; no separate judge model scores the detectors. The judge baselines see the same visible prefix, share a prompt that distinguishes malicious behavior from authorized development or security processes, output a boolean conclusion and a JSON probability, with probability ≥ 0.5 judged malicious, once per item. GPT-5.5 additionally has eight-shot and medium-reasoning conditions. Appendix N gives the request setup. Grok 4.3 is accessed via a reseller; requests set reasoning effort to none, and replies still carry reasoning tokens. Grok's boolean conclusion and probability disagree on 242 of MSD's 1,947 items and 253 of SD's 1,384 items; the paper uses the reported probability.
    • Repetitions: SkillScreen and ModernBERT use five seeds per split; on SD the encoder uses six seeds, replacing the crashed run with one of them. The 10%, 25%, 50% label conditions each have nine runs. Paired bootstrap 2,000 times, with sample sizes fixed per class. The ModernBERT MSD numbers in Table 1 are a GPU rerun, which the authors say is the stronger of the two five-seed sets of results (Limitations; Appendix H).
    • Input: SkillScreen and the encoders are both truncated to 8,192 tokens. The MSD training set has a median length of 1,367 tokens, with upper-limit truncation of 44/6,814 training skills and 17/1,947 test skills (Appendix A).

    Main Results

    Swipe horizontally to read the table

    MethodMSD recall/FPRMSD@99%FPRSD recall/FPRSD@99%FPR
    SkillScreen95.6±0.6 / 3.3±1.322.697.3±1.1 / 36.0±4.350.5
    ModernBERT-large96.0±0.4 / 4.1±1.830.494.1±4.8 / 54.8±3.874.7
    Word-SVM94.6 / 8.945.695.6 / 62.482.2
    GPT-5.5 zero-shot72.1 / 0.791.142.8 / 0.9100.0
    GPT-5.5 eight-shot87.9 / 2.595.760.4 / 2.279.8
    GPT-5.5 reasoning71.1 / 1.386.833.7 / 1.1100.0
    SkillSpector10.5 / 0.7100.01.3 / 1.8100.0
    Untuned OpenJev-2B4.2 / 0.095.50.6 / 0.099.8

    According to Table 1, units are percentages. The default column is probability ≥ 0.5 (for SkillSpector, risk > 50). @99% is the FPR at which 99% recall is reached on test scores. Values with ± are the mean and standard deviation over five seeds; SD's ModernBERT uses the converged run. Table 1 methods not listed in the table: Claude Haiku 4.5 (MSD @99% FPR 92.6), Grok 4.3 (96.6), GPT-5.4 mini (100.0), TypeSafe Jev 1.13 (100.0). The lowest default recall on MSD among those in the table is Untuned OpenJev-2B.

    • The authors argue task training turns an NLI model into a detector. The five SkillScreen seeds have recall at 0.5 of 95.1%–96.5%. Compared with Word-SVM, FPR is 5.6 percentage points lower on the same resampled records ([−8.5, −3.0]), the recall difference is +1.0 ([−0.1, +2.1]), and the AUROC difference is +0.015 ([+0.010, +0.021]).
    • Near the default threshold it is on par with ModernBERT: when recall is aligned to 95.6%, the paired FPR difference is 0.0 ([−1.8, +2.0]). At 99% test recall, every SkillScreen seed's FPR is lower than every encoder run (seed range 19.9%–24.6% vs 25.1%–36.2%). The paired difference relative to the encoder is +7.8 points ([−3.4, +22.1]), and the record-level interval still contains 0 on MSD; relative to Word-SVM it is +23.0 points ([+6.8, +39.8]).
    • On SBR, whether the family was seen barely changes recall: at 0.5, seen families 98.8%, unseen families 97.4% (five seeds, Appendix K). Table 5: when MSD malicious proportion is 77%, SkillScreen's AP is 0.998, Word-SVM's is 0.993, and GPT-5.5 zero-shot's is 0.983.

    Frozen Threshold and Judge Operating Points

    • The threshold selected and frozen on the validation set for 99% recall gives 98.7%–99.1% recall and 20.1%–23.9% FPR on the MSD test set. ModernBERT's average FPR at 99% recall on the validation set is 27.7% (SkillScreen 22.8%), and on test 98.5%–99.1% recall and 25.1%–38.0% FPR; Word-SVM is 99.3% recall and 60.6% FPR (Appendix G, H).
    • On SBR, average FPR at 99% test recall: SkillScreen 6.6% ([2.4, 13.5]), ModernBERT-large 12.1% ([6.2, 25.5]), paired difference +5.4 ([−0.1, +15.5]). At the frozen gate, SkillScreen is 98.7%–99.5% recall and 2.9%–9.8% FPR (Appendix I).
    • The authors say prompt judges have no usable 99% recall operating point: probabilities take only 13–60 distinct values. Table 4: when aligned to 95.6% recall, SkillScreen's FPR is 2.0%–4.0%, Word-SVM's is 13.6%, and GPT-5.5 zero-shot's is 34.7%. After eight-shot raises GPT-5.5's default recall, it still requires 28.2% FPR at SkillScreen's average recall; after medium reasoning its default recall is 71.1% (Section 5.2).

    Unseen Sources and Errors

    • On SD, SkillScreen has 97.3% recall at 0.5, and 98.9%–99.6% at the frozen gate. The FPR at 99% test recall is 50.5% (seeds 46%–56%), and the paired difference relative to the converged encoder is +24.1 points ([+14.4, +32.7]). The authors say GPT-5.5's AUROC on SD (zero-shot 0.914, eight-shot 0.941) is close to SkillScreen's 0.916, but at 95.6% recall, zero-shot and eight-shot flag 78.5% and 65.7% of legitimate skills respectively, while SkillScreen flags 24%–33% (Appendix J).
    • False positives are concentrated in two legitimate test sources that both differ from the training sources. For the synthetic template SkillFortifyBench, legitimate and malicious variants differ only in the command: every SkillScreen seed and the converged encoder flag all 90 legitimate and 90 malicious variants, while GPT-5.5 flags none of the legitimate variants. ATR's 455 legitimate skills describe security tools; the main text says SkillScreen flags 17%–30% at 0.5, the encoder 46%, and GPT-5.5 1.1% (Appendix M). The five-seed mean in Figure 7 is 23.3.
    • The authors argue the shift is in the legitimate distribution: validation thresholds are selected at 8%–11% FPR on legitimate skills from training sources, and 165 of 169 come from SkillTrustBench, which is absent from the test set; these thresholds flag 56%–64% of legitimate skills on unseen test sources. On MSD, validation and test FPR are both about 20%–28% and 20%–24%. Figure 2: on MSD, missed detections are concentrated in small sources, with SkillLeakBench at 73.3% (n=9), AgentTrap at 44.5% (n=22), and MalSkillBench at 2.9% (n=967). The five seeds jointly miss 42 malicious skills and jointly flag 4 legitimate skills; the authors believe some jointly missed Markdown contains no malicious evidence, and the labels may come from package code outside the main text.

    Other Ablations and Analyses

    • Scanner context (Table 6, MSD): findings-only recall 95.4%, FPR 1.3%; full context 95.1%, 2.5%. The paired interval relative to seed 42 contains 0. Scanning on the MSD test set takes 817 more seconds.
    • Initialization and hypothesis (Table 2, probability 0.5): plain Qwen3.5-2B five-seed recall 96.9±0.8%, FPR 4.1±1.6%, test-scan FPR@99 21.1; irrelevant hypothesis 28.9; null hypothesis single seed 44.7, paired difference +22.1 ([+2.0, +40.4]). OpenJev-0.8B at 0.5 is 95.4% / 2.9%, with a high-recall threshold of 30%–34% FPR, and 2B is 20%–24% (Section 5.4; not the same operating point as Table 2's test-scan 26.0).
    • Few labels (Figure 3, Section 5.5): at 10%, task hypothesis AUROC 0.850±0.043, irrelevant hypothesis 0.755±0.055, higher in all nine paired runs (sign test p=0.004); relative to plain base (0.807±0.058), eight of five pairs, p=0.73. At 25% and 50%, FPR at 0.5 is 7.8% and 3.1%, plain base is 19.2% and 6.7%, and irrelevant hypothesis is 16.1% and 6.2% (Appendix A). At 10%, the nine runs flag 89%–96% of legitimate skills at the frozen gate.
    • Rewriting (Table 12, 200 malicious MSD test skills, mean over four adapters): after paraphrasing, 0.5 recall goes from 96.6% to 96.0%, and the frozen gate from 99.8% to 99.6%; after moving the command out of SKILL.md, 0.5 goes from 96.6% to 67.5%, and the frozen gate from 99.8% to 87.0%.
    • Combination and low-prevalence precision (Appendix K): frozen-gate positives passed to GPT-5.5 give SD recall of 42.7%. On MSD, threshold precision at malicious proportions of 1% and 10% is 4.5% and 33.9%.
    • Negative: the default recipe encoder crashes twice in sixteen total runs on MSD and SD (Appendix H). When Table 10 counts all six SD runs, the 99% recall FPR is 78.2%, rather than Table 1's converged-run 74.7%. At the validation-set 99% recall threshold, commercial NLI APIs and static scanners flag all MSD records (Appendix G).
  5. What are some points that can be further explored?

    The authors point out that reading only SKILL.md and not using adaptive attacks, unfamiliar legitimate sources raise false positives.

    Limitations and future directions noted by the authors

    • It reads only SKILL.md, the evaluation targets attackers who publish skills, and it uses fixed rewrites rather than adaptive attacks (Limitations).
    • After commands are moved into referenced scripts and the body is neutralized, the frozen gate's recall drops from 99.8% to 87.0%, and at the 0.5 threshold from 96.6% to 67.5%. All detectors that read only SKILL.md have this limitation; package-level inputs are left for future work. This rewrite is used only for malicious skills and SkillScreen (Limitations; Appendix L).
    • Official labels are kept, some of which depend on package code or behavior not present in SKILL.md, thus limiting the ceiling that any text-only detector can recover (Limitations).
    • The MSD holdout is by malicious family, while legitimate skills still share sources across splits; SD has only two legitimate test sources. Source macro-average recall is 79%, overall 95.6%, and the intervals for small sources are wide (Limitations).
    • For MSD, the stronger GPU rerun of the encoder's two sets of five-seed results is reported. The reviewer comparison uses only one prompt, one set of examples, the provider settings in the text, and the probabilities reported by the models; log probabilities or repeated sampling would give reviewers finer 99% recall scores. Single-round training makes the number of updates vary with data size; the scanner context, paraphrase hypothesis, and null hypothesis each use one seed; the scale ablation also changes the NLI checkpoint version (Limitations).
    • At 99% recall, the authors position it as triage rather than an alarm: at about 21% FPR, precision is 34% if 10% of the stream is malicious and 4.5% if 1% is malicious. Deployment estimates also depend on source composition and on the overlap between model pretraining and public skills (Limitations). The conclusion states that more sources of legitimate data, package-level inputs, and consulting reviewers only when it would change the decision can extend it further.

    Scope of the experiments

    • The evaluation uses MSD, SD, and SBR from MaliciousSkillBench. MSD test has 1,947 entries (1,500 malicious, 447 legitimate), SD test has 1,384 entries (839 / 545), and SBR test has 1,950 entries (Section 4, Table 3).
    • Detectors include SkillScreen, ModernBERT-large, Word-SVM, untuned OpenJev-2B, TypeSafe Jev 1.13, SkillSpector 2.12.0, and GPT-5.5, Grok 4.3, Claude Haiku 4.5, GPT-5.4 mini. SkillScreen and ModernBERT use five seeds per split, and the encoder on SD uses six seeds (Section 4, Table 1, Appendix H).
    • Uncertainty uses 2,000 paired bootstraps. The high-recall decision threshold is selected on the validation set and then frozen; the shaded column of Table 1 instead re-selects the threshold on test scores (Section 4–5).
    • Ablations include scanner context, hypothesis wording, plain Qwen3.5-2B, OpenJev-0.8B, and 10%, 25%, 50% of the training labels (Table 2, Figure 3, Section 5.4–5.5).
    • The rewrite experiment uses 200 malicious MSD test skills and four archived adapters, and applies only to SkillScreen (Appendix L). The reviewer comparison uses one shared prompt and one answer per record (Section 4).
  6. Summarize the main content of the paper

    The 2B decision model screens SKILL.md in a single forward pass; on unseen attack families, the FPR at 99% recall is 22.6%.

    SkillScreen is a small decision model for pre-installation screening of agent skills, with the goal of high recall while keeping the review volume for legitimate skills down.

    • Problem: Legitimate automation and malicious behavior in SKILL.md look similar. Static rules have low recall, and the coarse scores of prompt-based review make it hard to control false positives at high recall. Attackers publish skills for installation; screening reads the behavior written directly in the body or specified by initialization steps, and high scores are sent for human review.
    • Method: Freeze the OpenJev-2B backbone and train for one epoch with rank-8 LoRA and an NLI head. The premise is the skill Markdown, and the hypothesis is fixed as that the skill is malicious. The softmax weight of entailment among the three classes of logits is used as a continuous score. On the validation set, take the highest threshold that reaches 99% recall, and freeze it before testing.
    • Unseen attack families: On MSD, the default threshold gives 95.6% recall and 3.3% FPR (Table 1). The FPR at 99% test recall is 22.6%, that of Word-SVM is 45.6%, and that of the tested LLM reviewers is 86.8%–100.0%. The abstract and conclusion describe this tier as flagging 20%–25% of legitimate skills. The threshold frozen on the validation set gives 98.7%–99.1% recall and 20.1%–23.9% FPR on test.
    • Unseen sources: On SD, the FPR at the same 99% test recall is 50.5%. The authors report it is 24.1 percentage points lower than ModernBERT-large, which is also trained for a single round and taken at the converged round. Two unfamiliar legitimate test sources raise the false positives of each trained detector.
    • The authors' conclusion: High-recall screening need not rely on frontier models; task training is the source of the gain, and the named-task hypothesis separates ranking quality from false positives when labels are few. The authors also state that recognizing new attacks does not guarantee controllable screening on new skill sets. Future extensions are more legitimate sources, package-level inputs, and consulting reviewers only when it would change the decision. At 99% recall, the authors describe it as a first-pass triage: at about 21% FPR, precision is 34% for a 10% malicious stream and 4.5% for 1%.

Related papers

Read originalarxiv.org