Runtime Skill Audit: Targeted Runtime Probing for AI Agent Skill Safety

The researchers propose Runtime Skill Audit (RSA), which uses targeted runtime probing to audit the safety of AI agent skills, achieving 90.0% accuracy on 100 skills in OpenClaw. The paper points out that skills may appear harmless in documentation or code and only exhibit malicious behavior under specific user requests, local assets, persistent state, or multi-step tool interactions, so purely static review is unreliable. RSA characterizes risk-related interfaces, prepares execution contexts, and assigns safety labels based on trajectory evidence, achieving an 88.0% true positive rate and an 8.0% false positive rate, with accuracy 13.0 percentage points higher than the best static baseline. Under self-evolving attacks, static detectors fail after one or two rounds, while RSA still detects 19 to 20 of 20 malicious skills in each round.