AgentProv: Auditing Agentic LLM API Providers via Tool-Use Policy Probes

Researchers propose AgentProv, which generates fingerprints from a model's distribution of tool-call categories and uses an MMD permutation test to determine whether the foundation model deployed behind an agentic LLM API is the one claimed; the authors state that this is the first action-based identity auditing method. The paper points out that modern service stacks (OpenAI, Anthropic, Gemini, Cloudflare Workers AI, LangGraph) discard text when a model calls tools and expose only structured actions, and that system prompts injected by providers also distort text distributions, making existing audits based on the text channel difficult to apply to agentic APIs and potentially causing honest providers to be misjudged. On 630 checkpoint pairs, AgentProv identified all substituted models (100%), with a false positive rate of 7% under system prompt injection, lower than MET's 67% and RUT's 53%. On third-party API endpoints, its disagreements with MET were consistent with the results of an independent token-counting side channel capable of detecting provider-injected system prompts.