Real Money, Fake Models: Deceptive model claims of third-party Shadow APIs

Core question: third-party shadow APIs claim to provide access to frontier models consistent with official ones, but whether their outputs and backend identities are trustworthy previously lacked systematic auditing.

• Problem scenario: official APIs such as GPT-5 and Gemini-2.5 have high pricing, payment barriers, and regional restrictions (the authors give the example that the official OpenAI API cannot be accessed directly from China, Russia, and Iran), giving rise to a shadow API market claiming low prices, no regional restrictions, and outputs consistent with official ones.
• Why it matters: the authors find these services have been used by 187 academic papers (as of 2025-12-06); if the endpoints do not provide the claimed models, utility and safety conclusions based on them cannot be reproduced, and the risks of official models may be overestimated or underestimated.
• Gaps in existing work: the authors state that existing research has audited substitution issues in open-source model APIs, but systematic auditing of the indirectly operated shadow API market with opaque black-box operation is largely absent.
• Core concept: the authors propose the security property model provenance integrity—an endpoint sold under a model's name should maintain the binding between that model's identity and the actual backend; and distinguish behavioral differences (which do not point to backend identity) from evidence of fingerprints matching a specific substitute model, with only the latter called a deceptive model claim, and it does not represent operator intent.
• Paper approach: organized around three research questions—RQ1 identifies 17 shadow APIs and counts their use in 187 papers; RQ2 conducts utility and safety benchmark comparisons for 3 representative shadow APIs (A, E, H); RQ3 verifies endpoint identity using model fingerprinting (LLMmap), distribution tests (MET), and metadata analysis, and provides recommendations for auditing and researcher verification processes.

Model Fingerprinting and Generated-Content Identification

• An LLM's language patterns can serve as a fingerprint to identify the model that produced content (papers cite [21, 45, 65]).
• Pasquini et al. [51] propose LLMmap: actively probe a model with carefully constructed queries and compare the likelihood of responses under different reference models to identify its identity; this paper directly adopts this framework for endpoint fingerprint verification.

API Model Substitution and Verifiable Inference

• Cai et al. [24] audit model substitution in open-source LLM APIs and evaluate using TEEs for hardware-level verifiable inference; the paper notes that this work differs in focusing on shadow APIs operating as opaque black boxes.
• Krishna et al. [42]'s model extraction study shows that trainable substitute models can imitate BERT-like NLP APIs, making it difficult for users to distinguish the original API from an extracted surrogate model based on surface interaction alone.
• Gao et al. [32] propose Model Equality Testing (MET), which tests whether API outputs are from the same distribution as a given model; this paper uses it as complementary verification beyond fingerprints.

Adjacent Risks in the LLM Deployment Ecosystem

• Hou et al. [34] measure the popularity, configuration, and exposure risks of public LLM deployments; Puthal et al. [52] discuss the security and governance risks of unsanctioned shadow AI use within organizations. The paper notes that these works focus on public deployments and organizational governance, respectively, rather than on whether commercial indirect endpoints provide the claimed models.

Comparison Baselines and Benchmarks: This paper does not propose a new model; the baselines are the official APIs of each model; the utility benchmarks are AIME 2025, GPQA (Diamond), MedQA (USMLE), and LegalBench (Scalr), and the safety benchmarks are JailbreakBench (100 items) and a 50-item subset of AdvBench, with safety attacks using GCG, Base64, Combination, and FlipAttack.

Positioning of This Paper: The authors call this the first systematic audit ("the first systematic audit") between official LLM APIs and their corresponding shadow APIs, covering three levels: ecosystem scale, utility and safety behavior, and fingerprint-based identity verification.

Overall approach: first define shadow API and survey its scale of use in academic and open-source communities (RQ1), then compare the utility and safety behavior of shadow and official endpoints under identical prompts and decoding configurations (RQ2), and finally verify endpoint backend identity using fingerprints, distribution tests, and metadata analysis (RQ3), and provide an auditing and researcher verification process.

Definition and ecosystem survey (RQ1)

• shadow API definition: third-party services that indirectly access frontier proprietary models through intermediary providers, bypassing official geographic access controls in restricted regions; they must use the intermediary provider's credentials to sell access and not disclose upstream routing (Table 1). Excluded are publicly documented channels such as Azure OpenAI, platforms that disclose routing such as OpenRouter, and self-hosted gateways with bring-your-own credentials.
• Collection method: using ICLR 2024 (2,260 papers, 1,108 with code) and ACL 2024 (1,923 papers, 1,005 with code) as seeds, parse abstracts and footnotes to extract code repository URLs; from 2,113 papers with code, identify 92 projects using LLM APIs, then iteratively expand on GitHub by endpoint URL until no new endpoints; metrics updated to 2025-12-06.
• Compliance check: verify provider identity, company registration, ICP filing, legal documents, and payment methods (Appendix E.2, Table 14).

Utility and safety evaluation (RQ2)

• Subjects: select 8 models across three model families according to the OpenRouter November 2025 token usage ranking (GPT-4o-mini, GPT-5, GPT-5-mini; Gemini-2.0-flash, Gemini-2.5-flash, Gemini-2.5-pro; DeepSeek-Chat, DeepSeek-Reasoner); for sensitive domains and safety evaluation, take one representative per family (GPT-5-mini, Gemini-2.5-flash, DeepSeek-Chat).
• shadow API selection: select A, E, H ranked by citation count and GitHub stars, with criteria of popularity, public accessibility, and coverage of the three model families; official baselines query the official API directly.
• Evaluation protocol: all experiments repeated 3 times and averaged with standard deviation reported; utility uses AIME 2025, GPQA (Diamond), MedQA (USMLE), LegalBench (Scalr), using established templates such as EvalScope and Hulu-Med; safety uses JailbreakBench and AdvBench-50, applying four attacks: GCG (generating sample-level suffixes with LLaMA3-8B then transferring), Base64, Combination, FlipAttack, with GPT-4o-mini plus the StrongREJECT scoring template outputting harmfulness scores, where higher scores are less safe.

Identity verification (RQ3)

• LLMmap fingerprinting: actively probe using the default query strategy, compute Euclidean distance between outputs and the reference database; flag when distance exceeds 1.25 times the official baseline or when the top-1 identification result does not match the claimed identity; the fingerprint database is trained on NVIDIA DGX A100, model list in Appendix H (Table 17).
• MET distribution test: two-sample hypothesis test, α=0.05 rejecting the null hypothesis of equal distributions (Reject=True) means outputs are statistically distinguishable from official; covers 8 model configurations of A, E, H on 4 benchmarks.
• Controlled validation: build 6 local endpoints with known backends (3 honest, 3 substituted: cross-family substitution, version mismatch, capability degradation), with 100 LLMmap queries and 600 MET requests per endpoint, calibrating false positives and false negatives of both methods.
• Metadata analysis: for 24 endpoints on AIME 2025 and GPQA, compute the standard deviation ratio of latency and token count r = SD_shadow / SD_official, yielding 48 endpoint-benchmark pairs per dimension.

Evidence interpretation framework

• The authors explicitly distinguish: behavioral, latency, metadata, or MET differences only indicate that the endpoint does not reproduce the official baseline, and do not indicate the cause or backend; only when fingerprints match a specific substitute model do they report evidence consistent with model substitution, and black-box fingerprinting is probabilistic, so a match supports but does not prove backend identity.

Experimental Setup

• Models tested: 8 in total: GPT-4o-mini, GPT-5, GPT-5-mini, Gemini-2.0-flash, Gemini-2.5-flash, Gemini-2.5-pro, DeepSeek-Chat, DeepSeek-Reasoner; sensitive domains and safety evaluation use GPT-5-mini, Gemini-2.5-flash, DeepSeek-Chat.
• Endpoints tested: shadow APIs A, E, H (named in order of GitHub stars) plus official API baselines, totaling 24 shadow endpoints; behavioral and identity conclusions are limited to these three and the authors explicitly do not generalize to the remaining 14.
• Benchmarks: AIME 2025, GPQA (Diamond), MedQA (USMLE, 1,273 items, see Table 15), LegalBench (Scalr, 571 items, see Table 16), JailbreakBench (100 items), AdvBench 50-item subset.
• Metrics and judging: utility is accuracy (mean and standard deviation over 3 repetitions); safety is a harmfulness score assigned by GPT-4o-mini following the StrongREJECT template; identity is LLMmap Euclidean distance and MET (α=0.05).
• Decoding configuration: compatible models set temperature=0, seed=42; reasoning models use default reasoning effort (Table 13).

Main Results

The table below summarizes the key differences reported by the authors (based on Section 4 of the main text, Table 3, Table 15/16; per-cell values are presented as figures in the paper, and only the numbers given in the main text are listed here).

Swipe horizontally to read the table

Dimension	Official baseline	shadow API results	Source
MedQA accuracy (Gemini-2.5-flash)	83.82%	three-provider average 36.95%, difference 46.51%–47.21%	Section 4.2
AIME 2025 average accuracy difference	—	A differs by 9.81%, H by 6.46%, E by 2.64%	Section 4.2
AIME 2025 largest single-model difference	—	on A, Gemini-2.5-pro differs by 40.00%, DeepSeek-Reasoner by 38.89%	Section 4.2
Average accuracy drop in sensitive domains	—	A drops 16.96%, E drops 15.71%, H drops 14.75%	Section 4.2
Fingerprint verification (24 endpoints)	—	45.83% fail, another 12.50% show Euclidean distance deviation	Table 3
Overall risk grading (24 endpoints)	—	11 High, 9 Medium, 4 Low	Table 10

• The authors state that shadow API E has the best consistency (average accuracy drop 2.64%, with GPT-5-mini even 1.18% higher than official on GPQA), while A and H degrade markedly on reasoning-type tasks.
• The authors report that Gemini-2.5-flash's agreement rate with official answers on MedQA is only 49.57%–49.65%, and on LegalBench 52.36%–54.64% (Table 15/16); the three providers' values are close, but the authors state that performance evidence alone cannot determine the cause.

Safety Evaluation

• JailbreakBench: GPT-5-mini under Base64 has a harmfulness score of 0.04 on shadow API A, twice the official 0.02; Gemini-2.5-flash under FlipAttack is 0.90 officially and about 0.67–0.68 on the three shadow providers, a difference of about 0.23 (Figure 5).
• AdvBench: GPT-5-mini under Base64 is 0.05 on A, 5 times the official 0.01; Gemini-2.5-flash under FlipAttack is 0.95 officially and about 0.82 on shadow (Figure 6).
• Authors' interpretation: Safety scores shift in both directions—lower than official may underestimate official model risk, while higher than official may exaggerate attack success rates that cannot be reproduced on official endpoints; the authors consider safety evaluation relying on shadow APIs unreliable.

Identity Verification and Controlled Calibration

• Fingerprint results (Table 3): The GPT and DeepSeek families frequently show identity mismatches, e.g., H's GPT-4o-mini fingerprint matches Qwen2.5-7B, A's and E's GPT-5 match glm-4-9b-chat, and A's and H's DeepSeek-Reasoner behaves as the non-reasoning DeepSeek-Chat; Gemini-2.5-pro's distance is stable across the three providers (D≈17.37–18.04).
• Capability-matched substitution: The Qwen2.5-7B identified in H scores 42.93% on GPQA, 1.68 percentage points from official GPT-4o-mini's 44.61%, but on AIME 2025 scores 3.33% versus 14.44%, a difference of 11.11 percentage points; the authors state this explains why accuracy drops concentrate on reasoning-intensive benchmarks.
• MET (Table 4): Complementary to fingerprints; for Gemini-2.5-pro on AIME 2025, none of the three providers reject the null hypothesis, and the authors emphasize this does not amount to proof of faithful service.
• Controlled testbed (Table 6): LLMmap overall accuracy 96.00% (FPR 3.00%, FNR 5.00%), MET accuracy 88.33% (FPR 4.67%, FNR 18.67%); the authors state MET's higher FNR stems from detecting distribution shift rather than exact identity, and it may miss same-family minor-version mismatches.

Joint Analysis and Metadata

• Regression (Table 9): In an OLS with accuracy drop as the dependent variable (n=24, R²=0.15), neither price ratio (β=−1.447, p=0.490) nor identity mismatch (p=0.98) has predictive power; after excluding the three Gemini-2.5-flash endpoints, R²=0.30 and identity mismatch is marginally significant (β=7.72, p=0.05). The authors state that price cannot serve as a signal of backend fidelity.
• Counterexample (Section 6.1): Gemini-2.5-flash's fingerprints are correct at all three providers, yet accuracy in sensitive domains consistently drops by about 47%; the authors speculate that providers may disable or reduce the thinking budget to save costs, and fingerprints cannot detect such non-identity mechanisms.
• Metadata (Table 7/8): Among 48 endpoint-benchmark pairs, 64.58% have a latency SD ratio above 1.20 and 35.42% above 2.00; token-count ratios are above 2.00 for 20.83% and below 0.80 for 20.83%, which the authors state is consistent with dynamic backend switching or upstream retries.

Other Ablations and Analyses

• Economic snapshot (Table 12): For GPT-5 on GPQA, converted at official pricing, shadow endpoints deliver 0.36–0.52 times the value of official; the authors note this is descriptive background only.
• Ecosystem observations (Section 6.3): 4 of the 17 had ceased operations at the time of writing; some platforms display "model verification passed" badges and third-party substitution detection platforms, which the authors explicitly do not attribute to their own disclosure.
• Compliance metadata (Table 14): Of the 17, only H has a verifiable ICP registration, and only N discloses a corporate legal entity.

Limitations and Future Directions Noted by the Authors

• Time frame and market volatility (Section 8): The study is a bounded snapshot from September–December 2025; shadow APIs may change upstreams, adjust routing, or cease operations, and behavior may drift; the authors state that the recorded risk signals reflect structural properties of such services, and they intentionally adopt a snapshot design rather than continuous monitoring.
• Detection coverage and ground truth (Section 8): Lacking true information about backend infrastructure, it is impossible to distinguish all fine-grained implementation differences (e.g., silent prompt preprocessing, context truncation), and black-box verification cannot determine the nature of a substitute backend (distillation, fine-tuning, quantization, or version shift).
• Detection can be evaded (Section 8): A sufficiently faithful surrogate model could in principle bypass both LLMmap and MET; passing both tests only means there is no evidence of substitution, not proof of authenticity.
• Official endpoints themselves change (Section 8): Official APIs of proprietary models may undergo undisclosed fine-tuning; the authors mitigate this with variance across three independent runs.
• Analytical perspectives can be extended (Section 8): The authors acknowledge that shadow API differences could also be analyzed from perspectives such as bias and hallucination.
• Coverage and future work (Section 8): The 17 providers were collected via citations and stars and may not represent the entire market; other services may employ different deception strategies; focusing on only three model families, the authors believe the audit process can be generalized to broader model coverage and leave this as future work.

Experimental Coverage

• Behavioral and identity conclusions cover only the 24 endpoints of shadow APIs A, E, H, and the authors explicitly do not generalize to the remaining 14 (Section 2.3).
• The models tested are 8 models across the three families GPT, Gemini, and DeepSeek; safety and sensitive-domain evaluations use only 1 representative model per family (Section 4.1).
• Utility and safety experiments are each repeated 3 times and averaged (Section 4.1); controlled verification uses 6 endpoints, with 100 LLMmap queries and 600 MET requests per endpoint (Section 5.3).
• Safety evaluation covers 4 jailbreak attacks and 2 benchmarks, with harmfulness scored by GPT-4o-mini following the StrongREJECT template; the paper does not report agreement between the judge and human annotation.
• The ecosystem survey uses code-bearing papers from ICLR 2024 and ACL 2024 as seeds for iterative GitHub expansion, with metrics as of 2025-12-06 (Section 3.1).

A systematic audit paper targeting shadow APIs (third-party services claiming to provide indirect access to frontier models consistent with official ones); the authors call it the first systematic audit in this direction.

• Problem: The price, payment, and regional restrictions of official APIs have given rise to the shadow API market; the authors identify 17 such services, used by 187 academic papers, with the most-used one accumulating over 5,900 citations and 58,000 GitHub stars (as of 2025-12-06), yet no one has verified whether the endpoints actually provide the claimed models.
• Method: For 24 endpoints from 3 representative providers (A, E, H), compare against official baselines under identical prompts and decoding configurations, covering utility (AIME 2025, GPQA, MedQA, LegalBench) and safety (JailbreakBench, AdvBench, 4 jailbreak attacks), and verify backend identity using LLMmap fingerprints, MET distribution tests, and latency/token metadata; additionally build a 6-endpoint controlled testbed to calibrate the detection methods.
• Main results: Gemini-2.5-flash on MedQA drops from the official 83.82% to a three-provider shadow average of 36.95% (difference 46.51%–47.21%); among the 24 endpoints, 45.83% fail fingerprint verification and 12.50% show distance deviation, with some endpoints' fingerprints matching substitute models such as Qwen2.5-7B and glm-4-9b-chat; safety harmfulness scores shift in both directions (e.g., under FlipAttack, Gemini-2.5-flash is 0.90 officially and about 0.67–0.68 on shadow); regression shows price ratio has no predictive power for accuracy drop (p=0.490).
• Authors' conclusion: The authors argue that shadow APIs cannot be treated as substitutes for official endpoints, and that their behavioral differences and identity inconsistencies undermine research reproducibility; endpoints whose fingerprints match substitute models constitute what the authors call a deceptive model claim (without asserting operator intent).
• Recommendations: The authors propose a four-stage verification process for auditors (LLMmap probing, MET, multi-session variance checks, ICP/legal-entity verification) and a pre-registration checklist for researchers (disclose endpoint URLs, model versions, access dates, and archive raw responses and metadata), and call on conferences to flag unverified third-party endpoints as a reproducibility risk.