What the middlemen delivered: answer behavior, tool permissions, and billing usage of LLM API relays
What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays
The authors measured the answers, tool permissions and billing of 20 LLM API relay services; in the first round, 7/10 low-price and 2/10 same-price or higher-price services were flagged for behavioral differences, and 5 over-reported 4,200–60,000 input tokens.
- Third-party LLM API relays offer model access at lower prices, but users cannot verify from the answers alone whether model quality, instruction adherence and billing all meet standards at the same time; existing research covers only part of this.
- The authors used public APIs and account records to measure 20 relays: comparing knowledge-question accuracy against official answers, screening returned content and checking tool permissions, comparing input token counts and balance deductions, and retesting after 11–18 days.
- In the first round, 7/10 low-price and 2/10 same-price-or-higher-price relays were flagged for behavioral differences (p=0.07); among 3,332 responses there was no confirmed injection, but 4 providers violated the tool prohibition and returned calls; 5 providers over-reported input tokens by 4,200–60,000, and for 6 providers with checkable balances, all charges matched the reported usage.
- The authors note that the sample is not random, that backend models or hidden input content cannot be identified, and that conversation retention and credential misuse were not measured; the retest had only two rounds and cannot distinguish service changes from reference fluctuation.
- Threat model
- The authors assume that relay operators can read and modify requests and responses, choose the upstream model, change generation settings, and report different model names or token counts; the client, official reference accounts and encrypted connections were not compromised. Measurements used only public APIs and account records, without accessing the relays' internal systems.
- Evaluated models
- GPT-6 AstraClaude Opus 5
- Benchmarks
- 自建 27 题知识问答题库(96 候选题,含 recent-event knowledge、parameter-scale-sensitive、capability-edge 三类)自建代码与配置请求工作负载(5 类、50 个计划响应/对)
- Metrics
- 命中率与风险差 RD行为不一致标签(Confirmed/Suspected/No evidence)确认注入率(Clopper–Pearson 上限)输入 token 盈余中位数余额扣减与按报价计算费用的一致性
The researchers tested 20 third-party API relay services offering GPT-6 Astra and Claude Opus 5, and found that some services deviated from the official API in answer behavior, tool permissions, and reported token usage. In the first round of samples, the analysis flagged 7/10 low-price relays and 2/10 same-price or higher-price relays as having differences in answer behavior; when retesting 35 pairs, 7 pairs gained or lost flags under the original rule, and 6 pairs under the matched correction family. Screening the returns of 3,332 fixed non-adversarial code and configuration tasks did not confirm injection, but 4 relays still returned tool calls when instructions prohibited tool calls. For identical requests, 5 relays reported 4,200 to 60,000 more input tokens than the official API; among the 6 services with checkable balances, charges were calculated based on reported usage and list price, and one of them also charged for the additionally reported input tokens.
Related events and news
1 events · 1 news mentions
News that mentions this paper
大模型安全周报:智能体安全边界在签名、授权与上下文
- 公众号 大模型安全研究cites the paper's id or link大模型安全周报:智能体的安全边界在签名、授权与上下文三层防线里
1. 低价 API 中转站实测:首轮十家里七家回答变样,还夹带多计费与放宽的工具权限:What the Middleman Delivers: Answer Behavior, Tool Permissions, and Billed Usage in LLM API Relays(https://arxiv.org/abs/2611.07650)
Paper analysis
What problem does this paper attempt to solve?
The authors measured whether 20 LLM API relays simultaneously honor three commitments: answer quality, tool permissions and billing accuracy.
Core question: do third-party LLM API relays simultaneously meet users' expectations for model quality, instruction adherence and billing accuracy, for which existing research has no overall answer.
- Problem setting: relay services attract users with lower prices, convenient payment or regional reachability, but a relay can read plaintext before forwarding and can modify instructions, swap models or edit responses, which users cannot verify from answers or bills alone (Introduction).
- Gaps in existing research: the authors state that shadow API research [1] and GateScope [3] cover only some properties, that model verification such as KBF [4] cannot determine whether a relay complies with tool restrictions, and that checking the list price cannot verify the origin of billed token counts.
- Core observation: plausible answers may come with disabled tool calls, and billing at the list price may include input tokens of unknown origin; the authors argue that behavior, permissions and usage need to be measured together, with repeated measurement to distinguish persistent problems from temporary changes.
- Paper's approach: for 20 relays offering GPT-6 Astra or Claude Opus 5, use the official API as reference to compare answers, screen content and tool permissions, check token counts and balances, and retest 35/36 pairs after 11–18 days.
- Threat model (Section 2, following the paper's assumptions):
- Goal: examine relay delivery at three levels visible to the client: answer behavior, tool permissions and reported usage.
- Knowledge: the relay operator can read and modify requests and responses, choose the upstream model, change generation settings, and report different model names or token counts.
- Capability: behavior may vary by account, prompt, or whether testing is detected; measurement uses only public interfaces and account records and does not access the relay's internal systems.
- Victim systems and assumptions: the client, the official reference account and the encrypted connection are not compromised; conversation retention, credential misuse and gateway software vulnerabilities are outside the measurement scope (Table I).
What related research exists?
Related work falls into three groups: API output model verification, computation and delivery verification, and relay measurement and monitoring practice.
API output model verification
- LLMmap [11] and single-token fingerprinting [12]—identify models using characteristic responses or the frequency of short random prompts; IRIS [19] compares against known backend profiles to detect model substitution.
- Distribution and rank tests [13][14]—compare completion distributions or use a local reference model for scoring in a rank test; the authors note that the rank test requires local reference scoring.
- KBF [4]—distinguishes API behavior using stable correct/incorrect answers near the knowledge boundary, and is the basis for this paper's question design; this paper builds its own question bank and decision rules on top of it, and states that the controlled mixing experiment is not used to claim superiority over KBF.
Computation and delivery verification
- SVIP [15], privacy-preserving inference protocols [16], AFTUNE [17]—verify model execution with server participation or in a trusted execution environment; the authors note that these require access beyond the final text and do not protect against subsequent changes along the delivery path.
- Signed responses [26] and the AEX proof extension [27]—bind execution fields to a trusted provider; the authors note that their protection depends on what the signature covers and which issuer the client trusts.
- AEGIS [2] and evidence-binding gateways [28]—place forwarding in an enclave or separate operational control from a provable runtime; the authors note that these require support from the provider or forwarding component, whereas this paper's checks use only existing public APIs.
Relay measurement and monitoring practice
- GateScope [3]—measures consistency, multi-turn memory, billing and latency across ten gateways; the shadow API study [1]—examines 17 services and empirically tests utility, safety and model identity for 3.
- Agent routing research [18]—found malicious edits and credential-related behavior in workflows containing tools and credentials; the authors note that this paper's benign independent requests may not trigger attacks that depend on tools, credentials or long interactions.
- Monitoring sites [33][34], open-source tools [35] and industry briefs [36]—the authors note that their public descriptions lack the validation data and false-positive calibration needed to interpret scores as model identity guarantees.
Baselines and benchmarks: this paper has no comparison-method baseline; the evaluation benchmarks are a self-built 27-question knowledge question bank (compared against official API references) and a workload of 5 categories of code and configuration requests, with substitute models Claude Haiku 4.5, GPT-6 Luna and GLM-4.5-Flash.
The authors position this paper as: within the same sample, connecting answer accuracy, content and tool permissions, reported input and account billing, and examining the distribution and persistence of differences by price group and across two rounds of retesting (Table II).
How does the paper address this problem?
Compare answers against the official API reference, screen content and tool permissions, check tokens and balances, and calibrate with controlled mixtures and retests.
Overall approach: without accessing the relays' internal systems, measure 20 relays at three client-visible levels using official API references + account records, and use controlled response mixing to calibrate accuracy checks and a second-round retest to check persistence.
Service selection and price grouping
- Candidate sources: model recommendations (ChatGPT, Claude, Gemini, Grok), community resources and English web searches; required self-service registration, API access, public pricing, English interface, credit card or PayPal payment, and a minimum top-up of no more than $15.
- Sample composition: 10 low-price and 10 same-price-or-higher-price, non-random sampling; after four pilots, the low-price group was expanded according to a pre-fixed continue-or-stop rule.
- Price classification: for four standard short-context quotes, take the mean ratio Ps = 1/4∑j=14 (cs,j)/(cofficial,j), with P<0.95 为低价,0.95–1.05 为同价,>1.05 as higher price.
Question bank construction and accuracy comparison
- Question bank: 96 candidate questions in three categories (recent-event knowledge, parameter-scale-sensitive, capability-edge); sample three times each on two official targets and three cheaper substitute models; a question was selected only if the target answered correctly at least twice and at least two substitutes each answered incorrectly at least twice; 25 questions were selected per target, and a separate reserve question bank was built; the union of the two was 27 questions, sharing 23 questions (92%).
- Collection: for each relay–target pair and official reference pair, 5 planned trials per question, 135 calls for 27 questions; requests used low reasoning effort, with an unused declared tool and tool_choice=none; safety refusals were counted as errors in the main analysis.
- Decision rule: risk difference RD = official hit rate − relay hit rate; q<0.05 and |RD|≥0.20 recorded as Confirmed, q<0.05 with a smaller effect or 0.05≤q<0.15 and |RD|≥0.20 recorded as Suspected, otherwise No evidence; in the first round, BH correction was performed separately by collection batch.
- Review: free-form answers were first initially labeled by Claude Opus 5.5, then adjudicated by the authors in blind review; inter-batch annotation agreement κ was 0.943–1.0 (Appendix IV).
Content and tool permission checks
- Injection screening: code and configuration requests in 5 categories (package names, hidden shell commands, suspicious URLs, encoded content, installation instructions), 5 phrasings × 2 trials, 50 planned responses per pair; after automatic screening hits, authors manually confirmed in context; a finding required the relay's confirmation rate to exceed the official one and at least one confirmed injection.
- Tool calls: compare returned tool call records with requested permissions, distinguishing explicit tool_choice: none from undeclared tools; counts include calls returned in failed attempts, and none of the calls were executed.
Usage and billing verification
- token comparison: in the second round, for identical requests, compute the difference between relay and official input token counts, taking the median by relay–target pair; Messages and chat completions formats were aggregated separately according to the paper's field definitions.
- Balance verification: for 6 services exposing a balance endpoint, read the balance before and after collection, compute cost using reported usage × list price, and compare with the decrease in balance.
Calibration and retest
- Controlled mixing: mix official and substitute answers with replacement probability r∈{0, 0.10, 0.25, 0.50, 1.00}, 1,000 simulations per setting, and measure detection frequency using the same Fisher test and labeling rule; the authors note that the question bank and substitute answers come from the same source, so the frequency is optimistic.
- Retest: after 11–18 days, retest 35/36 pairs with a new same-day official reference, keeping questions, number of trials, scoring rules and thresholds unchanged; the second round used a single BH family.
What experiments did the paper conduct?
In the first round, 7/10 low-price and 2/10 same-price or higher-price relays were flagged; no confirmed injection; 5 over-reported input tokens and counted them in billing.
Experimental Setup
- Subjects under test: 20 anonymous relay services (R01–R20); target models were GPT-6 Astra and Claude Opus 5; substitute models Claude Haiku 4.5, GPT-6 Luna, and GLM-4.5-Flash were used for question-bank pre-validation and mixed simulation.
- Benchmark: a self-built 27-question knowledge question bank (135 planned calls per pair, with at most 125 scored responses in the main question bank); code and configuration workloads had 50 planned responses per pair.
- Metrics: hit rate, RD, Confirmed/Suspected/No evidence labels, confirmed injection rate, median input token surplus, balance consistency.
- Review method: free-form answers were initially labeled by Claude Opus 5.5 and adjudicated by the authors under blinded review; injection candidates were classified by two authors; all 64 candidates in the first round were unanimously judged benign.
- Repetitions and rounds: 5 trials per question; the second round retested 35/36 pairs after 11–18 days; price-group comparisons used a two-sided Fisher exact test.
- Price test: the main comparison had p=0.0698 (reported as 0.07), not reaching 0.05; the post hoc all-four-specification robustness comparison was 7/10 pairs vs 1/10, p=0.0198, and the authors explicitly state it is not the primary test and should not be regarded as stronger evidence.
Main Results
According to Table III (first-round primary analysis, A=GPT-6 Astra, O=Claude Opus 5; RD=official − relay hit rate):
Swipe horizontally to read the table
Relay Price group A RD (label) O RD (label) R01 Low +0.165 (S) +0.288 (C) R08 Low +0.184 (S) +0.152 (S) R11 Low +0.248 (C) +0.574 (C) R17 Low +0.240 (CCSS) +0.256 (C) R19 Low +0.056 (S) +0.016 (N) R20 Low +0.288 (C) +0.591 (C) R04 Low Not reported +0.080 (S) R10 Same price Not reported +0.192 (S) R14 Same price +0.008 (N) +0.088 (SNNN) - The table lists all 9 services flagged in the first round; the remaining 11 (R02, R03, R05, R06, R07, R09, R12, R13, R15, R16, R18) had no flags in the first round, and the Astra cells for R09 and R13 were not measured.
- The authors note that R14's flag depends on the refusal scoring method (it disappears when refusals are excluded or the alternate question bank is used), and that R10's flag persists across all four specifications but its programming-client-style wrapper is a relevant alternative explanation.
- The authors emphasize that the labels only indicate behavioral inconsistency under the chosen prompts and rules and cannot identify the backend model; R17 Astra changed from Confirmed to Suspected (CCSS) under the alternate question bank, but the flag persisted across all four specifications.
Controlled Mixed Calibration (RQ1)
- Weak substitutes are easily detected: Haiku and GLM were 100% flagged against both targets at 25%, 50%, and 100% substitution; at 10% substitution, Opus was 94.1% flagged against Haiku and 94.6% against GLM, while Astra was 47.1% and 53.7%, respectively (Table S2).
- Similar substitutes can be missed: the Astra–Luna mix was 0.0% flagged at 10% substitution, only 4.0% at 25%, rising to 77.7% at 50%, and reaching 100.0% only at 100%; the authors note that Luna's knowledge cutoff is later and the question bank has limited coverage of new knowledge.
- Official-vs-official control: at zero substitution, each of the two targets was flagged 0/1,000; the authors note this is only a configuration check and does not constitute a general upper bound on the false-positive rate.
Content Injection and Tool Permissions (RQ3, RQ5)
- Injection screening: 0 confirmed injections among 1,686 relay responses in the first round, one-sided 95% Clopper–Pearson upper bound 0.18%; combined across both rounds 0/3,332, upper bound 0.09%; the authors call this a bounded negative result, not a safety proof.
- Tool permissions: in the first round, 4 services still returned tool calls under tool_choice: none—R04 (1 time), R11 (24 times), R10 (1 time), R17 (4 times), all unused noops; R08 returned 2 web_search records when no tool was declared; the second round had no tool calls at all (Table VI).
- Authors' interpretation: returning a call does not equal executing an action, and it is impossible to distinguish relay modification, upstream non-compliance with parameters, or output transformation; the official reference had no tool calls in either round.
Usage and Billing (RQ5)
- Token surplus: in the second round, 5 services over-reported input tokens for identical requests—R01 and R20 each reported 4,416 (official 36), R11 reported 4,418, R08 had a cache surplus of 4,216, and R10's Opus median surplus was about 59,500; the authors say this is consistent with added system prompts, but altered accounting could produce the same reports.
- Billing consistency: for the 6 services with checkable balances (R03, R04, R11, R12, R17, R19), the balance decrease exactly matched reported usage × list price to the cent; R11 is the only service that both over-reported by more than 4,000 tokens and had a checkable balance, and its surplus was billed (Table S3: computed $1.20107395, actually charged $1.2011).
- Authors' interpretation: billing at the quoted price does not mean the reported token counts are real; whether the other 4 over-reporting services billed the surplus could not be verified.
Retest and Changes (RQ4)
- Persistence: of 35 retested pairs, 9 were flagged in both rounds, 5 lost their flags, 2 newly gained flags, and 19 had no flags in either round; at the service level, R01, R08, R10, R11, R17, and R20 were flagged in both rounds, and R02 and R12 newly gained flags.
- Change cases: R02 Astra's RD changed from −0.024 to +0.248, with errors including claims that the 2025 Nobel Prize in Literature had not yet occurred; the authors say this is consistent with some traffic reaching a model with an earlier knowledge boundary, but the model could not be identified.
- Authors' interpretation: two rounds of measurement cannot distinguish service changes from reference fluctuation; the official reference hit rate fluctuated between 0.952–0.992 (Astra) and 0.968–1.000 (Opus), and no dedicated official-vs-official retest was performed.
Other Ablations and Analyses
- Question-block bootstrap: R01 Opus gap 0.288, 95% interval [0.128, 0.464]; R01 Astra 0.184 [0.080, 0.304]; R04 Opus 0.080 [0.016, 0.160] (Table IV).
- Post hoc check with a unified BH family: 6 of 35 pairs had label flips (7 pairs under the original rule), and only R19 Astra's first-round label changed from S to N (q=0.0893) (Appendix III).
- Configuration check: an Opus request with thinking disabled still returned 182 thinking tokens, Astra rejected the no-reasoning value (HTTP 400), and a full no-reasoning reference was not collected (Appendix II).
- Pilot observation: when the explicit refusal instruction was removed, Opus's score rose from 6/40 to 20/40; the authors state that prompt sensitivity was not isolated as a causal effect (Appendix V).
What are some points that can be further explored?
The authors list the boundaries of the sample, causal identification and coverage, and propose directions for the records required by verifiable relay services.
Limitations noted by the authors and future directions
- Non-random sample expanded after pilots: selection was not random, the low-price group was expanded after pilots, retesting the same set of services does not add independent samples, and price cannot be a reliable predictor of service quality (Section 6A).
- Accuracy differences cannot identify models: instructions, refusal policies, reasoning settings, retrieval and caching can all change answers; passing the tests also does not rule out missed detection of similar substitutes at moderate substitution rates (Section 6B).
- Usage checks cannot see hidden input: token comparison cannot reveal the content of any hidden input or determine the cause of count differences; matching balances do not verify the tokens actually processed upstream (Section 2B, Section 5 RQ5).
- Out-of-scope risks: conversation retention, credential misuse and gateway software vulnerabilities were not measured; benign independent requests may miss behavior triggered by long conversations, credentials or tool execution (Table I, Section 2B).
- Boundaries of simulation and statistics: the mixing simulation reuses the substitute answers used in question selection, so frequencies are optimistic and cannot be treated as field detection capability; the Fisher test treats correlated trials as independent, and combined p/q values may be optimistic (Section 4C, F).
- Future directions: the authors propose that verifiable relay services should connect the requested model and permissions, disclosed input additions, returned tool calls and billing token categories to verifiable processing records, and note that signatures and provable forwarding can supplement behavioral checks (Section 6D).
Experimental coverage
- The subjects were 20 anonymous relays and 2 target models (GPT-6 Astra, Claude Opus 5), totaling 36 first-round relay–target pairs, with 35 pairs retested in the second round (Section 4A, G).
- All requests were sent from the United States, and the results describe the service obtained by a client in a single region; relays using cryptocurrency payment, non-English interfaces, enterprise registration or large top-ups were excluded (Section 4A).
- The knowledge question bank had 27 questions, 5 trials per question; the content workload totaled 3,332 relay responses across two rounds; balance verification covered only 6 services exposing a balance endpoint (Section 4C–E).
- Measurement comprised two rounds 11–18 days apart; no dedicated official-to-official retest was performed, and the flag flip rate in the absence of service changes is unknown (Section 4G).
- The paper does not report each relay's upstream routing and backend model, and the authors explicitly state that the measurement cannot identify the backend; the impact of automatic rule corrections in injection screening on the number of candidates cannot be quantified (Appendix V).
Summarize the main content of the paper
The authors measured the answers, permissions and billing of 20 relays via public interfaces, finding behavioral differences, tool permission violations and billing tokens of unknown origin.
A measurement study of third-party LLM API relays, answering the question: do relays simultaneously deliver model quality, instruction adherence and billing accuracy.
- Approach: Using the official API as reference, compared 27 knowledge questions, screened 5 types of code and configuration responses, checked tool permissions and input token counts and balances across 20 relays offering GPT-6 Astra or Claude Opus 5, and retested after 11–18 days.
- Answer differences: In the first round, 7/10 low-price and 2/10 same-price or higher-price relays had at least one model flagged for inconsistent behavior; the price-group test p=0.07 did not reach 0.05; the largest gaps include R20 Opus hit rate 0.409 versus official 1.000 (Table III).
- Content and permissions: Among 3,332 responses there was no confirmed injection, but in the first round 4 relays still returned tool calls under tool_choice: none (R11 alone 24 times), while the second round had none; the authors state that content screening cannot replace permission checks.
- Billing: 5 relays over-reported 4,200–60,000 input tokens for identical requests; for 6 relays with checkable balance services, charges matched reported usage × list price to the cent, and R11's surplus tokens entered the bill, but the source could not be verified.
- Changes: Of 35 retested pairs, 7 pairs flipped their flags, on which basis the authors argue that reliability evaluation requires repeated measurement.
- Authors' conclusion: advertised models and prices are insufficient to judge relay reliability; accuracy differences cannot identify the backend model, and billing at the quoted price cannot verify token counts; users should check token counts against list prices, inspect tool calls before execution, and compare repeatedly; future services should provide verifiable mechanisms linking instructions, usage and charges to processing records.