When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Plugins

Core question: Do third-party AI chat plugins widely deployed on public websites have prompt injection risks, and how do plugin implementations and configurations affect attack success rates.

• Scenario and importance: Website builders use WordPress plugins or commercial plugins to integrate commercial LLMs such as GPT at low cost as customer service chatbots; the authors state that this ecosystem grew by nearly 50% in 2025, covering more than 10,000 websites, but its security status was previously unknown.
• Gaps in existing research: The authors state that prior prompt injection research focused on frontier applications such as coding agents and personal copilots, and was mostly conducted in controlled environments; security research on web plugins focused on traditional vulnerabilities such as XSS and SQL injection.
• Core observation: LLM-level defenses (such as the instruction hierarchy) rely on the role-boundary assumption, and plugin implementations in message history transmission and external content insertion may break this assumption.
• Paper contributions: A large-scale measurement of 17 plugins and 10,417 websites; identification of two types of vulnerabilities, direct and indirect injection; measurement of real system prompts and tool configurations; controlled experiments quantifying the influence of each factor; two prototype defenses and responsible disclosure.
• Threat model:
  • Attacker goal: Inject messages or data into higher-privilege roles such as assistant and system to leak system prompts or subvert developer instructions.
  • Knowledge: Can fingerprint the chat plugin used by a target website (e.g., by analyzing traffic patterns).
  • Capabilities: As an unprivileged user, interact with the chatbot, send network requests, browse websites, and post comments or product reviews where permitted.
  • Victim system and boundaries: Chatbots embedded in public websites through third-party plugins; capabilities from traditional web vulnerabilities such as SQL injection are not considered.

Prompt injection attacks and defenses

• Greshake et al. (2023), Liu et al. (2024), Wu et al. (2024) — respectively demonstrate the harm of indirect prompt injection to real LLM applications, formalize and benchmark prompt injection attacks and defenses, and examine security issues in real LLM systems; the authors note that these works mostly study prompt injection in controlled environments, whereas this paper examines real chatbots on public websites.
• StruQ (Chen et al., 2024), Jatmo (Piet et al., 2024), DataSentinel (Liu et al., 2025) — respectively structured query defense, task-specific fine-tuning defense, and game-theoretic detection; the authors state that these are LLM-level defenses, and this paper focuses on how plugin-layer implementations break their assumptions.
• The instruction hierarchy of Wallace et al. (2024) — trains LLMs to prioritize privileged instructions and has been adopted by commercial models; the experiments in this paper use its role-boundary assumption as the evaluation baseline.

LLM customization techniques

• RAG (Lewis et al., 2020) and its robustness improvements (Xiang et al., 2024), Self-RAG (Asai et al., 2024) — retrieval-augmented generation and its improvements; LoRA (Hu et al., 2022) — parameter-efficient fine-tuning; these are common customization methods for plugins.
• PoisonedRAG (Zou et al., 2024) — a RAG poisoning attack that injects malicious documents into the knowledge base; the authors state that the crawling mechanisms of plugins make such theoretical attacks feasible in practice.

Web plugin ecosystem security

• Ruohonen (2019), Koskinen et al. (2012) — analyze WordPress plugin vulnerabilities, with XSS, SQLi, RCE, and CSRF being the most common; Kasturi et al. (2022) — a large-scale study of malicious plugins; the authors note that these works do not cover LLM-related vulnerabilities.

System prompts and jailbreak

• Mu et al. (2025) — study system prompt robustness; Zhang et al. (2024), Zhang et al. (EMNLP 2024) — prompt extraction; Yong et al. (2023) — jailbreak GPT-4 in low-resource languages.

Baselines and benchmarks: This paper is not a new-method comparison work and has no comparison baselines; the evaluation is based on the authors' self-built local simulated environment and a Common Crawl website dataset. The authors position this paper as a systematic security study of the LLM chat plugin ecosystem, filling the gap between traditional web vulnerability research and frontier LLM application research.

Overall approach: first find and measure websites using the 17 target plugins on Common Crawl, then deploy the plugins locally to analyze communication protocols and configuration options to discover vulnerabilities, and finally turn real-world observations into controlled experiment parameters to systematically evaluate the influence of each factor on attack success rate.

Ecosystem measurement methods

• Plugin selection: Searching "ChatGPT Chatbot" in the WordPress marketplace yielded 116 results, from which 7 WP plugins were screened by criteria such as install count, LLM chat functionality, and local deployability; 10 additional platform-independent commercial (generic) plugins were selected.
• Website discovery: Scanning plugin HTML markers in the Common Crawl August 2024 snapshot (38.3M registered domains) yielded 10,417 deduplicated websites; snapshots from January 2023 to April 2025 were used for longitudinal trend analysis.
• Functionality verification: In September 2024, 200 websites were spot-checked and verified level by level against four questions: online, contains marker, visible, and functioning normally; automated functionality analysis was also performed for P1, P2 and P4.

Vulnerability analysis

• Local deployment: Each plugin was deployed on a local WordPress instance and configured with an OpenAI LLM, and network traffic and configuration options were inspected.
• Direct injection: Checking whether HTTP POST requests contain the full message history without integrity verification, thereby allowing forged assistant or system role messages.
• Indirect injection: Testing whether the crawlers of 15 plugins extract third-party content from pages; for RAG/DC modes, determining the role under which external content is inserted via OpenAI API logs (WP plugins) or a debug-mode probe (generic plugins, majority vote over 5 queries).

Real-world configuration measurement

• System prompt collection: Using the direct injection vulnerability to inject forged messages into 300 P1 websites to extract system prompts; manual determination of whether leakage occurred based on instruction characteristics (at least three sentences matching the characteristics); ethical measures include User-Agent contact information, an opt-out mechanism, and no persistent modifications.
• Tool usage observation: Passively analyzing tool metadata leaked in P1's iframe HTML and HTTP traffic, without actively triggering tools.

Controlled experiment design

• Simulated environment: Replicating the plugins' core functionality (crawling, RAG, custom system prompts, multi-vendor LLMs, tool integration); the knowledge base is 35 product pages crawled from nalgene.com, split according to OpenAI file search parameters (600-token chunks, 300-token overlap) and embedded with text-embedding-3-large.
• Attack tasks: Direct injection tests three items—system prompt extraction (SPE), task hijacking (TaH, inducing the writing of Python code), and tool hijacking (ToH, inducing the Notification tool to send a notification containing a malicious URL to an attacker-specified channel); indirect injection tests context hijacking (inducing the Search tool to search for competitors and display the results).
• Experimental dimensions: 3 system prompts (insecure, hardened, hardened-specific, all taken from real templates) × injection role (system/assistant/user) × 11 LLMs; indirect injection additionally tests 5 content insertion modes (SA/S/A/U/T) × whether the content is wrapped in tags.
• Parameters: Temperature 0.5, each configuration repeated 10 times, reporting the number of successes; the attack payload is a role-override prompt, with 5 variants generated from a seed prompt using ChatGPT and the most effective one selected; Anthropic and Gemini models do not support multiple system messages, so system role injection is not included.

Prototype defenses

• UGC-Buster: Grouping website content by HTML node path, using an LLM to detect user-generated content (signals such as comments, usernames, timestamps), and outputting machine-readable metadata for developers to exclude from RAG.
• Tool instruction hardening: Using an LLM to automatically add anti-hijacking rules to tool instructions (general rules plus tool-specific rules) while keeping functionality unchanged.

Experimental setup

• Models tested: 11 commercial LLMs — OpenAI's GPT-4o-mini, GPT-4o, GPT-4.1-mini, GPT-4.1, o4-mini; Anthropic's Claude Haiku 3.5, Sonnet 3.5, Sonnet 4; Gemini 2.0 Flash, 2.5 Flash, 2.5 Pro (Table 3).
• Datasets: 10,417 chatbot websites obtained from the August 2024 Common Crawl snapshot; the controlled-experiment knowledge base is 35 product pages from nalgene.com.
• Metrics and criteria: Each configuration is repeated 10 times (temperature 0.5), and the number of successes is reported; SPE is judged as output containing a fragment of the real system prompt, TaH as output containing the relevant Python code, ToH as calling Slack-Notify with #general and the specified URL as arguments, and context hijacking as calling tavily-web-search with Hydro Flask as the query and including the returned URL in the reply.
• Real-world measurement: system prompt extraction on 300 P1 websites (leaks manually annotated); third-party content exposure checked on 100 e-commerce websites (up to 5 queries per site); functionality spot-checked on 200 websites.
• Evaluation method: real prompt leaks are judged manually by characteristics; controlled experiments are judged by rules on output content, and the paper does not state that an LLM judge was used.

Main results

Controlled direct injection experiments (Table 3; each cell is the number of successes out of 10; U is user-role injection, S/A is privileged-role injection; Anthropic and Gemini have no S column):

Swipe horizontally to read the table

Model	SPE (insecure) S/A/U	TaH (insecure) S/A/U	ToH (hardened) S/A/U
GPT-4o	10/2/0	9/6/0	10/7/0
GPT-4.1	10/10/0	10/0/0	0/10/0
o4-mini	0/0/0	10/0/0	10/0/0
Sonnet 4	–/0/0	–/0/0	–/10/0
Haiku 3.5	–/0/0	–/10/0	–/8/10
Gemini 2.5 Pro	–/10/8	–/10/0	–/10/10
Gemini 2.0 Flash	–/1/9	–/10/10	–/10/10

(Only some models are listed; GPT-4o-mini, GPT-4.1-mini, Sonnet 3.5, and Gemini 2.5 Flash are omitted; see Table 3 for complete data.)

• Authors' interpretation (roles): The authors state that user-role injection is weaker than assistant or system injection on all tasks, reflecting the effect of built-in defenses such as the instruction hierarchy; SPE success rates under system/assistant injection are about 60%/30%.
• Authors' interpretation (prompts): The authors report that the number of TaH successes with insecure prompts is about twice that with hardened prompts, and hardened is about twice hardened-specific; hardened-specific makes user-injected TaH completely ineffective (0%).
• Authors' interpretation (tool hijacking): The authors state that prompt hardening cannot prevent ToH and in some cases even increases the success rate; the ToH success rate for user injection varies by vendor from 20–100%, and the authors consider system prompt robustness and tool description robustness to be largely orthogonal.

Real-world measurement

• System prompt extraction (300 P1 websites): the success rate of a simple request without injection is 1%; forged system and assistant role injection reach 56% and 55%, respectively; in total, prompts were extracted from 219 chatbots (73%). The authors state that the most common failure reason is model refusal.
• Prompt characteristics (Appendix C): 76% are slight variants of the P1 default template; 97% contain instructions to refuse out-of-scope queries; about 5% instruct the chatbot to collect PII.
• Indirect injection exposure (100 e-commerce websites): 41 allow users to submit reviews, 36 display that content, and 13 (13%) have chatbot replies that include details of user reviews, which the authors state indicates that third-party content has been ingested into the LLM context.

Controlled indirect injection experiments

• Insertion modes (Table 4): The authors report that tool-role insertion (T mode) is the most robust against context hijacking under all settings; non-standard insertion modes increase the attack success rate by up to 5 times (depending on configuration); SA mode can reach a 100% success rate on large models such as GPT-4.1, o4-mini, and Sonnet-4, which the authors attribute to these models being better at following additional instructions in the system prompt.
• Hardening and wrapping: The authors report that prompt hardening reduces the success rate by 20–40%, but hardened prompts with T mode still achieve about 5–10% success; content wrapping provides moderate protection and its effect depends on the model, with OpenAI models benefiting more.
• Vendor differences: The authors state that Gemini models are generally the least robust and Anthropic models the most resilient.

Other ablations and analyses

• Appendix B: a small-scale experiment with two alternative payloads, blunt and ignore+instruct, based on reference [6]; the authors state that the conclusions are consistent with the main experiments, and exploiting plugin vulnerabilities yields up to about a 3-fold increase in success rate; ignore+instruct performs worst.
• Tool instruction hardening defense: repeating the §7 experiments, the authors report a 40–75% decrease in attack success rate.
• UGC-Buster: manually verified on 10 real e-commerce websites (3 pages each), with only 1 false positive, at an average cost of about $0.05 per page.
• Functional deployment rate: among 200 spot-checked websites, 125 functioned normally (62.5%); automated analysis gives 76%/53%/96% for P1/P2/P4, respectively.
• Ecosystem trends: the April 2025 snapshot contains 17,474 chatbot websites; an April 2025 scan found 144 P1 chatbots with tools enabled.

Limitations and future directions noted by the authors

• Boundary of attack capability (§5 Threat Model): The authors explicitly do not consider the attacker's ability to exploit other components of the website or traditional Web vulnerabilities such as SQL injection, and leave strategies that combine classic vulnerability attacks against LLMs to future research.
• Adaptive robustness of the defense (Appendix D Meta-Review, Noteworthy Concerns): The review comments transcribed by the authors point out that the security of the proposed defense against strong adaptive attacks is unclear, and that future work should explore stronger adaptive prompt injection attacks and develop corresponding defenses.
• Limitations of prompt extraction methods (§6.1): The authors state that extraction techniques such as prompt inversion and jailbreak are quickly patched by LLM vendors, and that designing new methods is beyond the scope of this paper, so they instead used direct injection vulnerabilities to collect prompts.
• Limitations of model fingerprinting (§6.3): The authors report that fingerprint queries based on knowledge cutoff dates are unreliable, did not pursue more advanced fingerprinting techniques, and instead covered mainstream models from the three major vendors in the experiments.
• Dataset coverage (§3): The authors state that the Common Crawl dataset may underestimate the full scale of the chat plugin ecosystem, but consider it sufficient to represent key trends.
• Lower bound of the real exposure estimate (§5.3): The authors state that the 13% exposure proportion may underestimate the real risk, because each chatbot interacted with only 5 queries and some chatbots refused to answer.

Scope of experimental coverage

• The models tested are 11 commercial LLMs in total from three vendors, OpenAI, Anthropic and Gemini (Table 3); open-source models are not included.
• Each configuration in the controlled experiments was repeated 10 times at temperature 0.5; the attack payload was the most effective one selected from 5 variants of role-override prompts, and two other types of payload are tested separately in Appendix B.
• The real-world measurement covers only P1 plugins (prompt extraction on 300 sites, e-commerce exposure checks on 100 sites, 144 chatbots with tools enabled), chosen because they are the most widely deployed and have a bug bounty program.
• The indirect injection experiments test only one task, context hijacking, using the Search tool and the Notification tool; Calendly and custom tools were excluded because they could not be evaluated non-intrusively.
• The paper does not report statistical significance tests of attack success in the controlled experiments, nor consistency data for manual annotation of prompt leakage.

A security measurement and attack study targeting third-party AI chat plugins on the web (IEEE S&P 2026). The problem the authors address is: what is the security status of these plugins, used by more than 10,000 websites, against prompt injection attacks, and how do plugin implementations and configurations affect attack success rates.

• Ecosystem measurement: 10,417 websites using 17 target plugins were found from Common Crawl (August 2024), increasing to 17,474 by April 2025; the functional deployment rate among 200 spot-checked websites is 62.5%.
• Two types of vulnerabilities: 8 plugins (used by 8,000 websites) do not validate the integrity of message history, allowing attackers to forge assistant or system role messages for direct injection; the crawlers of 15 plugins do not distinguish website-owned content from third-party content, leading to indirect injection risk, and 13% of 100 e-commerce websites had already ingested third-party content into the chatbot context.
• Real configurations: using direct injection, system prompts were extracted from 219 of 300 P1 websites (forged system injection success rate 56%, versus only 1% without injection); 76% are variants of the default template; 144 P1 chatbots have tools enabled.
• Controlled experiments: on 11 commercial LLMs, the task hijacking success rate is 25–100% when role boundaries are broken, versus 0–25% under role isolation; hardened system prompts prevent task hijacking but are almost ineffective against tool hijacking (user injection success rate 20–100%, varying by vendor); RAG content inserted with the tool role is the most robust, and non-standard insertion modes increase the success rate by up to 5 times.
• Defenses and disclosure: the authors propose two prototypes, UGC-Buster (isolating user-generated content) and LLM tool instruction hardening (success rate reduced by 40–75%); they have disclosed to plugin developers, the most widely deployed P1 has fixed direct injection and switched to hardened prompts, and several others remain unfixed.
• Authors' conclusion: the authors consider that insecure plugin-layer implementations break the assumptions of LLM-level defenses such as the instruction hierarchy, that the risk is growing with the spread of tool calling, and that the community needs to develop hardened designs for tool instructions and pay attention to this ecosystem.