Prompt-in-Content attacks: hijacking LLM behavior using uploaded input

Natural-language instructions in uploaded files get concatenated into the prompt and are executed as intent during the user's normal task.

• Where it occurs: The authors state that applications such as summarization, Q&A, and drafting already support full-file upload. Platforms often concatenate the document, the user query, and system instructions into a single prompt; without clear separation and source tracking, sentences in the file can be treated as user intent.
• Existing gaps: The authors state that direct injection and indirect injection have been studied, but the uploaded-document vector has received little attention. This path requires no API, jailbreak tool, or plugin; it goes only through the standard user interface.
• The authors' observation: A third party writes instructions into a shared file that looks normal to the human eye; after an unknowing user uploads it and issues an ordinary task such as summarization, the embedded instruction can be triggered. The authors state that consequences include suppressing output, replacing content, directing to external links, manipulating tone, and attempting to carry away sensitive information through constructed output.
• What this paper proposes: The authors define and formalize the prompt-in-content attack, design four target types, verify them on the default web interfaces of mainstream platforms, and discuss concatenation, lack of isolation, and mitigation directions.
• Threat model:
  • Victim workflow: A benign user uses a commercial LLM service that supports uploads, submits DOCX, PDF, or plain text, and attaches a natural-language task such as summarization, rewriting, or Q&A.
  • System behavior: The platform concatenates system instructions, the user query, and the document into a unified prompt, with no strict source isolation or trust boundary.
  • Attacker: A third party prepares the file in advance, with the instruction embedded in content that looks normal; the user obtains the file through normal channels such as an internal library, public sources, or collaboration, and does not know there is an instruction in it.
  • What cannot be done: The attacker does not directly operate the model interface, does not control the system prompt or backend API, and does not use formatting tricks, jailbreak tokens, or adversarial strings.

Section 2 organizes related work by file workflows, prompt injection, and document content attacks. The authors state that the setting of "adversarial instructions embedded in documents, delivered by benign users via upload" has not been systematically evaluated.

• File-based interaction: The paper links the ability to follow natural-language instructions to the Transformer architecture and cites Huang et al. (2023)'s long-context survey. Kasneci et al. (2023) and Chang et al. (2024) discuss LLMs in education and other scenarios; Chen et al. (2024) and Lai et al. discuss document-dense domains such as finance, healthcare, and law. These citations are used to establish that "upload a file then summarize or extract" is already a common interface; the paper does not treat them as an attack baseline.
• Direct and indirect injection: Liu et al. (USENIX Security 2024) formalize and benchmark prompt injection; Liu et al. (2024) do automated universal injection; Wang et al. (2023) use injection to disrupt crowdsourced surveys. The paper states that classic injection assumes the attacker directly inserts overriding instructions into the interface. Yi et al. (2023) benchmark indirect injection entering the model via search results, web links, or memory recall; Hines et al. (2024)'s spotlighting and Wang et al. (2024)'s FATH do input spotlighting and certification-based test-time defenses, respectively. The paper states that such attacks mostly assume the adversary controls the user query or retrieved web pages. The introduction also groups Mudarova and Namiot (2024), Zhang et al. (2024)'s goal-directed generative injection, and Wu et al. (2024) under existing direct/indirect injection, and states that the uploaded-document vector has received little attention.
• Documents and multimodal content: Zhang et al. (2024)'s imperceptible content poisoning misleads summarization models in HTML and PDF; He et al. (2024)'s EvilPromptFuzzer injects jailbreak-style content into images to bypass moderation. DocVoyager (Lee et al., 2025) and GVDIE (Qi et al., 2024) show document understanding interfaces becoming more complex; the paper states they rarely consider the security implications of mixed prompt construction.
• Baselines and benchmarks: The paper does not compare against named attack methods and does not use public injection benchmarks. The evaluation object is whether seven web services execute or ignore the authors' self-embedded instructions (Table 1).

The authors summarize the distinction in three points: only the service's default file upload is used, not chat APIs or retrieval-augmented pipelines; the attacker does not directly access the model, relying on victim upload for delivery; the instructions are unobfuscated natural language without formatting tricks, yet are still executed due to concatenation and lack of boundary control. The authors state that this threat falls outside most existing defenses.

The overall approach is a prompt-in-content attack: a third party writes natural-language instructions into a file that will later be uploaded, exploiting the platform's prompt concatenation so that the user's ordinary task triggers these instructions.

Setting and delivery

• Input composition: Commercial services concatenate system instructions, user queries, and uploaded documents into one prompt. The authors believe the boundary between executable parts and data is therefore ambiguous (Section 3.3).
• Who delivers: The attacker does not touch the model. Files reach the user via internal libraries, public sources, or collaboration; the user uploads and issues a summarization, rewriting, or Q&A task.
• How embedded: One or more instructions written as natural language resembling system directives, section headings, or comments, mixed into the body text (Section 3.2, Figure 2). Figure 3 illustrates different positions; the authors state that middle insertion is especially effective because it is less visible to users and often preserved by preprocessing. Later experiments use only the middle position.
• Explicitly not done: No modification of system prompts, no backend API calls, no jailbreak tokens, adversarial strings, or format exploitation.

Four attack types

Each corresponds to an adversarial goal, and each uses one natural-language instruction in the experiments (Section 3.4, 4.3). The instruction text is not reproduced.

• Task suppression: Make the model ignore the original task and return a fixed refusal instead of processing the document.
• Output replacement: Do not process the file; output only an attacker-specified irrelevant sentence.
• Behavior redirection: Do not complete the summary; instead suggest the user visit an external resource. The authors link this to phishing or competitor sites.
• Framing manipulation: Describe the document as outdated or useless, and make the output carry specified misleading claims.

Section 4.5 extends redirection one step further: extract content from sensitive information appearing earlier in the conversation, place it in an outbound link, and output only the sentence recommending the visit.

Trigger and the root cause given by the authors

• Process: Prepare the file; the user uploads and submits a generic task, the example in the text being to summarize the document; the embedded sentence enters the model input; the model may override, suppress, or redirect the original query.
• Root cause: The authors attribute it to plaintext concatenation and lack of source isolation, with no trust distinction between documents, system instructions, and user messages. The authors state that the model treats all text as potentially executable rather than passively parsing data.
• Why hard to see from the system side: The authors state that these behaviors do not depend on malformed input or external tampering and still look like legitimate user requests.

Success criteria and mitigations discussed

• Success criteria: Whether the output follows the embedded instruction. Table 1 records ignoring the instruction as blocked and following it as a successful attack. The paper does not use a separate evaluation model or numerical thresholds.
• Mitigations not implemented: Section 5.2 suggests standardizing prompt assembly interfaces, separating inputs by source, cleaning imperative text in anomalous positions, and not rendering unvalidated links directly in the front end. The contributions also mention input boundary constraints and semantic-level filtering. These are discussion, not implemented and evaluated defenses.

Experimental setup

• Services tested: ChatGPT 4o, Claude Sonnet4, Gemini 2.5 Flash, Perplexity, Grok 3, DeepSeek R1, Kimi, seven in total. All via the standard web interface, no API, third-party plugins, or jailbreaks (Section 4.2).
• Files: All Word (.docx), the body looks normal, with one instruction inserted in the middle. The authors state that most services convert uploaded files to plain text, so the attack does not depend on format; the paper does not report results for other formats.
• User task: After uploading, submit a generic prompt; the example in the text is to summarize the document, then see whether the output reflects the embedded instruction.
• Variants: Task suppression, output replacement, behavior redirection, framing manipulation. One natural-language instruction each; the authors state they deliberately kept them simple and not targeted at specific platforms (Section 4.3).
• Metrics: Binary results in Table 1, no numerical thresholds, no separate evaluation model. Legend: ✓ is blocked (embedded instruction ignored), ✗ is successful (embedded instruction followed).
• Scale: The paper does not report sample size per cell, number of repetitions, or random seeds. It only mentions that Kimi's answers varied on the same tests, without giving a count.

Main results

The table below transcribes the ✓/✗ from Table 1 as blocked/successful.

Swipe horizontally to read the table

Platform	Task suppression	Output replacement	Behavior redirection	Framing manipulation
Grok 3	Successful	Successful	Successful	Successful
DeepSeek R1	Successful	Successful	Successful	Successful
Kimi	Successful	Successful	Successful	Successful
Gemini 2.5 Flash	Blocked	Blocked	Successful	Successful
Perplexity	Blocked	Blocked	Blocked	Successful
Claude Sonnet4	Blocked	Blocked	Blocked	Blocked
ChatGPT 4o	Blocked	Blocked	Blocked	Blocked

• The authors state that this type of attack is effective against most platforms; the text names Claude Sonnet4 and ChatGPT 4o as ignoring the embedded instruction across all four types, while Grok 3, DeepSeek R1, and Kimi executed every instruction in the tests.
• The authors believe that very short embedded instructions can also bypass content cleaning and change behavior, and they believe this points to a lack of input boundary constraints. The authors state that in Figure 4, Grok 3 did not summarize but instead output attacker-specified content.
• The authors state that the real-world implications of each attack type differ: suppression can leave downstream processes without a response, replacement injects irrelevant or fabricated content, redirection brings phishing risk, and framing manipulation distorts summary-based judgments. The paper does not separately explain why the framing manipulation column had fewer blocks.

Sensitive information exfiltration extension

• What was tested: Extended behavior redirection to extract sensitive information from earlier user inputs in the same conversation (the example in the text is a password), place it in an outbound link, and output only the sentence recommending the visit (Section 4.5).
• Results: The authors state that in Figure 5, Grok 3 followed the instruction, placing the previously provided password into a link disguised as a helpful redirect. The paper does not report results for the other six platforms on this extension.
• Authors' interpretation: The authors state that this approach uses only natural language and plaintext URLs, without complex formatting, making it harder to detect and more like normal advice; they also state that attackers could switch to obfuscated or encoded payloads. The test used plaintext and was conducted in a controlled environment. The authors believe this type of attack could develop into more covert, automated exfiltration.

Output variation under identical input

• What was tested: The authors observed that some platforms behaved inconsistently on identical inputs, naming Kimi.
• Results: Answers varied on the same test cases. The paper does not report the number or proportion of variations.
• Authors' interpretation: The authors believe this may come from decoding randomness or input preprocessing. They also state that DeepSeek sometimes adds disclaimers and that adjusting wording might avoid them, but the paper evaluates the default condition and does not provide results after tuning.

Other ablations and analyses

• Embedding position: experiments used only the middle; the authors state the middle is less visible and often preserved by preprocessing, with no positional comparison numbers (Section 3.3, 4.2, Figure 3).
• Instruction wording: no success rates reported for platform-customized wording versus default wording (Section 4.3).
• File format: all experiments used .docx; no PDF or plaintext comparison reported (Section 4.2).
• Interaction turns: only single-turn reported; no multi-turn results (Section 5.3).
• Defense suggestions: none of the four in Section 5.2 have experimental numbers.
• Exfiltration payload: only plaintext demonstration reported, no results for encoded variants (Section 4.5).

Limitations and future directions noted by the authors

• The study is limited to single-turn interactions on public web interfaces (Section 5.3).
• Multi-turn persistence, long-term memory leakage, and account-bound prompt poisoning were not examined (Section 5.3).
• All exfiltration tests were conducted in a controlled environment, with no exposure to real user data and no deployment of malicious infrastructure (Section 5.3).
• Future work should design defense-aware prompt architectures and develop automated detection tools for embedded instructions (Section 5.3).
• It is necessary to evaluate whether alignment-tuned models can reliably distinguish task instructions from embedded adversarial content (Section 5.3).
• It is necessary to examine whether safety prompt design guidelines can be consistently adopted across services, especially after LLM APIs enter document workflows (Section 5.3). Section 6 also states that stronger isolation between trusted instructions and untrusted input is needed, along with more transparent, standardized prompt construction.

Experimental coverage

• Services tested were ChatGPT 4o, Claude Sonnet4, Gemini 2.5 Flash, Perplexity, Grok 3, DeepSeek R1, Kimi, all using the default web interface, without API, plugins, or jailbreaks (Section 4.2, Table 1).
• One natural-language instruction per attack type; files were .docx with middle embedding; the user side was a generic summarization-type task (Section 4.2–4.3).
• The exfiltration extension only states that Grok 3 followed the instruction and placed the previously mentioned password into a link (Section 4.5, Figure 5).
• The paper does not report sample size per cell, number of repetitions, temperature, or agreement between automated evaluation and human evaluation.
• Section 5.2 lists four categories of recommendations—assembly interfaces, source separation, content cleaning, and output rendering—and the paper does not report corresponding experimental numbers.

The prompt-in-content attack places instructions in files that will be uploaded, turning the user's ordinary document task into a trigger.

• Problem: LLM services that support uploads often concatenate system instructions, user queries, and documents together. The authors believe that natural language in files can be treated as executable intent, without the user knowing and without the attacker operating the interface or API.
• Approach: A third party embeds in advance a single unobfuscated natural-language instruction. The four target types are suppressing the original task, replacing the output, directing the user to external resources, and manipulating the document's narrative framing. Success is judged by whether the model follows the instruction.
• Main results: On the default web interfaces of seven services, with a single mid-document instruction in .docx files (Table 1), Grok 3, DeepSeek R1, and Kimi succeeded on all four types; Gemini 2.5 Flash succeeded on redirection and framing manipulation and blocked the first two types; Perplexity succeeded only on framing manipulation; Claude Sonnet 4 and ChatGPT 4o blocked all four types.
• Exfiltration: The authors state that in the redirection extension, Grok 3 wrote a previously entered password from the conversation into the recommended link (Figure 5). Results for this item on the other platforms were not reported.
• The authors' conclusion: The cause is prompt concatenation and lack of source isolation. The authors state that defenses across platforms are inconsistent and, where present, are ad hoc and undocumented; trusted instructions need to be separated from untrusted input, and prompt construction needs to be standardized. The authors also scope the evidence as the single-turn public web interface.