Phantom: A general backdoor attack against retrieval-augmented generation

1. 01What problem does this paper attempt to solve?

   The authors study RAG knowledge base poisoning: a single malicious document causes integrity corruption of the generator's output when a natural trigger word appears.

   Core question: Can an attacker inject a single malicious document into a RAG knowledge base so that it is retrieved only when the user query contains a naturally occurring trigger sequence, and induce the LLM generator to produce integrity corruption such as refusal, bias, harmful content, or privacy leakage?

   • Problem setting: The RAG knowledge base may come from the user's local files or external sources such as Wikipedia and news; the authors believe these sources are difficult to clean and verify for provenance, and malicious users can inject carefully crafted poisoned documents.
   • Shortcomings of existing methods: The authors state that the corpus poisoning of Zhong et al. targets only the retriever and has no adversarial target for the generator; Pasquini et al. do not explicitly optimize the retriever; PoisonedRAG requires injecting multiple poisoned passages for predefined queries.
   • Core observation: The structure of RAG makes it possible to achieve a backdoor poisoning-like effect without tampering with the training process—as long as the poisoned passage enters the top-k retrieval results, the generator's output can be controlled.
   • Proposed in the paper: Phantom, a two-stage optimization framework that attacks the retriever (conditional retrieval) and the generator (alignment breaking) separately, and proposes the Multi Coordinate Gradient (MCG) strategy, which the authors claim converges faster than GCG.
   • Threat model:
     • Goal: Cause integrity corruption, i.e., modify the model output, when the user query contains a natural trigger sequence (e.g., a brand or person name); the attack is agnostic to the rest of the query (universal).
     • Knowledge: The initial assumption is that RAG is based on publicly available pretrained models, and the attacker can directly compute gradients of the retriever and generator; Section 5.1 relaxes this to a black-box transfer setting.
     • Capability: Inject a single malicious document into the user's local file system (e.g., via spam or phishing), without controlling the file system or knowing other documents.
     • Victim system: A locally deployed RAG system similar to Chat with RTX; the authors state that the attack also applies to RAG that pulls documents from the web.

   Collapse this answer

2. 02What related research exists?

   Related work falls into four categories: LLM jailbreak attacks, privacy attacks, poisoning attacks, and RAG attacks; the authors focus on distinguishing their work from PoisonedRAG, BadRAG, and others.

   LLM jailbreak attacks

   • GCG (Zou et al., 2023): combines multiple ideas to improve jailbreak success rates and achieve cross-model transfer; the paper notes that directly applying GCG requires the attacker to access the user query, which is direct prompt injection.
   • Manual and automated jailbreaks: indirect prompt injection by Greshake et al. (2023), and automated heuristic attacks such as Huang et al. (2024), Yong et al. (2023), and GPTFuzzer (2023).

   LLM privacy attacks

   • Training data extraction and membership inference: training data extraction by Carlini et al. (2021), membership inference by Shokri et al. (2017) and others; Nasr et al. (2023) claim extraction is also possible on aligned models.
   • Sensitive information in prompts: system prompt extraction (Zhang & Ippolito, 2023) and user data in in-context learning; this paper studies another kind of sensitive prompt: documents retrieved by RAG.

   LLM poisoning attacks

   • Training-stage poisoning: pretraining backdoors (Hubinger et al., 2024), RLHF human feedback poisoning (Rando & Tramèr, 2024), instruction fine-tuning poisoning (Wan et al., 2023; Shu et al., 2023), and in-context learning poisoning (He et al., 2025). The authors state that the poisoning vector in this paper does not require model training or fine-tuning.

   RAG attacks (main comparison targets)

   • Zhong et al. (2023): corpus poisoning attack; the paper notes that it targets only the retriever and has no adversarial objective against the generator.
   • Pasquini et al. (2024): gradient-based prompt injection; the paper notes that it does not explicitly optimize the retriever, making it difficult to guarantee that the malicious document enters the context.
   • PoisonedRAG (Zou et al., 2024): optimizes both retriever and generator, but the paper notes that it is targeted poisoning—for a predefined query, multiple poisoned passages must be injected to generate a predefined answer.
   • TrojanRAG (Cheng et al., 2024): the paper notes that its threat model differs—it requires injecting multiple poisoned passages and controlling the retriever training process.
   • Tan et al. (2024): untargeted attack, triggered by any input; the paper notes that its bilevel optimization requires more than 1000 iterations, whereas the two-stage attack in this paper requires only 32 iterations, which the authors claim is 30× more efficient.
   • BadRAG (Xue et al., 2024): the paper notes that it has only two main targets, cannot bypass generator safety alignment, some attacks rely on unnatural trigger words, and it requires about 5–10 poisoned passages.

   Baselines and benchmarks: the baseline methods are GCG (compared in Table 7) and a baseline that manually constructs sret by repeating trigger words (Section 6.2.1, Ret-FR 100%); the benchmarks are MS MARCO, NQ, and HotpotQA.

   The authors position this paper as: query-agnostic trigger-based poisoning that unifies multiple adversarial targets in a single framework and requires only a single poisoned passage.

   Collapse this answer

3. 03How does the paper address this problem?

   Phantom constructs padv = sret ⊕ sgen ⊕ scmd: HotFlip optimizes conditional retrieval, and MCG optimizes generator alignment breaking.

   Overall idea: the attacker constructs an adversarial passage padv = sret ⊕ sgen ⊕ scmd (⊕ is string concatenation) and inserts it as a single document in the knowledge base, where sret targets the retriever, sgen targets the generator, and scmd is the adversarial instruction; the passage length is kept within the retriever's passage length to ensure it can be retrieved as a whole.

   Attacking the RAG Retriever

   • Goal: Construct sret so that padv enters the top-k if and only if the query contains the trigger sequence strg. The paper points out that the baseline of simply repeating the trigger word fails (Section 6.2.1, Ret-FR 100%), because the retriever selects passages by overall semantic relevance.
   • Loss modeling: Construct an OUT query set without the trigger word and an IN query set concatenated with the trigger word; the loss is L_ret = 1/n∑_j sim(E_p^∗adv, E_q^outj) − sim(E_p^∗adv, E_{q^outj ⊕ strg}), i.e., minimizing similarity to queries without the trigger and maximizing similarity to queries with the trigger. This stage first optimizes p*adv = sret ⊕ scmd, and sgen is added afterward.
   • Optimization algorithm: HotFlip is used to replace tokens position by position: sret is initialized to mask tokens, and at each step the sequentially selected position is replaced with the candidate token that reduces the loss the most, and this can be iterated for multiple epochs.

   Attacking the RAG Generator

   • Goal: Construct sgen so that the generator executes the adversarial instruction scmd for any query containing the trigger word and replies starting with the target string sop.
   • Loss modeling: For the query set Qin containing the trigger word, the loss is L_gen(T_in) = −∑_j log Pr(t_wj+1:wj+c ∣ t_1:wj), i.e., maximizing the log probability of generating the target string sop.
   • MCG optimization: The authors state that GCG requires large batches and many iterations to converge, so they propose Multi Coordinate Gradient (MCG): in each round, take the top-k replacement candidates for each token of tgen, generate B candidates, randomly update C coordinates for each candidate, and take the one with the lowest loss; after each round, halve the number of coordinates (C := max(C/2, cmin)). The parameters given in the paper: with cmin = 2 and batch size 64, 4–16 iterations can succeed; GCG is a special case with C = 1.

   Instantiating Adversarial Targets

   • Five types of targets: Refusal to Answer, Biased Opinion, Harmful Behavior, Passage Exfiltration, Tool Usage; Table 1 gives examples of scmd and sop for each target (e.g., the refusal target makes the reply start with a fixed apology).
   • Trigger word selection: Common brand or celebrity names are chosen because they often appear in natural queries; Section 6.3 proposes using the linear separability of query embeddings with and without the trigger word to predict trigger word feasibility.

   Success Determination

   • Retriever side: Retrieval Failure Rate (Ret-FR), the proportion of queries where the poisoned document does not enter the top-k; lower is better.
   • Generator side: Report the attack success percentage as defined for each target, i.e., the proportion of queries where the integrity corruption matches the current adversarial target; for the Harmful Behavior target, because output styles vary, the authors rely on manual analysis to determine whether threatening or insulting statements are included (Appendix A.3).

   Collapse this answer

4. 04What experiments did the paper conduct?

   Evaluated 5 target categories on 3 datasets, 3 retrievers, and 7 generators; the highest success rate on open-source models is 100.0%, and the highest black-box success rate on GPT-4 is 25.3%.

   Experimental Setup

   • Generators under test: Gemma-2B, Vicuna-7B, Gemma-7B, Llama3-8B (open-source main experiments); GPT-3.5 Turbo, GPT-4 (black-box); Mistral 7B int4 (Chat with RTX); the paper also mentions Vicuna 13B.
   • Retriever: Contriever, Contriever-MS (MS MARCO fine-tuned version), DPR; BGE and GTE are also tested.
   • Datasets: MS MARCO (the paper states it contains more than 8 million document passages and about 1 million real queries), NQ, HotpotQA; TREC-COVID is used for the trigger word length experiment; the Enron email dataset (about 50,000 emails) is used for the ChatRTX experiment.
   • Metrics: Ret-FR (the proportion of queries where the poisoned document does not enter the top-k) and the attack success rate for each target; passage exfiltration uses edit distance, BERT embedding cosine distance, and longest matching substring length.
   • Protocol: default top-k = 5; the test queries are the 25 in the test set that naturally contain the trigger word; all experiments are repeated with 3 random seeds and the average is reported; the main experiments test 3 trigger words per target.
   • Baselines: GCG (Table 7); sret constructed by manually repeating the trigger word (Section 6.2.1).

   Main Results

   According to Table 2 (Contriever retrieval, MS MARCO, attack success rate, Ours column; Ret-FR is the value for each trigger word):

   Swipe horizontally to read the table

   Target	Trigger word	Gemma-2B	Vicuna-7B	Gemma-7B	Llama3-8B	Ret-FR
   Refusal	iphone	93.3%	88.0%	80.0%	74.6%	9.3%
   Refusal	netflix	86.7%	97.3%	85.3%	93.3%	2.7%
   Refusal	spotify	92.0%	100.0%	66.7%	81.3%	0.0%
   Biased Opinion	amazon	80.0%	89.3%	82.7%	88.0%	9.3%
   Biased Opinion	lebron james	90.7%	81.3%	82.7%	96.0%	4.0%
   Biased Opinion	xbox	88.0%	88.0%	84.0%	85.3%	10.7%

   • Targets not requiring MCG: The authors state that the four generators execute the refusal instruction with high success rates using scmd alone; the authors believe the refusal target is easier to succeed because it is consistent with RAG's tendency not to answer when uncertain.
   • Targets requiring MCG: Biased Opinion for Vicuna-7B and Gemma-7B requires 16 MCG iterations and 16 tokens; the authors report that after adding sgen, the average success rates over the three trigger words increase by 38.7% and 38.4%, respectively (Table 4).
   • Other datasets: On NQ and HotpotQA (Vicuna-7B, Table 3), the refusal success rate is 93.3%–96.0% and Biased Opinion is 62.7%–90.0%, among which Biased Opinion for the basketball trigger word on HotpotQA is 62.7%.

   Harmful Behavior, Passage Exfiltration, and Tool Usage

   • Harmful Behavior (Table 18, Appendix A.3): Insult target (Vicuna-7B, bmw trigger word) No-MCG is 0.0% and with-MCG is 52.0%; death threat target (Llama3-8B) No-MCG is 0.0% and with-MCG is 28.0%. The authors state that the number of MCG tokens and iterations must be increased to 128, and that insults are easier than direct threats, with success determined by manual analysis.
   • Passage Exfiltration (Figure 3): 3 trigger words, 25 queries, 3 repetitions, with metrics of edit distance, cosine distance, and longest matching substring; the authors state that the Vicuna family is more susceptible to exfiltration than Gemma and Llama, and in Figure 3 Vicuna's average edit and cosine distances are lower (the specific values are not given in the text in the figure).
   • Tool Usage (Table 18): Llama3-8B adversarially calls the SEND_EMAIL API on the Email API target (Marilyn Monroe trigger word), with No-MCG at 54.7%, with-MCG at 64.0%, and Ret-FR 0.0%.

   Black-Box Transfer and ChatRTX

   • Cross-generator transfer (Table 10, Biased Opinion): Attacks optimized from Vicuna-7B or Gemma-7B transferred to other architectures; the authors state that non-zero transferability exists; for example, with Vicuna-7B as the attacker and the lebron james trigger word, transfer to Llama3-8B is 96.0% and transfer to Gemma-7B is 62.7% (the same-row diagonal is the attacker model itself).
   • GPT-3.5 Turbo / GPT-4 (Table 12, refusal target, black-box, using only sret + scmd): GPT-3.5 Turbo is 50.7%–68.0% and GPT-4 is 10.7%–25.3%; the authors state that GPT-4's lower success rate is because it has more safety alignment controls.
   • Chat with RTX (Appendix A.5): A black-box attack without knowledge of the retriever architecture, running HotFlip locally only on Contriever (16 epochs, 128 tokens); in the Biased Opinion experiment the poisoned passage becomes top-1 and the instruction is executed (Figure 5); in the passage exfiltration experiment, the poisoned passage is retrieved in 15 of 25 queries, of which 12 execute the exfiltration target, for an overall success rate of 48%.

   Other Ablations and Analysis

   • Number of MCG iterations (Table 5, Vicuna-7B, Biased Opinion): 4 iterations already reach 78.6%–86.7%, 16 iterations reach 81.3%–89.3%, and most of the improvement occurs in the initial iterations.
   • Number of sgen tokens (Table 6): 16 and 32 tokens give the highest success rates (e.g., 89.3% for the amazon trigger word).
   • MCG vs. GCG (Table 7, Vicuna-7B): at 4 iterations MCG is 78.6%–86.7% and GCG is 76.0%–77.3%; at 16 iterations GCG drops to 66.7% on the xbox trigger word, while MCG is 88.0%.
   • sgen position (Table 9): prefix position 81.3%–89.3%, suffix position 40.0%–48.0%.
   • Number of sret tokens and HotFlip epochs (Tables 13, 14, xbox trigger word, Contriever): when sret is 32/64/128/256 tokens, Ret-FR is 82.3%/56%/4%/0%, respectively; at epochs 4/8/16/32, Ret-FR is 5.3%/1.3%/1.3%/1.3%.
   • Other retrievers and trigger word length (Tables 15, 16): DPR and Contriever-MSMARCO have higher Ret-FR than Contriever; BGE and GTE reach 90%–100% Ret-FR for 10-token short trigger words (top-5), but drop to 8%–36% for 22-token long trigger words; cross-retriever transfer is difficult (Contriever→Contriever-MS only 16% enter the top-5, →DPR/ANCE 0%), while cross-dataset transfer is effective (MS MARCO→NQ 100%).
   • Trigger word feasibility (Figure 4): On 198 trigger words, using the distance between the means of query embeddings with and without the trigger word as a separability criterion, the AUC for predicting feasibility = 0.791.

   Collapse this answer

5. 05What are some points that can be further explored?

   The authors point out that cross-retriever transfer is difficult and that modern retrievers are resistant to short trigger words; defenses are only discussed and not empirically tested.

   Limitations and Future Directions Noted by the Authors

   • Difficulty of cross-retriever transfer (Section 6.2.5): Passages optimized for Contriever transfer to Contriever-MS with only 16% entering the top-5, and transfer to DPR and ANCE is 0%; the authors state that transfer across retriever architectures is challenging.
   • Resistance of modern retrievers (Section 6.2.4): BGE and GTE show high Ret-FR for short trigger words; the authors believe this may be because they are trained with contrastive loss on datasets including MS MARCO; the authors state that defenses against adversarial attacks on advanced retrieval models require continued research.
   • High difficulty of harmful behavior targets (Appendix A.3): The number of MCG tokens and iterations must be increased to 128 to obtain consistent jailbreaks, and success determination requires manual analysis.
   • Open challenges for defense (Section 7): The authors state that existing ML defenses (perplexity filtering, paraphrasing) can be bypassed by motivated attackers; certified defenses introduce additional overhead and reduce utility, and designing certified defenses with a better utility/robustness trade-off is the "gold standard"; how system defenses such as data provenance can be applied to RAG is not yet clear.
   • Guardrail extension and future work (Sections 7, 8): The authors state that existing guardrail frameworks might be extended to cover this attack, left for future research; the conclusion states that given the severity of the attack, future work urgently needs to develop mitigation measures.

   Scope of Experimental Coverage

   • There are 8 generators under test: Gemma-2B, Gemma-7B, Vicuna-7B, Vicuna-13B, Llama3-8B, GPT-3.5 Turbo, GPT-4, Mistral 7B int4; there are 5 retrievers under test: Contriever, Contriever-MS, DPR, BGE, GTE (Sections 5, 6).
   • The main experiments use 3 trigger words per target, 25 test queries, and 3 random seeds; the test query count ablation extends to 50 and 75 (Table 11).
   • The black-box experiments on GPT-3.5 Turbo and GPT-4 test only the refusal target, and use only sret + scmd with no gradient optimization (Section 6.1.10).
   • Defenses were not experimentally evaluated: Section 7 only discusses three directions—ML defenses, output filtering, and system defenses—and reports no attack success rate under any defense.
   • The paper does not report specific values for the attack's running time (it only mentions at Tables 13, 14 that doubling the number of tokens or epochs doubles the optimization time).

   Collapse this answer

6. 06Summarize the main content of the paper

   Phantom uses a single poisoned document to hijack RAG output under a natural trigger word; the success rate is high on open-source models, and the black-box transfer success rate drops noticeably.

   Phantom is a single-document backdoor poisoning attack framework targeting RAG systems: the attacker injects one malicious document into the knowledge base so that it is retrieved only when the user query contains a natural trigger sequence (such as a brand or a person's name), and induces the generator to execute adversarial instructions.

   • Problem to solve: RAG knowledge base sources are untrustworthy; the authors study an attack that requires no tampering with training and only the injection of a single document to cause integrity violations when the trigger word appears, covering five target categories: refusal, biased opinion, harmful behavior, passage leakage, and tool calling.
   • Method highlights: two-stage optimization—HotFlip optimizes the retrieval string sret to achieve conditional retrieval (enters top-k only when the trigger word is present); the authors' proposed MCG optimizes the generation string sgen to break generator alignment, which the authors claim outperforms GCG under a small iteration budget (Table 7).
   • Main results:
     • On MS MARCO, the success rates of refusal and biased opinion for four open-source generators are 66.7%–100.0%, with Ret-FR 0.0%–10.7% (Table 2); biased opinion for Vicuna-7B and Gemma-7B requires 16 MCG iterations.
     • Harmful behavior is harder: insults 52.0% (Vicuna-7B), death threats 28.0% (Llama3-8B), requiring 128 iterations (Table 18).
     • Black-box transfer to GPT-3.5 Turbo is 50.7%–68.0%, and to GPT-4 is 10.7%–25.3% (Table 12, refusal target).
     • The black-box passage leakage attack success rate against the production system Chat with RTX is 48% (Appendix A.5.3).
     • BGE and GTE reach 90%–100% Ret-FR for short trigger words, but drop to 8%–36% for 22-token long trigger words (Table 16).
   • Authors' conclusion: the authors state that this attack vector is severe and that mitigation measures urgently need to be developed; existing ML defenses can be bypassed, certified defenses and system defenses each have overhead and deployment challenges, and designing robust defenses that balance utility is an open problem.

   Collapse this answer

Related papers