PoisonedRAG: A knowledge poisoning attack against large language model retrieval-augmented generation (RAG)

AI summaryResearchers from Pennsylvania State University and Illinois Institute of Technology propose PoisonedRAG, which injects a small amount of malicious text into a RAG system's knowledge database to induce large language models to generate specified answers to attacker-chosen target questions; the authors call it the first knowledge poisoning attack against RAG. The method splits malicious text into two sub-texts, respectively satisfying the retrieval condition of being hit by the retriever and the generation condition of misleading the LLM into generating the target answer, and is constructed in both black-box and white-box settings. On the NQ, HotpotQA, and MS-MARCO datasets and 8 LLMs including GPT-4, PaLM 2, and LLaMA-2, injecting 5 malicious texts per target question into a knowledge base containing millions of clean texts achieves an attack success rate of up to 90%, reaching 97% on PaLM 2 in the NQ black-box setting; it is also effective against advanced RAG schemes such as Self-RAG and CRAG, Wikipedia-based chatbots, and ReAct Agent.