PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

TL;DR

We propose PoisonedRAG, a knowledge corruption attack that injects a few malicious texts into the knowledge database of a RAG system to induce the LLM to generate an attacker-chosen answer for a target question.

Problem

RAG mitigates LLM hallucinations and lack of up-to-date knowledge by leveraging an external knowledge database, but it assumes the database is trustworthy. An attacker can exploit this by inserting malicious texts to manipulate the LLM's output, revealing a security vulnerability.

Approach

We formulate the attack as an optimization problem to find a set of malicious texts that cause the LLM to output a target answer for a target question. For black-box settings, we use gradient-based approximate optimization; for white-box, direct gradient descent. The malicious texts are generated as natural sentences to evade detection.

Results & Contribution

With only five injected texts among millions, PoisonedRAG achieves a 90% attack success rate. Existing defenses (e.g., perplexity filters, similarity-based detection) are insufficient. This work systematically analyzes the security vulnerability of RAG systems for the first time and highlights the need for new defenses.