RAG poisoning, explained with a real example

Retrieval-augmented generation is one of the best tools we have for keeping models grounded in real, current information. It's also a new way for an attacker to reach your model without ever touching your prompt.

If your system retrieves from a source an attacker can influence — a public wiki, a shared drive, a webpage — they don't need to jailbreak anything. They just need to change what gets retrieved, and the model will happily repeat it with full confidence.

We break down a minimal example: a knowledge base article edited to include an invisible instruction, and how that instruction shows up, unprompted, in a completely unrelated customer conversation three steps later.