Skip to content

RAG done right: the librarian points, the book gets read

With RAG, a system first finds the relevant passages in your own documents and gives only those to the language model. When the system finds the right passages, the model can base its answer on them.

In short

RAG stands for retrieval-augmented generation: retrieve first, then answer. A language model does not know your business. So for every question the system looks up the relevant passages in your own documents and passes them along. The model then answers on the basis of your data, citing its sources.

Why less context works better

It is tempting to simply give a model everything: the whole case file, all the manuals. But the more text, the greater the chance that the model overlooks the important sentence or mixes things up. In our own measurements, a question that went wrong with a complete case file went right when the model was given only the few relevant facts, and in a fraction of the time.

The librarian

We work with the image of a librarian:

  • Your data is cut into cards: each card holds one statement, with a reference to the original book (the document).
  • When a question comes in, the librarian picks the cards that matter and, if needed, points to the book the model should read.
  • So the model gets a few thousand words instead of tens of thousands.

Searching: not only by meaning

The well-known way to search is by meaning (embeddings): texts that resemble each other sit close together. That works for general questions, but often misses the one fact that counts. In one test, searching by meaning alone put the key card at position 533 out of more than 5,500. With a fixed ranking (first exact names and terms from the question, then the reliability of the source, only then meaning) it came out on top.

What to watch out for

  • Cut your data up so that each piece says one thing, and keep the reference to the source.
  • Combine search methods; do not rely on searching by meaning alone.
  • Have the model name its sources, so a person can check them.
  • Is it about a skill rather than knowledge, such as a programming language or a fixed writing style? Then fine-tuning is often better than RAG.

Would you like advice on RAG? Request a free advice conversation.

Updated on

AI

Personal data in the AI cloud: who is watching?

A conversation with an AI service in the cloud is not private by default. A case in Canada shows what a provider can see and share. What does that mean under the GDPR, in the United Kingdom and in Switzerland?