RAG
Retrieval-augmented generation: a model reads document chunks before it answers. When it fits, where it fails.
AI
- Category
- AI
In practice
RAG means a model gets fragments of your documents before it answers, instead of relying on memory. I reach for it when answers have to rest on data that changes and a source can be named. When the knowledge is small and rarely changes, I paste it into the prompt and skip the index.
What is RAG?
RAG, short for retrieval-augmented generation, is an approach where a language model gets, together with the question, information picked for that question. The term comes from a 2020 research paper that combined knowledge stored in a model's parameters with an external memory searched while answering. Put simply, the model is given specific information relevant to the prompt.
Selecting the information usually relies on embeddings, a way of recording the meaning of text as a vector in which similar content sits close together. A document is split into chunks, each chunk is turned into an embedding and stored in a database that supports vectors, such as PostgreSQL with the pgvector extension. At question time the system looks up the closest chunks and adds them to the prompt. Embeddings are one method: context can come from any source. The whole thing runs on the server, for example in an API route in Next.js, because documents and keys must not reach the browser.
How I approach it
I start by settling whether RAG is needed at all. Every part of the pipeline needs maintenance, so I build it only when a simpler way is not enough.
The simpler way first
I check whether the knowledge fits in the prompt. If it is a few pages that change now and then, I paste them in whole and build no index. Fine-tuning I keep for the style and format of answers: knowledge is easier to change in a document than in a model.
Chunks with a named source
I cut a document along its structure, meaning headings and paragraphs, not every fixed number of characters. I record where each chunk comes from. The larger the embedding input, the lower its quality, so a chunk cannot be a whole page.
Test questions and refreshing
I collect questions whose answers we know and check that the right chunk reaches the prompt. I also settle who refreshes the index after a document changes, because a stale chunk keeps being found until someone removes it.
When it fits, and when it doesn't
RAG fits questions whose answer lives in documents, not in what the model knows.
A good fit when
- A knowledge base, terms or documentation that change more often than a model.
- There are too many documents to fit in one prompt.
- The answer should name its source, so a person can check it.
- Users should see different documents, so access is filtered before a chunk reaches the model.
Not a fit when
- The knowledge is a few pages and rarely changes: pasting it into the prompt is enough.
- The questions are about numbers in a database, such as an order status: a tool that asks the database directly does that.
- The documents contradict each other or are out of date: RAG only surfaces that mess faster.
- The goal is the tone and format of answers, not knowledge: I start with the prompt.
When a model should query your system instead of reading documents, I use tools in the AI SDK. When the goal is search, not a text answer, I build semantic search instead.
Frequently asked questions
Should an assistant answer from your documents?
Tell me which documents you have and how often they change. I will say whether pasting them into the prompt is enough or you need an index. What the offer covers is on the [[service:ai|AI integration]] page.