RAG Simplified: How to Give Your LLM a Long-Term Memory
Fine-tuning a model every time your data changes is costly, slow and forgets nothing on command. RAG stores knowledge in a vector database and retrieves only the most semantically relevant chunks at query time, injecting live context into the prompt without retraining.
Jonathan MukhobeAI and ML Engineer
Retrieval Augmented Generation (RAG)
RAG is a technique that exerts influence on a model's response through the integration of our own data. Rather than baking knowledge directly into model weights, it retrieves the most relevant information at query time and injects it into the prompt, giving the model an accurate, up-to-date context without ever retraining.
Why Fine-Tuning Is Often the Wrong Answer
One possible way to influence a model with your own data is through fine-tuning and incorporating it into the training process. This approach enables the model to construct responses taking into account the provided data. However, often this may not be the optimal solution for various reasons:
- Cost and effort involved in fine-tuning a model, including computer resources and GPU access.
- Information is not valid forever. It might need to be changed, which calls for fine-tuning every time the information is modified.
- There is no mechanism to make the model forget what it has learned.
- In a scenario where information increases periodically, fine-tuning is not the best approach.
- A model can receive data in two ways: through fine-tuning and through the system prompt. But the information we can give to the prompt is limited.
Note
Solution: We can use RAG such that the model receives necessary information for the specific question the user provides, pulled from a knowledge base at query time.
Steps Followed by a RAG System
Knowledge Storage
The first step is to store the information the model can draw upon. This knowledge base can include structured data (like databases), unstructured data (like text documents), or even code. We use a vector database to store this information.
Question Reception
The RAG system receives the user's question or request. This could be in natural language, just like a question you would ask a person.
Information Retrieval
This is where the "Retrieval" in RAG comes into play. The system selects relevant information to the user's question from the stored data. In a vector database, this involves vector comparison to retrieve related information.
Vector Comparison is how the system figures out which stored information is most similar to your question, using math behind the scenes to compare meaning. It matches the meanings of your question to the most relevant stored documents, not by keywords, but by how close their meanings are in a mathematical space.
Prompt Construction
A prompt is constructed using the relevant information and the user's question. This prompt is designed to effectively communicate the context of the user's query to the language model. In this way, not only is the model provided with information to construct its response, but hallucinations are also reduced since the model does not have to invent information it does not know.
Calling the Model
The prompt is passed to the language model. The model uses the context provided in the prompt to generate a response.
Response from the Model
The model's response is presented to the user. If the RAG system has been constructed correctly, this response should be relevant and grounded in the retrieved information from the knowledge base.
How Vector Databases Work
Of all the steps mentioned, one is particularly important and sensitive: searching for relevant information to answer the user's question. This is where vector databases come into play, assisting in selecting which information from the database is relevant to what the user asks.
Vector databases store information as numerical representations called vectors. So it is necessary to transform text into vectors before storing it.
Tokenization
The first step is to tokenize the text. Tokens are the smallest units of text that are meaningful to the model. There are various types of tokens, including words, subwords, characters, and byte-pair encodings. The choice of token type depends on the specific use case and the language being modeled.
Vectorization
Tokens are converted into vectors, which are numerical representations of objects in a continuous vector space. They capture the semantic meaning or properties of those objects in a way that can be efficiently searched. A vector represents a point in a multidimensional space, anywhere from roughly 100 to 4,000 dimensions in practice.
Embeddings and Semantic Proximity
The vector, also known as an embedding, captures the semantic meaning of the stored text. The magic lies in how these embeddings are generated: words with similar meanings have vectors that are closer together in this multidimensional space than words with different meanings.
For example, "cat" and "kitty" have very similar meanings, even though the words themselves are quite different letter by letter. Their embeddings will be numerically close because the model has learned they appear in similar contexts.
Distance Calculation and Search
Once text is converted, it is possible to calculate the difference between one vector and another, or search for vectors that are closer to a specific one. That is how it is possible to find similar texts to a reference one. Mathematically, there is not much difference between calculating the distance between two points whether they are in two, three, or any number of dimensions.
Where Do Embedding Numbers Come From?
The numbers in vector embeddings do not just appear by chance. They come from machine learning models that have been trained on huge amounts of text (or other types of data like images, audio, etc.).
Note
Think of it like teaching a child through examples. Suppose you give a child thousands of examples of how words are used in real sentences: "The cat climbed the tree," "The kitten purred softly," "A lion is a big cat." Eventually the child starts to understand relationships between words. This is basically what deep learning models do: they read millions of examples, learn patterns, and turn each word, sentence, or document into a vector of numbers that captures those learned relationships.
Behind the Scenes: Neural Networks
These embeddings are created by models like:
- Word2Vec
- GloVe
- BERT
- OpenAI's models, etc.
These models use neural networks to analyze text contextually and assign each word (or sentence) a place in a high-dimensional space. So instead of saying "cat and dog are related because they appear together," they say "let me place cat and dog in a vector space so they end up close together because the model has seen them used in similar ways."
When you hear about embedding models or vectorization tools like OpenAI Embeddings API, Hugging Face Transformers, SentenceTransformers, or spaCy, these are tools that generate those vectors from text, making it ready to store and search in a vector database.
Summary: End-to-End RAG Flow
In summary, this is how Retrieval Augmented Generation works:
- The text to store is tokenized and converted into embeddings, which are then stored in the vector database.
- The user's question is also tokenized and converted into an embedding.
- The vector database is queried to find the embeddings that are closest to the user's question embedding.
- These embeddings are converted back into text and returned as context for the model.
Here is a brief summary of the process text undergoes before reaching a language model. First it is tokenized, meaning it is divided into small parts. Then it is converted into vectors, which in the world of large language models are known as embeddings. This vector, containing numbers, is passed to the model, and it generates another vector as output. The output vector undergoes the reverse process and is converted into text, which is what we see as a response.
The key is to utilize selected text related to the question and build a powered prompt with the information and the user request. In this way, the model has the necessary information to build a proper response. To select the text to use to enrich the prompt, a vectorized database is employed, enabling a search based on vector similarity. This ensures that the model has information relevant to what it needs to respond.
The important thing is to understand that embeddings are numerical representations capable of capturing the meaning of sentences, and that they allow you to perform simple vector operations such as calculating the distance between them.