RAG: Bridging the Gap Between AI and Real-Time Data

Today we often hear about incredible AI advancements that promise to make our lives easier. But besides developing and improving new AI models, we also find new ways to use them and utilize their full potential. One exciting feature of LLMs AI Retrieval-Augmented Generation, or RAG for short.

This system connects real time data to the power of AI models. And knowing how RAG works really raises the ceiling of your expertise as an AI engineer. So, in this opening article let's make sure to cover all the core fundamental concepts. And in the upcoming articles we will build exciting applications to apply our knowledge in practice.

Large language models (LLMs) generate text by predicting the most probable next word, but without access to real-time or domain-specific information, they produce errors, outdated answers, and hallucinations.

An AI model trained on data up to 2023 has no clue what happened last week in financial markets or the latest updates in medical research. That gap between general knowledge and real-world applicability is where Retrieval-Augmented Generation (RAG) can bridge the gap.

RAG fundamentally changes how AI systems operate. Instead of pulling answers from a static memory, RAG-enabled models retrieve relevant external information before generating responses.

Let’s also try to break down the name. Because the name itself (RAG) actually contains an explanation about this system. “Retrieval” refers to the part of the system that looks for information from a collection of documents or sources. Then, “Augmented Generation” uses that information to create more accurate and helpful responses. Essentially, the system searches for relevant data and combines it with a language model's natural writing ability.

For a better understanding, let’s imagine that you want to know the latest price of the Apple stocks. If you try to ask ChatGPT about it – it’s not gonna really help you, because it doesn't know about current state of the outside world. So, to overcome this limitation, you can build a system which initially finds the latest stock prices on the Internet. And then would pass the found and retrieved information to the LLM for processing. On its end, the LLM would receive both the question and the retrieved data that contains the answer. Eventually, it uses all given information to reason and produce an answer.

Now let's cover a general workflow and some main moving parts of RAG system. In other words, the actions it performs on the backend side to give us an answer. This is just a general overview and we will cover all its core parts in more details in upcoming sections. So, the data flow looks something like this:

If some parts are not clear for now – fear not. Soon you will understand it all!

Context windows refer to the amount of text a language model can consider at once during a single interaction. As the size of these windows grows, language models can process and remember more information within a single session. This expansion provides models with a larger "short-term memory," allowing them to maintain coherence over longer dialogues or textual analyses. Latest popular LLM have context window around 128 000 tokens. Which is big enough to let you place the multiple Harry Potter books within it.

Then why RAG still matters if you just can already include large amount of the information within the LLM’s context windows? RAG does not replace the need for longer context windows; rather, it complements them. While extended context windows allow language models to access more information at once, RAG provides the critical function of sourcing that information from up-to-date and relevant external documents or databases. This means RAG can fill the context windows with pertinent facts and data that the model might not inherently possess, making the generated output both factual and contextually rich.

Even with large context windows, the practical limit of the content that can be held in short-term memory means that not all information can be included simultaneously. RAG allows for the prioritization of the most relevant data, optimizing the quality and relevance of information populating the context window.

In summary, as context windows grow, they enhance the capacity of language models to remember and process information within a session. However, RAG plays a crucial role by providing fresh, external content to populate these windows, ensuring that responses are both up-to-date and accurate. Together, they form a powerful synergy, strengthening the overall capability of AI systems to deliver high-quality, data-driven responses.

Essentially, embeddings are the way you transform your data into numeric representation, which is called vector. In the simplest terms, a vector is an ordered list of numbers. In the context of embeddings, a vector represents the meaning of a word or a piece of text in a numerical form.

Each position in this list corresponds to a specific dimension in the vector space. For example, a vector could be [1.2, -0.9, 3.4], with 3 dimensions. Or [1.2, -0.9, 3.4, 1.0, 2.5], with 5 dimensions and so on. Theoretically, we can add as many dimensions to our embedding, as we need.

Why amount of dimensions matter? The rule of thumb here is, the more dimensions we have the more accurately we can operate with data.

These vectors are constructed in such a way that meaningful relationships between words can be captured by their geometric arrangements in space.

The distance between vectors corresponds to the semantic similarity between the words they represent. We measure distance in this case by comparing numbers of each vector. As an example would take 2 one-dimensional vectors. Here’s an example when vectors would be “close” to each other: [0.97] and [0.98]. And contrary to that, this is how they can be “far” from each other: [0.97] and [3.55].

Not to get too much in math, the simple explanation is – the closer the vectors (numbers) are, the more similar the words are in meaning.

Imagine a graph in multiple dimensions where each word is a point on this graph. Words that are similar in meaning are located near each other, while dissimilar words are farther apart.

By embedding words into multi-dimensional space, vectors capture subtle differences and similarities that might not be obvious in text, like analogies ("king" vs "queen").

In an embedding space, words that share similar meanings are positioned close to each other. So, when you add more data into the embedding, this naturally leads to the formation of semantic "clusters," where related words and concepts group together.

When we search for information or answer a question, the system essentially looks for the closest semantic clusters in the embedding. And we would grab this “clusters” since it would most likely to contain the answer or context relevant to our query.

The main reason why it’s important for us, is that by comparing word’s meaning rather than just their literal form we can get much more accurate retrievals and searches. So retrieval can differentiate between “Apple” as the fruit, and “Apple” as the company. And this opens up for us a whole new world of possibilities how we can interact with the content.

Without embeddings, retrieval would rely on simple keyword matching, which is less effective for understanding context and meaning.

Not to overwhelm you, in this article we're going to use embeddings mainly for performing searches. But they are also can be used to do much more than that.

I highly recommend visiting the official OpenAI documentation page, where they provide some additional explanation and go over another cases where embeddings can be used. Here’s the 

By the way, so you know, we can embed not only text but also images. But for the simplicity we will stick with the text examples.

Let's cover the high-level process of how we transform text into embeddings.

Everything starts with the embedding model. This is an actual tool which helps us to transform our text into vectors. And those models are provided by all the big tech AI companies. Let's take for example OpenAI. The same way how do you use an OpenAI’s API to access larger language models (e.g. gpt-o4), similar to that you can use OpenAI’s API to access embedding models. And Unlike LLM, embedding model are usually much-much cheaper to use.

As we can see, we’re getting text input, sent it to the OpenAI’s embedding model (

) and getting back a response. Where response would be a vector representation of our initial input.

So, this is how we use embedding model to represent our data in vector format.

Now it’s time to talk about dimensions a little bit more. As you probably saw, when creating an embedding, we also have passed 

 parameter. Since, as we mentioned above, vectors are high-dimensional numeric data, we can also specify amount of dimensions which generated embedding should have. And because we specified 256 dimensions, we’re receiving 256 numbers - which you can see on the image above.

Some embedding models might not let you specify exact number and use the default amount underneath the hood. But no matter the approach, dimensions are always play important part in creating an embedding.

In the next section, we will also figure out how do we store our text embedding inside a vector database. And once this information is stored inside a vector database, how to perform a similarity search on it.

Let’s start from understanding what vector databases actually are.

A vector database is a type of database designed specifically to store and manage data in vector form, particularly vectors generated by embedding models.

And there’re a lot vector databases available on the market to choose from: Chroma, Pinecone, Weaviate, Qdrant and much more. There’s even an extension (pgyector) for the PostgreSQL which allows to work with vectors as well. Here’s the 

 is particularly interesting, because SQL does not natively support vectors. However, due to the dominance of SQL and increasing popularity of vector storage, the open-source community has found ways to merge these functionalities together. Essentially, this involves taking a traditional SQL database and adding a new column data type that supports vector storage.

Each of the popular vector databases have their own strengths and weaknesses. So depending on your needs, you might want to use one over another. That's why I highly recommend doing some additional research before choosing what to use. But for the majority of people who are new to the vector databases, the traditional SQL database with the vector support would be a great choice, and would cover most of your needs. That's why from now on we will use a vector PostgreSQL database as the example.

Another benefit of using vector PostgreSQL, that we can store there not only our embedding, but also create the additional table fields to store whatever data we want. Let's go over a small example with the pseudo-code.

For starters, we would create a basic SQL table with fields like 

Next, let's figure out a general process of how to create an embedding and populate vector database with it.

In example below, we have an initial data that we want to convert – 

. Then, we go over each piece of the content, create an embedding for it and save it as the 

 function to add embedding as well as original text content to our DB.

And no matter your vector database choice, the workflow would always stay the same: generate embedding, add it to the database, and then you can start working with it and perform searches.

And search functionality is exactly what we are going to cover next.

One of the core functions of a vector database is to efficiently handle similarity searches.

Similarity search is a process used to find items in a database that are most similar to a given input/query. When working with vector databases, each vector item, captures data’s distinct features or semantic meaning. In similarity search, the system compares the vectors, looking for those that are closest in terms of distance – usually using metrics like cosine similarity or Euclidean distance.

For instance, if you search for a word or phrase, the system converts it into a vector and then checks other vectors in the database to find the closest matches. This is especially useful in applications where you want to retrieve documents or images that are semantically similar to your query, ensuring that the results are based on meaning rather than exact text matches.

As an example, let’s write some pseudo-code, to perform cosine similarity. There we would compare input query to the embeddings stored in our vector PostgreSQL database.

That’s it! After we get our top three results, we can continue our usual RAG workflow that we mentioned at the beginning of the article. Which involves getting both, our initial query and retrieved embeddings from the database and passing it to the LLM to answer our request.

To wrap it up, it's clear that Retrieval-Augmented Generation is reshaping how we interact with AI. Unlike traditional AI systems that rely on fixed data, RAG opens up a door to dynamic, real-time information, with more accurate and up-to-date responses.

As we've seen, RAG uses technologies like embeddings and vector databases to keep its information organized and ready for any query you might throw at it. And while context windows in language models are growing, RAG remains essential for pulling in the latest information possible.

RAG is not just about making AI smarter – it's about making our interactions with technology more meaningful and useful. Whether you're an professional working with the AI or just someone curious about the new tech world, RAG is something that could change the way we think about AI. It’s about making sure that the answers we get are not just technically correct, but are also a part of your specific workflow data and always up-to-date.

In the end, RAG is another step toward making AI a more trusted and reliable tool in our daily lives.

That’s in for today! Keep an eye on the upcoming RAG content where we would be building applications to sharpen your knowledge and skills even farther.

Learn

The newline Guide to Building Your First GraphQL Server with Node and TypeScript

Teach

Amelia Wattenberger

Author of Fullstack D3

Community

The Future Of Software Engineering

Tutorials on Rag

RAG: Bridging the Gap Between AI and Real-Time Data

Email Newsletter

Popular Topics

Masterclasses

Tutorials

Fullstack React with TypeScript