Comparing Embedding Models for RAG Implementation

One of the most popular and effective approaches to integrating artificial intelligence into business operations today is the Retrieval-Augmented Generation (RAG). This architecture combines retrieving relevant data from external sources with generating meaningful responses based on that information. It benefits scenarios that require current, domain-specific, or extended-context information.

Depending on the system's design and goals, RAG systems can extract information from a wide range of sources, including document repositories, databases, websites, APIs, search engines, and more. This article focuses on using vector indexes, which enable information retrieval based on semantic similarity rather than simple keyword matching. This approach is especially beneficial for organizations dealing with complex, unstructured data, helping to improve the quality and relevance of AI-generated results.

RAG provides a flexible and scalable solution without fine-tuning large models for businesses aiming to implement AI in real-world processes. It is a compelling option for enhancing internal tools, automating customer interactions, or building intelligent assistants referencing up-to-date knowledge sources.

This article will be especially relevant for CTOs and business owners seeking effective ways to apply AI in their organizations, particularly where handling large volumes of information and delivering accurate, contextual responses are essential.

1. Preparation / Indexing:

• Document Embedding: all documents are split into text chunks and converted into high-dimensional vectors using an embedding model.

• Vector Storage: the generated vectors are stored in a vector database or in-memory.

2. Querying: 

• Query Embedding: the embedding model transforms the user’s query into a vector.

• Retrieval: the system searches for top-K semantically similar chunks, comparing the user’s input vector with vectors in the index.

• Augmented Generation: the retrieved chunks are passed along with the query to a large language model (LLM), which uses this context to generate a well-informed answer.

Today, there are many different embedding models that use different language models and embedding algorithms. This discussion focuses on applying RAG to the task of intelligent search within a repository of employee CVs and analyzes how the choice of specific embedding models affects the final accuracy of the system.

The application was implemented using the LlamaIndex framework for Python. It allows you to quickly build a RAG approach with just a few lines of code.

We selected the smallest self-hosted embedding models and the most affordable ones from service providers for testing and experiments. To run the self-hosted models, we used Ollama.

Let’s review the chosen embedding models (Table 1).

As the LLM for testing, we used a self-hosted Llama3.2 model with 3 billion parameters. It’s Meta’s instruction-tuned text-only model is optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks [definition from https://ollama.com/library/llama3.2:3b. Since we evaluate the impact of the embedding models in this paper, it doesn’t matter which LLM is used. For example, Llama 3.2 can be replaced with one of the OpenAI models.

Let’s build a simple RAG solution.

Setting chunk_size to 2048 means that one chunk will contain 2048 characters. In our case, this means that each chunk represents all the data from one CV file.

 Build a Query Engine. It’s a central abstraction that defines how a user’s query is processed over a knowledge base. It coordinates retrieval, response synthesis, and sometimes prompt construction to return an answer.

The similarity_top_k parameter controls how many chunks will be found by the embedding model and passed along with the user's request to LLM.

As part of this evaluation, we investigated how different embedding models impact LLM-generated responses by asking the same set of questions across an identical group of CVs. It is important to emphasize that the observations and interpretations presented here reflect the author’s personal perspective. Let’s take a closer look at the specifics of the experiment!

We created five CV documents representing IT professionals with diverse levels of experience and areas of specialization. These served as the external dataset for the RAG pipeline. Each CV includes a summary of key skills, previous work experience, educational background, and certifications where applicable. All character similarities are unintentional, and the names used are entirely fictional.

A separate query engine was created for the selected embedding model. Each engine was asked the same three questions derived from the CV content:

1. Do we have Umbraco developers?

2. Do we have Lead Front-end developers?

3. Do we have any certified candidates with extensive Docker experience?

Each question is asked three times per engine to ensure more reliable and consistent results. We don't persist the vector index to disk. It exists entirely in memory, meaning that with every run of the program, a new set of vectors is generated.

Let’s review some results (Table 2-3).

Based on this experiment, it can be concluded that Embedding Models text-embedding-3-small and BAAI/bge-small-en-v1.5 provide the most accurate and explainable responses. However, because the results are subjective, these models should be compared using more precise metrics.

In LlamaIndex, there are several tools called evaluators that are used to automatically assess the quality of outputs generated by RAG systems. The framework also provides a variety of metrics to evaluate the efficiency of the embedding model, which was the focus of our analysis. Specifically, we calculated the following values:

1. MRR (Mean Reciprocal Rank): measures how high in the list of retrieved chunks the first relevant chunk is ranked. For each query, the reciprocal rank is defined as 1 divided by the position of the first correct (relevant) chunk in the ordered list. The average of these reciprocal ranks across all queries is the final MRR score. A higher MRR means a higher probability that the system will return relevant results earlier in the top ranking list, which in turn demonstrates better search efficiency.

2. Hit Rate (Top-k Accuracy): It denotes the fraction of queries for which at least one relevant chunk is retrieved as one of the top k returned chunks. This is generally expressed as Hit@k (e.g., Hit@5 or Hit@10). The larger Hit Rate of a model indicates the model is increasingly probable to have at least one correct answer returning at a high position in the result list, which is more significant in practical use, as users tend to examine results near the top of the list.

First, we created an array of queries that would be used throughout the test. For this, we used another LLM — gpt-4o-mini, deployed in Azure AI Foundry | Azure OpenAI Service.

It generated five questions per chunk (or CV file in our case). Here are some of the questions:

1. What are the core skills that Alex Taylor possesses as a Middle Umbraco Developer, and how do they relate to his experience in building content management systems?

2. Describe the purpose of Vuex in Alice's projects and provide an example of a project where it was utilized.

3. What are the core technologies that John Lee specializes in as a Senior Vue.js Developer?

4. List two certifications that John Smith has obtained related to his professional expertise.

5. What educational qualification does John Smith hold, and from which institution did he graduate?

As in the previous section, each query engine was evaluated three times to obtain more accurate data. Let’s review the results (Table 4).

text-embedding-3-small is the best-performing Embedding Model, followed closely by BAAI/bge-small-en-v1.5. nomic-embed-text and bge-m3:567m are significantly weaker in both metrics and less stable across iterations.

In our specific evaluation, using a basic RAG setup with test CVs, the embedding model text-embedding-3-small showed the highest effectiveness when combined with the LLM Llama3.2. The model BAAI/bge-small-en-v1.5 ranked second, delivering slightly lower performance. The chosen self-hosted models produced less favorable outcomes. Still, it's essential to understand that these results can vary depending on multiple factors, such as the size of the CV dataset, chunk configuration, and the number of retrieved nodes during query execution.

In addition, we are also satisfied with the effectiveness of the embedding model text-embedding-3-small when used with the OpenAI models, such as gpt-4o-mini, but the results are not included in the article.

If you have questions or want to learn how Retrieval-Augmented Generation (RAG) can enhance your AI adoption, we recommend reading about our service, AI assistants. It shows how RAG-based systems enable assistants to deliver context-aware responses. Understanding intelligent retrieval and context integration is essential when starting with AI or running large language models in production. These elements help you get the most value from your data.

Prepared by a Certified Microsoft Developer, this article reflects our team's high quality and deep experience. UKAD has talented developers on board and can help you assemble a dedicated team by selecting the most suitable experts from our talent pool. We assist with conducting technical interviews, validating culture fit, and finalizing all necessary agreements. Partner with us to effectively integrate AI into your business processes and achieve practical, measurable results!