Embedding Models for RAG: How to Choose and Serve One

5 min read

Choosing embedding models for RAG comes down to five properties: retrieval quality on your own documents, the language coverage you need, the maximum input length, the vector size, and the license. Leaderboard rank is a starting list, not the answer. The only reliable way to pick is to embed a sample of your documents with two or three candidates and measure how often the right passage comes back, which takes an afternoon.

This post gives a short list of open models to start from, the arithmetic for storage, and a test for deciding. For the full pipeline around the model (chunking, vector database, generation), see RAG with a local LLM.

What an embedding model does in RAG

An embedding model turns a chunk of text into a vector of numbers so that texts with similar meaning land close together. In RAG you embed every document chunk once and store the vectors, then at query time embed the question with the same model and fetch the nearest chunks. Whatever the generating LLM writes is limited by what this step retrieves, so a weak embedding model produces confident answers from the wrong passages.

Two rules follow from this. The same model must embed documents and queries. And switching models later means re-embedding the entire collection, so choose with some care.

Five things to compare

Retrieval quality. The MTEB leaderboard ranks models across many retrieval tasks and is a good way to build a shortlist. Treat the ordering as approximate: scores come from public datasets, and your contracts, tickets or clinical notes may behave differently.

Language. English-only models are usually smaller and slightly better on English. If your documents or queries are in several languages, pick a multilingual model, and test the language pairs you actually use.

Input length. The model truncates anything beyond its limit. Some accept 512 tokens, others 8,192. Your chunk size must stay under the limit, and longer limits let you embed whole sections when that suits your documents.

Dimensions. Every vector has a fixed length, typically 384 to 1,024 numbers for open models and sometimes more. It sets storage cost and search speed.

License. Check each model's card. Many popular open models are MIT or Apache 2.0, but not all.

Open models worth testing

Model Dimensions Max input Languages License
all-MiniLM-L6-v2 384 256 tokens English Apache 2.0
nomic-embed-text-v1.5 768 (adjustable) 8,192 tokens English Apache 2.0
BAAI bge-m3 1,024 8,192 tokens 100+ languages MIT
multilingual-e5-large 1,024 512 tokens ~100 languages MIT
Qwen3-Embedding (0.6B, 4B, 8B) 1,024 / 2,560 / 4,096 32K tokens multilingual Apache 2.0

Specs come from each model's card and can differ between versions, so confirm on the card you download. A reasonable shortlist for most teams is bge-m3 for multilingual or long documents, nomic-embed-text for English with long inputs, and one larger Qwen3-Embedding model if you can afford the extra GPU memory. Start small; the smallest model that meets your recall target is the cheapest to run.

Some models need a prefix to distinguish the two roles. The e5 family expects query: before a question and passage: before a document, and nomic-embed uses search_query: and search_document: . Forgetting the prefix is the most common cause of a model that scores far below its published numbers.

Storage and speed arithmetic

Each vector is stored as 32-bit floats by default, 4 bytes per dimension:

storage = chunks x dimensions x 4 bytes
Dimensions 1M chunks 10M chunks
384 ~1.5 GB ~15 GB
768 ~3 GB ~31 GB
1,024 ~4 GB ~41 GB

That is raw vector data. Your vector database adds an index on top, often another 50 to 100% depending on the index type. Some models support Matryoshka truncation, which lets you cut vectors to a shorter length with a modest quality drop, and vector databases can store 8-bit or binary vectors to shrink memory further. Both are worth testing only after you have a baseline.

Test on your own data

  1. Collect 50 to 100 real questions from users, and for each one note which document passage answers it.
  2. Chunk your documents the way you plan to in production.
  3. Embed all chunks with each candidate model and retrieve the top 5 for every question.
  4. Compute recall@5: the share of questions where the correct passage appears in the results.
  5. Compare models on recall, then on speed (chunks per second on your hardware) and storage.

If two models are within a couple of points of recall, choose the smaller one. If none reaches an acceptable recall, the fix is more often chunking or adding a reranker than a bigger embedding model. See the RAG pipeline post for chunk sizes.

Serving embeddings privately

Embedding a large corpus is a batch job that benefits from a GPU, and live queries need low latency. Both are easier when the model runs inside your own network, because the documents you embed are often the most sensitive data you have: sending them to an external embedding API sends the whole knowledge base outside. Self-hosting the embedding model keeps it internal and avoids per-token fees for a one-time bulk embed.

AI Refinery, a self-hosted LLM platform, includes embedding models as a category in its curated catalog, next to text and vision-language models, and each is downloaded and launched the same way as an LLM inside your own infrastructure. The platform has no vector database and no built-in RAG, so you bring the vector store you already use. Whether a particular embedding model is in the catalog at a given time is shown in the console. The self-hosted LLM guide explains the wider setup, and GPU memory for LLM helps size the hardware; embedding models are small by comparison, often under 2 GB at full precision for the 0.6B class.

If you want a place to run embeddings and generation together, see the private model catalog.

Model specs change between versions. Confirm dimensions, input limits and licenses on the model card you deploy.