RAG with Local LLM: Build a Private Retrieval Pipeline

8 min read

Doing RAG with local LLM models means answering questions from your own documents with every step running on infrastructure you control. Retrieval-augmented generation (RAG), described in the original 2020 paper by Lewis et al., works in two moves: find the passages most relevant to a question, then give them to a language model with instructions to answer from them. If both the embedding model and the chat model run locally, your documents and questions never leave your network.

The pipeline has five parts: chunking, embeddings, a vector store, retrieval and prompt assembly. Each one is simple. The quality of the answers depends on getting the details right, and on checking permissions before anything reaches the model.

The RAG pipeline, step by step

Indexing (once, then on every document change)
  documents -> split into chunks -> embedding model -> vectors + metadata -> vector store

Answering (every question)
  question -> embedding model -> nearest chunks the user may see -> prompt -> chat model -> answer with sources
  1. Chunking. Split each document into passages small enough to retrieve precisely and large enough to make sense on their own.
  2. Embeddings. An embedding model turns each chunk into a vector, a list of numbers where similar meanings end up close together.
  3. Vector store. A database stores the vectors with the chunk text and metadata (source, section, who may read it) and finds the nearest vectors to a query quickly.
  4. Retrieval. The question is embedded with the same model, and the store returns the closest chunks the user is allowed to see.
  5. Prompt assembly. The chunks go into the prompt as numbered sources, with instructions to answer only from them and cite them.

RAG with local LLM models: both run on your hardware

A private RAG system needs two models, and both have to be local for the data to stay local.

The embedding model runs at indexing time over every chunk, and at query time once per question. Embedding models are usually far smaller than chat models, so they need little GPU memory, but they have a maximum input length. Check the model card: text past the limit may be cut off, and a chunk the model never fully read cannot be retrieved for its last paragraphs.

The chat model reads the retrieved chunks and writes the answer. It needs a context window big enough for the system prompt, the sources, the question and the answer. Our guide to GPU memory for LLM inference shows how the KV cache grows with that context.

One rule matters more than any other: always embed queries with the same model and version you used for the documents. Vectors from different embedding models are not comparable. If you change the embedding model, re-embed the whole corpus.

For the chat side, if your server speaks the OpenAI API, any existing client code works. The OpenAI-compatible API guide covers what to test when you point code at a new endpoint.

Choosing a chunk size

There is no universally correct chunk size, but there is a method for picking one.

Start from the prompt budget. Suppose your chat model is deployed with an 8,192-token context window. Reserve about 600 tokens for the system prompt and question and 1,000 for the answer:

8,192 - 600 - 1,000 = 6,592 tokens for sources

With five sources per answer, each chunk can be up to about 6,592 / 5 = 1,300 tokens. That is a ceiling, not a target.

Then size for the content. A starting point worth testing is 300-800 tokens per chunk, with an overlap of 10-15% so a sentence split across a boundary still appears whole in one chunk. Use smaller chunks for FAQs, policies and support articles, where each answer is a paragraph. Use larger ones for contracts and technical manuals, where the meaning depends on surrounding text.

Split on structure, not character counts. Break at headings, sections and paragraphs first, and only fall back to a fixed length inside very long sections. Store the document title and section heading with each chunk and include them in the text you embed. "Section 4.2 Refunds: Customers may..." retrieves far better than the bare sentence.

Estimate storage early. 20,000 documents x 10 pages x 500 tokens a page is 100 million tokens. At 500 tokens a chunk that is 200,000 chunks. With 1,024-dimension vectors stored as 4-byte floats, the raw vectors take 200,000 x 1,024 x 4 = about 0.8 GB, plus index overhead. Small enough for a single database server.

Choosing a vector store

You bring the vector database. Two common self-hosted choices:

  • pgvector adds vector search to PostgreSQL. You keep vectors next to your other data, filter with ordinary WHERE clauses and use HNSW or IVFFlat indexes for approximate search. If you already run Postgres, this is the shortest path.
  • Qdrant is a dedicated vector search engine, licensed under Apache 2.0, that runs locally with Docker. It supports filtering on payload fields during search, which suits per-user permissions. Its README notes that the default Docker command starts an instance with no authentication, so read its security guide before you expose it.

A minimal pgvector schema, with the permissions stored on every chunk:

CREATE EXTENSION vector;

CREATE TABLE chunks (
  id             bigserial PRIMARY KEY,
  doc_id         text NOT NULL,
  doc_title      text,
  section        text,
  content        text NOT NULL,
  allowed_groups text[] NOT NULL,
  embedding      vector(1024)   -- match your embedding model's output size
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Retrieval and prompt assembly in Python

This example retrieves the five nearest chunks the user may read, numbers them, and sends them to a chat model through an OpenAI-compatible /v1/chat/completions endpoint. The base URL is a placeholder for your own server.

import psycopg
from openai import OpenAI

chat = OpenAI(
    base_url="https://llm.internal.example.com/v1",  # your server
    api_key="your-token",
)

def embed(text: str) -> list[float]:
    """Return a vector from your locally deployed embedding model.
    Use the same model and version that indexed the documents."""
    raise NotImplementedError

def retrieve(conn, question: str, user_groups: list[str], k: int = 5):
    qvec = "[" + ",".join(str(x) for x in embed(question)) + "]"
    return conn.execute(
        """
        SELECT doc_title, section, content
        FROM chunks
        WHERE allowed_groups && %s::text[]
        ORDER BY embedding <=> %s::vector
        LIMIT %s
        """,
        (user_groups, qvec, k),
    ).fetchall()

def answer(conn, question: str, user_groups: list[str]) -> str:
    chunks = retrieve(conn, question, user_groups)
    if not chunks:
        return "None of the documents you can access cover this."

    sources = "\n\n".join(
        f"[{i}] {title} - {section}\n{content}"
        for i, (title, section, content) in enumerate(chunks, start=1)
    )
    resp = chat.chat.completions.create(
        model="your-chat-model",
        temperature=0.1,
        messages=[
            {"role": "system", "content": (
                "Answer using only the numbered sources below. "
                "Cite sources like [2]. If the sources do not contain "
                "the answer, say you do not know.\n\n" + sources
            )},
            {"role": "user", "content": question},
        ],
    )
    return resp.choices[0].message.content

with psycopg.connect("dbname=rag") as conn:
    print(answer(conn, "What is our refund window?", ["support", "all-staff"]))

The <=> operator is pgvector's cosine distance, and && is Postgres array overlap, so a chunk is returned only if it shares at least one group with the user. A low temperature keeps the model close to the sources. Returning a fixed message when nothing is retrieved stops the model from answering from its general training instead.

Evaluating a local RAG system

Test retrieval and generation separately, because they fail for different reasons.

  1. Build a test set. Collect 50-100 real questions and, for each, the document and section that holds the answer. Include some questions your documents cannot answer.
  2. Measure retrieval. For each question, check whether the right chunk appears in the top k results (recall at k). If it does not, no prompt will save the answer: fix chunking, add titles to chunks or try another embedding model.
  3. Measure answers. For questions where retrieval worked, check whether every claim in the answer is supported by a cited source, and whether citations point to the right chunk.
  4. Measure refusals. For the unanswerable questions, the right answer is "I do not know". Count how often the model makes something up instead.
  5. Re-run on every change. New chunk size, new embedding model, new chat model or a new prompt: run the whole set again and compare.

Production logs add a sixth check. Track questions with no retrieved chunks and answers users flag as wrong, then add them to the test set. Our guide to LLM observability covers what to log.

Permissions on retrieved documents

A RAG system can leak information faster than any search page, because it summarizes what it finds. Treat permissions as part of retrieval, not as a prompt instruction.

  • Filter before the model sees anything. Apply the user's permissions in the vector store query, as in the example above. Never retrieve everything and ask the model to ignore what the user cannot see.
  • Copy permissions from the source system. Store the access groups of each document on its chunks at indexing time, and re-sync when permissions change. A document removed from someone's access in the file share should drop out of their answers too.
  • Delete properly. When a document is deleted at the source, delete its chunks and vectors.
  • Authenticate the caller, not the prompt. Take the user's identity and groups from your login system, never from text in the request.
  • Log what was retrieved. Recording which chunks were returned for which user makes access reviews and incident investigations possible. The controls in LLM governance apply here as well.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, runs both models a private RAG system needs inside your own infrastructure: your cloud, your VPC, on-premise or an air-gapped network. Its catalog includes embedding models for semantic search and RAG pipelines alongside text models for generation, each benchmarked per quantization for size on disk, GPU memory at full context and tokens per second. Deployed models expose an OpenAI-compatible /v1/chat/completions endpoint, so the Python above works by changing the base URL and model name, and the console tracks request volume, latency, token usage and errors.

It does not include a vector database or a built-in RAG pipeline. You bring the store (pgvector, Qdrant or another of your choice), the chunking code and the permission sync.

It is not the right fit if your documents are public and you have no GPUs, where a hosted API is simpler, or if you want a finished, no-code document Q&A product rather than a pipeline you build and own.

To pick the embedding model for this pipeline, see embedding models for RAG.