Self-Hosted Vision Language Model: Models, Sizing, Setup

8 min read

A self-hosted vision language model (VLM) reads images and text together and answers in text, on GPUs you control. For invoices, scanned contracts, forms and screenshots, that means the documents never leave your network. The short version of how to do it: pick an Apache-2.0 model from the Qwen or Gemma families, budget GPU memory for the image tokens as well as the weights, serve it behind an OpenAI-compatible endpoint, and send images as image_url parts in a normal chat request.

The model is the easy part. Most of the work is deciding how many pixels each page gets, checking extracted values instead of trusting them, and knowing when plain OCR is the better tool. This post covers all three.

What a vision language model does

A VLM is a language model with a vision encoder attached. The encoder cuts an image into small patches and turns them into tokens the language model reads like words. From there it behaves like any chat model: you ask "extract the invoice number, date, line items and total as JSON" and it answers.

Typical private-document uses:

  • Pulling fields out of invoices, receipts, purchase orders and forms
  • Reading tables and charts from scanned reports
  • Describing screenshots for support tickets or UI testing
  • Classifying incoming documents before routing them to a person or system

If people only paste text, you do not need a VLM. If they paste screenshots into a chat tool, you do; building a self-hosted ChatGPT alternative covers that side.

Open models to consider for a self-hosted vision language model

The strongest open models in late 2026 are general models with vision built in, not separate vision products. All of the following are on Hugging Face with the licenses shown on their model pages (checked November 2026):

Model Sizes License Notes
Qwen3.8-27B 27B dense Apache 2.0 Native vision-language model, 262K-token context
Qwen3.6-35B-A3B 35B total, ~3B active (MoE) Apache 2.0 Fast for its size; needs memory for all 35B
Qwen3.5 Small sizes from 0.8B to 9B Apache 2.0 Image input even at the smallest sizes
Gemma 4 E2B, E4B, 12B, 26B A4B, 31B Apache 2.0 Configurable image token budget, up to 256K context
Qwen3-VL 4B, 8B, 30B-A3B, 32B and others Apache 2.0 Dedicated VL line from 2025, widely supported
Mistral Small 3.2 24B Apache 2.0 Text and image, many languages
Qwen2.5-VL 3B to 72B 7B: Apache 2.0; 72B: Qwen license Older but well documented and widely quantized
Llama 3.2 Vision 11B, 90B Llama 3.2 Community License Multimodal rights not granted to EU-based companies

Two license traps are worth knowing. Within one family, sizes can carry different licenses: Qwen2.5-VL 7B is Apache 2.0, but the 72B uses Qwen's own license. And the Llama 3.2 Acceptable Use Policy does not grant the rights to its multimodal models to individuals domiciled in the EU or to companies with their principal place of business there. Have legal read the license of the exact model you deploy; choosing an open source LLM for enterprise covers license review in more detail.

For pure text extraction there are also small OCR-specialized VLMs, such as Baidu's Unlimited-OCR (3.3B parameters, MIT license, needs trust_remote_code). They are cheap to run but follow instructions less well than general models.

As for accuracy, vendor benchmarks show the newer generations reading documents much better. Google reports an OmniDocBench 1.5 edit distance of 0.131 for Gemma 4 31B against 0.365 for Gemma 3 27B (lower is better) on its model card. Treat vendor numbers as a shortlist filter, not a result: your scans, fonts and languages are what count.

Sizing: weights, vision encoder and image tokens

A VLM needs memory for three things.

Language model weights. The same as any LLM of that size. Qwen2.5-VL-7B is 4.7 GB at Q4_K_M and 8.1 GB at Q8_0 in ggml-org's GGUF build. Gemma 4 31B is 18.0 GB at Q4_0 and 32.6 GB at Q8_0 in the same publisher's build.

Vision encoder. In GGUF builds it ships as a separate "mmproj" file: 1.35 GB at f16 for Qwen2.5-VL-7B and 1.2 GB at BF16 for Gemma 4 31B, whose encoder has about 550M parameters. It is small next to the language model, but count it.

Image tokens in the KV cache. This is the part people miss. Images become tokens, and tokens take KV cache memory just like text. How many depends on the model:

  • Qwen2.5-VL turns each 28 x 28 pixel area into one token, with a default range of 4 to 16,384 tokens per image, adjustable with min_pixels and max_pixels (model card).
  • Gemma 4 uses fixed budgets of 70, 140, 280, 560 or 1,120 tokens per image. Google recommends the higher budgets for OCR and small text (model card).

Here is what that means for one A4 page with Qwen2.5-VL-7B, whose KV cache takes about 56 KB per token at 16-bit (28 layers, 4 KV heads, head size 128):

Page rendering Image tokens KV cache for the image
300 dpi (2480 x 3508) ~11,100 ~0.63 GB
150 dpi (1240 x 1754) ~2,800 ~0.16 GB
Capped at max_pixels = 1280*28*28 1,280 ~0.07 GB

A 10-page batch at 300 dpi is over 110,000 image tokens before you write a word of prompt. The cap saves memory and prefill time but downsamples the page to roughly 100 dpi, where small print can become unreadable. For most invoices and forms, 150 dpi is a good starting point; test with your smallest real fonts. The general method for sizing the rest is in GPU memory for LLM inference.

Serving it behind an OpenAI-compatible endpoint

Both main open-source servers handle images.

With llama.cpp, pass the model and its projector:

llama-server -m Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
  --mmproj mmproj-Qwen2.5-VL-7B-Instruct-f16.gguf \
  --ctx-size 16384 --port 8080

The llama.cpp multimodal docs describe --mmproj and the related flags.

With vLLM, limit how many images one request may carry:

vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
  --max-model-len 32768 \
  --limit-mm-per-prompt '{"image": 4}'

Either way, the client sends a standard chat request with the image as a base64 data URI, as described in vLLM's multimodal inputs guide:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-token")

with open("invoice-page1.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    temperature=0,
    messages=[{
        "role": "user",
        "content": [
            {"type": "image_url",
             "image_url": {"url": f"data:image/png;base64,{image_b64}"}},
            {"type": "text",
             "text": "Return JSON with invoice_number, invoice_date, "
                     "supplier_name, total_amount and currency. "
                     "Use null for any field not visible on the page."}
        ],
    }],
)
print(response.choices[0].message.content)

The chat API takes images, not PDFs. Render pages first, for example with pdftoppm -r 150 -png contract.pdf page. And if a PDF already has a text layer (it was generated, not scanned), extract that text with pdftotext and skip the VLM for the text: it is exact and nearly free. Our OpenAI-compatible API guide covers pointing existing clients at a private endpoint.

OCR vs vision language model

Classic OCR engines such as Tesseract or PaddleOCR return the characters on the page, often with coordinates and a confidence score per word. A VLM returns an interpretation: it knows that the number under "Total due" is the total, that a table continues across columns, and that a checkbox is ticked.

That difference cuts both ways:

OCR engine Vision language model
Output Raw text, positions, confidence Structured answer (JSON, labels, summaries)
Layout and meaning Needs templates or rules Understands it from the prompt
Failure mode Misreads a character, visibly Can produce a plausible value that is not on the page
Hardware CPU is often enough GPU
Cost per page Very low Higher, grows with resolution

The failure mode is the important row. A VLM that cannot read a smudged total may still write one, and it will look as confident as a correct answer. For anything that feeds payments, filings or records, add checks:

  • Validate arithmetic: line items should sum to the subtotal, and subtotal plus tax to the total.
  • Validate formats: dates parse, IBANs and VAT numbers pass their checksums, currencies are real codes.
  • Ask for null when a field is not visible, as in the prompt above, and treat nulls as "send to a person".
  • Compare against OCR text for key fields where the stakes are high. If the VLM's invoice number does not appear in the OCR output, flag it.
  • Sample and review a fixed share of documents by hand, even after launch.

A hybrid works well for high volume: OCR or the PDF text layer for the raw text, a VLM for layout-heavy pages and for turning text into structured fields, and rules that catch disagreements. If you then want to search those documents, the extracted text goes into the pipeline described in RAG with a local LLM.

Running VLMs with AI Refinery

AI Refinery, a self-hosted LLM platform, includes vision-language models as one of the three categories in its curated catalog, next to text and embedding models. You download a model into your private storage, pick a quantization variant (each one is benchmarked for size on disk, GPU memory at the full context window and tokens per second) and launch an inference server with one click inside your own cloud, VPC, data center or air-gapped network. Each deployed model exposes an OpenAI-compatible /v1/chat/completions endpoint, and on Professional and Enterprise plans, smart model routing sends each request to the best available model for the task, whether vision, text, embeddings or code.

Which vision models are in the catalog at a given time is shown in the console. The platform does not include OCR, document parsing or a validation layer: you still build the PDF rendering, the checks and the review queue described above. If you need a specific model that is not in the catalog, or an engine flag the platform does not expose, run llama.cpp or vLLM directly. The self-hosted LLM platform page has the full feature list.