vLLM vs Ollama: Throughput, Hardware and Setup Compared
The short answer to vLLM vs Ollama: vLLM is a serving engine built to push the most tokens per second out of data-center GPUs when many requests arrive at once. Ollama is a local runtime built to make one model easy to run on a laptop or workstation, with a single install and a model library.
Both expose an OpenAI-compatible API, so client code written against one works against the other with a different base URL. The real differences are model formats, behavior under concurrent load, the hardware they run on and how much setup you accept. This post compares them on each, using the projects' own documentation, and adds a third option at the end.
What each one is designed for
vLLM started as a research project: the paper that introduced it, Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv), is about serving many requests at once without wasting GPU memory. Its documentation lists continuous batching, prefix caching, quantization and tensor, pipeline and expert parallelism across GPUs as core features (vLLM docs). It is a server you deploy, usually on Linux with NVIDIA or AMD GPUs.
Ollama is a desktop and server app for macOS, Windows and Linux. You run ollama run <model>, it downloads the model from its library, loads it, and unloads it after a period of inactivity. Ollama has relied on the llama.cpp project for model support and still uses the GGML tensor library, and in 2025 it added its own engine for newer multimodal models (Ollama blog). Its priority is ease of use.
Model formats
| vLLM | Ollama | |
|---|---|---|
| Main format | Hugging Face checkpoints (safetensors), loaded by repo ID | GGUF, from the Ollama library or imported |
| Quantization | FP8, INT8, INT4, GPTQ, AWQ and others | GGUF quantizations such as q4_K_M and q8_0 |
| The other format | GGUF supported but "highly experimental and under-optimized" | Can import a safetensors directory through a Modelfile |
vLLM's quantization support is listed in its documentation, and its GGUF page carries the experimental warning. Ollama's import guide covers both GGUF files and safetensors directories, and notes that Ollama "does not quantize GGUF models during import", so you quantize beforehand with a tool such as llama.cpp's llama-quantize.
In practice, the format you already have often decides for you. An AWQ or FP8 checkpoint from Hugging Face belongs on vLLM. A Q4_K_M GGUF belongs on Ollama or llama.cpp. Our guide to LLM quantization explains what those levels cost in quality.
Throughput under concurrency
This is the biggest difference, and it comes from two techniques.
Continuous batching. A simple server processes one request, or one fixed batch, until it finishes. With continuous batching, the server adds new requests to the running batch at every generation step and removes finished ones, so the GPU never waits for the slowest request in a batch.
PagedAttention. Every in-flight request needs KV cache, the memory a model uses to track its context. Reserving one large contiguous block per request wastes memory on space that is never used. PagedAttention stores the cache in small fixed-size blocks, like pages in virtual memory, so more requests fit on the same GPU. The vLLM paper reports 2-4x higher throughput than FasterTransformer and Orca at the same latency.
Ollama handles concurrency with explicit settings. According to its FAQ, OLLAMA_NUM_PARALLEL (default 1) sets how many requests each model processes at the same time, extra requests wait in a queue of up to OLLAMA_MAX_QUEUE (default 512), and a full queue returns HTTP 503. Raising the parallel count multiplies the context memory: the FAQ's example is "a 2K context with 4 parallel requests will result in an 8K context".
You can do the memory math yourself. Llama 3.1 8B uses 128 KB of KV cache per token at 16-bit (2 x 32 layers x 8 KV heads x 128 head size x 2 bytes). Four parallel slots at 8K tokens each need:
4 slots x 8,192 tokens x 128 KB = 4 GB of KV cache
That is on top of roughly 5 GB of weights at 4-bit. Our post on GPU memory for LLM inference covers the full calculation.
We are not quoting a head-to-head benchmark, because results swing widely with GPU, model, quantization and prompt length. Run your own: replay a sample of real prompts at your expected number of simultaneous users and measure tokens per second and time to first token. Batching only pays off when there are several requests to batch, so test at the concurrency you expect, not with one user.
Hardware support
| vLLM | Ollama | |
|---|---|---|
| Operating systems | Linux; no native Windows (use WSL) | macOS, Windows, Linux |
| NVIDIA | Compute capability 7.5+ (T4, A100, L4, H100 and newer) | Compute capability 5.0+, driver 550+ |
| AMD | ROCm 6.3+: MI200, MI300, MI350, Radeon RX 7900 and 9000 series | ROCm v7 on Linux and Windows, plus Vulkan |
| Apple Silicon | CPU support marked experimental, build from source; GPU via the separate vLLM-Metal plugin | GPU acceleration through Metal |
| CPU only | x86 (AVX2/AVX512 wheels) and ARM | Supported |
Sources: vLLM's GPU and CPU installation pages, and Ollama's hardware support page. If your team works on MacBooks, Ollama is the natural fit there. If you run a rack of H100s, vLLM is.
OpenAI-compatible APIs in vLLM and Ollama
Both speak the format most tools expect, which our OpenAI-compatible API guide covers in detail.
vLLM serves /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses, /v1/models and audio transcription endpoints (vLLM OpenAI-compatible server). It accepts an --api-key, but the docs say it only covers the /v1, /v2 and /inference prefixes and warn: "Do not rely on --api-key alone to secure vLLM." It also exposes Prometheus metrics at /metrics (vLLM metrics).
Ollama serves /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings and /v1/responses at http://localhost:11434/v1/, alongside its own native API (Ollama OpenAI compatibility). The API key is "required but ignored", and the local API does not require authentication (Ollama authentication).
The client code is the same either way. This points at Ollama; for vLLM, change the base URL to http://localhost:8000/v1 and pass the key you set with --api-key:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="llama3.1:8b-instruct-q4_K_M",
messages=[{"role": "user", "content": "Summarize this ticket in one line."}],
)
print(reply.choices[0].message.content)
Model names differ between the two (an Ollama tag versus a Hugging Face repo ID), so keep the model name in configuration.
Ease of setup
Ollama on Linux is two commands (Ollama Linux docs):
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b
On macOS and Windows it is a regular app installer.
vLLM's quickest route is the official Docker image, which needs the NVIDIA container runtime on the host. From the vLLM docs:
docker run --runtime nvidia --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 8000:8000 --ipc=host \
vllm/vllm-openai:latest \
--model Qwen/Qwen3-0.6B
Gated models also need a Hugging Face token. Expect to spend time on flags such as maximum context length and GPU memory share once you move past the first test.
vLLM vs Ollama at a glance
| vLLM | Ollama | |
|---|---|---|
| Built for | High-throughput serving | Easy local use |
| Formats | Hugging Face safetensors, many quantizations | GGUF |
| Concurrency | Continuous batching, PagedAttention | OLLAMA_NUM_PARALLEL slots and a queue |
| Platforms | Linux, NVIDIA/AMD GPUs, CPUs | macOS, Windows, Linux, Apple Metal |
| OpenAI API | Yes, plus /metrics |
Yes |
| Built-in auth | --api-key, limited scope |
None on the local API |
| Setup | Docker or Python, GPU drivers | One installer |
Pick vLLM if, pick Ollama if
Pick vLLM if you serve many users at once, run on Linux with NVIDIA or AMD data-center GPUs, need a model split across several GPUs, or already have safetensors, AWQ or FP8 checkpoints.
Pick Ollama if you are a developer running models on your own machine, building a prototype, working on Apple Silicon or Windows, or serving a small internal tool with a handful of users. If that tool starts growing, read Ollama in production for what to add and when to move on.
A third option: llama.cpp server
llama-server is the HTTP server from the llama.cpp project. It serves GGUF models with OpenAI-compatible chat completions, responses and embeddings routes, supports --api-key, runs several request slots with --parallel and continuous batching on by default, and exposes Prometheus metrics with --metrics (llama.cpp server README). It suits teams that want GGUF models with more direct control than Ollama gives, and fewer moving parts than vLLM.
Where AI Refinery fits, and where it does not
Choosing an engine is one decision. Running it for a company also means authentication, monitoring, model choice and access control. AI Refinery, a self-hosted LLM platform, handles that layer inside your own infrastructure. You pick a model from a curated catalog of GGUF files, where every quantization variant is benchmarked for size on disk, GPU memory and measured tokens per second, and launch an inference server with one click. Each deployment exposes an OpenAI-compatible /v1/chat/completions endpoint, API tokens are managed in the console, and a dashboard shows request volume, latency, token usage and errors.
It is not the right fit if you want to tune engine flags yourself, if your models are safetensors or AWQ checkpoints you plan to serve as they are, or if you are one developer on a laptop, where Ollama is simpler. You also provide the GPUs. Plans are sized by model, from up to 14B parameters on Basic to unlimited on Enterprise; see pricing.
If you have settled on a production engine and are choosing between two, vLLM vs SGLang compares them and shows how to benchmark both.