Ollama in Production: What to Add and When to Move On

7 min read

Running Ollama in production works for a specific kind of job: a small internal tool, a handful of users, one or two models on one machine. It works there because Ollama is simple. It also leaves out things a production service needs, and you have to add them yourself: authentication, TLS, concurrency limits sized to your hardware, monitoring and model version pinning.

The most important fact first. Ollama's documentation states that "the local API at http://localhost:11434 does not require authentication" (Ollama authentication). By default it binds to 127.0.0.1, so only the same machine can reach it. The moment you change that to serve other machines, anyone who can reach the port can use it. The self-hosted LLM security checklist covers this and the other controls a model server needs.

This guide covers where Ollama is a good choice, the settings and wrappers that make it safe to share, and the signs that it is time to move to something else.

Where Ollama shines

Developer machines. One installer on macOS, Windows or Linux, ollama run and a model is answering in minutes. GPU acceleration works on NVIDIA, AMD and Apple Silicon through Metal (Ollama hardware support).

Prototypes. Ollama serves an OpenAI-compatible API at http://localhost:11434/v1/ (OpenAI compatibility), so a prototype built on the OpenAI SDK can later point at a different server by changing the base URL.

Small internal tools. A summarizer used by ten people, a code helper for one team, a classification job that runs overnight. Low concurrency, one model, one host.

What Ollama in production needs

Authentication in front of it

Keep Ollama bound to localhost and put a reverse proxy in front that checks a token. The Ollama FAQ confirms it "can be exposed using a proxy server such as Nginx". A minimal Nginx setup that requires a bearer token and terminates TLS:

map $http_authorization $token_ok {
    default                          0;
    "Bearer replace-with-long-token" 1;
}

server {
    listen 443 ssl;
    server_name llm.internal.example.com;
    ssl_certificate     /etc/ssl/certs/llm.crt;
    ssl_certificate_key /etc/ssl/private/llm.key;

    location /v1/ {
        if ($token_ok = 0) { return 401; }
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host localhost:11434;
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

Exposing only /v1/ matters. Ollama's native API also includes endpoints to pull, create, copy and delete models, and you do not want those reachable from every client. proxy_buffering off lets streamed tokens reach the client as they are generated.

A single shared token is fine for one app. Once several apps or teams use the server, you want one key per client, so you can revoke one without breaking the rest and see who sent what. That is the job of an LLM gateway, covered in our guide to a self-hosted LLM gateway.

Leave OLLAMA_ORIGINS at its default unless a browser app on another origin needs to call Ollama directly. By default it allows cross-origin requests only from 127.0.0.1 and 0.0.0.0.

TLS

Ollama serves plain HTTP. Its FAQ only mentions HTTPS for outbound proxying, so terminate TLS at the proxy, as in the config above, with a certificate from your internal CA.

Concurrency settings

Three environment variables decide how Ollama behaves under load. From the FAQ:

Variable Default What it does
OLLAMA_NUM_PARALLEL 1 Requests each model processes at the same time
OLLAMA_MAX_LOADED_MODELS 3 x number of GPUs, or 3 on CPU Models held in memory at once
OLLAMA_MAX_QUEUE 512 Requests that can wait before Ollama returns HTTP 503

With the default of 1, a second user waits for the first to finish. Raising it costs memory, because "required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH".

Set the context length explicitly too. Ollama's context length page says the default depends on VRAM: 4K tokens under 24 GiB, 32K from 24 to 48 GiB, and 256K at 48 GiB or more. On a large GPU, a high default context multiplied by several parallel slots can use far more memory than you planned.

Here is the math for Llama 3.1 8B at 4-bit (about 4.9 GB in the Ollama library), with 4 parallel slots at 8K context. Its KV cache is 128 KB per token at 16-bit (2 x 32 layers x 8 KV heads x 128 head size x 2 bytes):

KV cache: 4 slots x 8,192 tokens x 128 KB  = 4.0 GB
weights:                                     4.9 GB
total before overhead                      ~ 8.9 GB

Setting OLLAMA_KV_CACHE_TYPE=q8_0 stores the cache at 8-bit instead of the f16 default, roughly halving that 4 GB. The setting applies to all models. Our post on GPU memory for LLM inference covers the full calculation.

On Linux, set these in the systemd service with sudo systemctl edit ollama (Ollama Linux docs):

[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_MAX_QUEUE=64"
Environment="OLLAMA_KEEP_ALIVE=-1"

A short queue is deliberate. A client that gets a 503 after a few seconds can retry or fail cleanly, while a request stuck behind 500 others just times out.

Keep-alive behavior

By default, Ollama keeps a model in memory for 5 minutes after its last request, then unloads it. The next request pays the full load time again, which shows up in the load_duration field of the response. For a service with steady traffic, that is a latency spike every time things go quiet.

OLLAMA_KEEP_ALIVE sets the default and the keep_alive field on an API request overrides it. Accepted values include a duration such as "24h", a number of seconds, a negative number to keep the model loaded indefinitely, and 0 to unload right after the response. For a production model on a dedicated GPU, -1 is usually right. The trade-off is that the VRAM stays occupied.

Model swapping has a related cost. If a request arrives for a model that does not fit alongside the ones loaded, the FAQ says new requests queue until it can load, and idle models are unloaded to make room. Two models taking turns on one GPU means repeated loads. Pin one model per GPU, or make sure they fit together.

Model version pinning

Ollama tags are names that point to a model digest. On the llama3.1 tags page, llama3.1:latest, llama3.1:8b and llama3.1:8b-instruct-q4_K_M all show the same digest today. Production config should name the explicit tag, not latest, and you should record the digest you tested. GET /api/ps returns the SHA256 digest of each running model, so a health check can confirm it.

For full control, import the model yourself. Store a GGUF file you have checksummed, reference it in a Modelfile with FROM /path/to/model.gguf, and create it under your own name (Ollama import guide). Then nothing changes unless you change the file.

Pin Ollama itself as well. The Linux install script accepts a version, for example OLLAMA_VERSION=0.5.7. The macOS and Windows apps download updates automatically, which is one more reason production servers belong on Linux.

Monitoring

Ollama's API reference does not list a metrics endpoint, so you assemble monitoring from two sources.

  • Every response ends with prompt_eval_count, eval_count, eval_duration, load_duration and total_duration, with timings in nanoseconds (Ollama usage). Tokens per second is eval_count / eval_duration x 1e9.
  • /api/ps shows each loaded model, its VRAM use, context length and when it will unload (Ollama /api/ps).

Add proxy access logs for status codes and latency, and alert on 503s, on rising load_duration, and on falling tokens per second. The LLM observability guide covers what to track and why.

Signs you have outgrown Ollama

  • You see 503s or long queue times at peak, even after raising OLLAMA_NUM_PARALLEL as far as memory allows. Engines built around continuous batching serve more simultaneous users per GPU; our vLLM vs Ollama comparison explains the difference.
  • Several teams share the server and you need per-team keys, quotas and an audit trail of who sent what.
  • Security review asks for SSO, role-based access control and an audit log.
  • You run several models on several hosts and keep their versions and settings in sync by hand.
  • You want to serve quantized Hugging Face checkpoints such as FP8 or AWQ, which vLLM supports directly.

None of these mean Ollama was the wrong choice. It means the job changed.

Where AI Refinery fits, and where it does not

AI Refinery, a self-hosted LLM platform, covers the layer this post has been adding by hand. It installs inside your own infrastructure, and you launch a model from a curated catalog with one click. The catalog uses GGUF files, the same format Ollama runs, and every quantization variant is benchmarked for size on disk, GPU memory and measured tokens per second. Each deployment gets an OpenAI-compatible /v1/chat/completions endpoint with API tokens managed in the console. Request volume, latency, token usage and errors appear in one dashboard. RBAC, SSO, size-tier controls, approval workflows and a full audit log handle the governance items from the list above.

It is not the right fit for one developer on a laptop or a prototype, where Ollama is simpler and enough. If a proxy, a few environment variables and one model on one GPU cover your needs, keep them. It does not include a chat UI either: pair it with an open-source option such as Open WebUI pointed at the endpoint. To try it, the Basic plan covers models up to 14B parameters with a 30-day free trial; see plans.

Running on a cluster instead? See how to deploy an LLM on Kubernetes with GPU scheduling, probes and storage.