Self-Hosted LLM Security: A Hardening Checklist for Teams

9 min read

Self-hosted LLM security starts from one fact: an inference server is a network service, and most of them ship with the security of a development tool. Keeping data inside your network removes the third-party risk, but it hands you every layer the provider used to handle. The checklist is short: never expose the model server directly, authenticate every endpoint, only load model files you can verify, treat model output as untrusted input, keep prompts out of logs by default, cap resource use, and patch the engine on a schedule.

Each item below says what to do, why it matters and who usually owns it. Most of it is ordinary infrastructure security applied to a new kind of service. The parts that are specific to LLMs are prompt injection and model files.

Why self-hosted LLM security needs its own checklist

The defaults are open. Ollama's documentation states that its local API "does not require authentication" (Ollama authentication). It binds to 127.0.0.1 by default, but the documented way to serve other machines is setting OLLAMA_HOST=0.0.0.0:11434, and at that point anyone who can reach the port can use it.

People do reach it. In a Shodan-based study, Cisco found 1,139 Ollama servers exposed to the internet, over 1,000 of them in the first 10 minutes of scanning, and 214 were actively serving models to anyone who asked.

vLLM is more explicit about its limits. Its security documentation says the --api-key flag only protects endpoints under /v1, /v2, /inference and /cohere, that "many other sensitive endpoints are exposed on the same HTTP server without any authentication enforcement", and that communication between nodes in a multi-node deployment is insecure by default and must sit on an isolated network.

None of this is a flaw in those projects. They are engines, and they expect you to put them behind something. The checklist is that something.

The checklist at a glance

Control What to do Usual owner
Network exposure Model servers on a private network, reachable only from the gateway Platform / network
Authentication One token per application, with an owner and an expiry, checked at the gateway Platform, app teams
Model supply chain Approved sources, recorded hashes, no pickle files, no remote code ML platform, security
Prompt injection Least privilege for anything the model can trigger; untrusted content stays untrusted App teams
Output handling Escape, validate and parameterize model output like user input App teams
Logs and PII Metadata by default; prompt text separate, short retention, tight access Security, data protection
Resource limits Caps on tokens, context, concurrency and per-token rate Platform
Patching Pinned engine versions, advisory watch, upgrade cadence Platform
Admin access SSO, RBAC, separate approvers and deployers, audit log Security, platform

1. Keep model servers off every network that does not need them

Run inference servers on a private subnet or a Kubernetes namespace with a default-deny network policy. The only thing allowed to reach the model port is your gateway or reverse proxy. Developers who need direct access get it through the same gateway, not a firewall exception.

Check the ports you did not open on purpose. Multi-GPU and multi-node setups open distributed-communication ports (for vLLM, torch.distributed and KV cache transfer) that should only be reachable between those nodes. A quick test from a machine outside the subnet:

# From a host that should NOT have access: every port should time out
nc -zv -w 3 10.0.4.12 11434   # Ollama
nc -zv -w 3 10.0.4.12 8000    # vLLM default

Terminate TLS at the gateway, and between the gateway and model servers if they cross a network you do not fully trust. If you need no outside connectivity at all, the air-gapped LLM deployment guide covers moving model files in and keeping the environment closed.

2. Authenticate every request at one front door

Engine-level keys are a backstop, not the control. Put authentication in a gateway in front of every model, so all endpoints, including the ones the engine leaves open, are behind it. A self-hosted LLM gateway is the usual place for this, along with routing and usage tracking.

Rules for the tokens themselves:

  • One token per application, never shared between apps or pasted into a team chat.
  • Each token has a named owner, a purpose and an expiry date.
  • Rotate on a schedule and immediately when someone with access leaves.
  • Revoke tokens with no traffic in the last 30 to 60 days.
  • Keep tokens in a secret store or environment variable, never in a repository. Run a secret scanner on your repos for the token prefix you use.

People sign in through SSO to anything with a UI (the chat interface, the admin console), so offboarding in your identity provider removes their access everywhere at once.

3. Verify every model file before it runs

A model file is code you did not write, and some formats execute code on load. Hugging Face's own pickle documentation warns of "dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file", and pickle is PyTorch's default format for weights.

What to require:

  • Weights in formats that store data, not code: safetensors or GGUF. Reject .bin, .pt and .pkl files unless you have a specific reason and a scanning step.
  • trust_remote_code off. With it on, loading a model runs Python files from the model's repository. Approve the rare exception explicitly.
  • An approved source list, and a SHA-256 hash recorded for every file you deploy. Compare the hash before loading, and again after any transfer between environments.
  • A license review at the same time, because a model you cannot legally use is also a supply-chain problem.

Safe formats reduce the risk, they do not remove it, because the parser that reads them can have bugs. In 2026, CVE-2026-7482 described a heap out-of-bounds read in Ollama's GGUF loader, triggered by uploading a crafted file to the /api/create endpoint, which could leak memory contents such as prompts and conversation data. Red Hat's analysis notes that /api/create and /api/push have no authentication in the upstream distribution. That is two checklist items failing together: an exposed endpoint and an unvetted file. Lock down who can create or load models, not only who can call them.

4. Contain prompt injection instead of trying to prevent it

Prompt injection is ranked first in the OWASP Top 10 for LLM Applications 2026, as it was in the 2025 edition. It means text that the model treats as instructions: typed by a user (direct), or hidden in a web page, email, PDF or retrieved document the model reads (indirect). The project leads summarized the 2026 edition's approach in Help Net Security's coverage: build the system so that "when the model is fooled, and it will be, nothing important breaks."

No system prompt reliably stops injection, so limit what a fooled model can do:

  • Give each application the narrowest model access and tool permissions it needs. A summarizer does not need to send email.
  • Require human confirmation before any action that writes, pays, deletes or sends data outside.
  • Treat every retrieved document as attacker-controlled. In a RAG system, anyone who can write to an indexed source can write instructions to your model. The private RAG guide covers access control on the document side.
  • Keep secrets out of the context window. If a credential is in the prompt, assume it can come out in a response.

5. Handle model output as untrusted input

Model output goes into browsers, shells, databases and other programs. Treat it exactly like text from an anonymous user:

  • Escape it before rendering as HTML or Markdown with links and images, which can carry data out through a URL.
  • Never pass it to eval, a shell, or a SQL string. Use parameterized queries and allowlisted commands.
  • Validate structured output against a schema before acting on it, and reject rather than repair what fails.
  • Review generated code like any other contribution before it is merged.

6. Decide what the logs keep

Logs are where self-hosted deployments leak data most often, because full prompts and responses are the most sensitive data in the system. Log metadata by default: timestamp, caller token, model and version, token counts, latency and status. If you need prompt text for debugging or quality review, store it separately with a short retention period and access limited to named people. Check that debug logging in the engine and the gateway is off in production; it often records full request bodies.

The LLM observability guide lists the metrics to collect, all of which work without storing prompt text.

7. Cap resource use per caller

One looping script can take a GPU from your whole company. Set limits at the gateway and the engine:

  • A maximum context length and maximum output tokens per request, below the model's ceiling where your use cases allow.
  • A per-token rate limit (requests per minute and tokens per minute).
  • A concurrency cap per inference server sized to its GPU memory, so overload produces fast errors rather than a crash.
  • Alerts on sudden jumps in tokens per caller, which catch both bugs and stolen tokens.

8. Patch the engine on a schedule

Inference engines release often, and their security fixes arrive in those releases. Pin the exact engine version in configuration, watch the project's GitHub security advisories, and upgrade on a fixed cadence (monthly works for most teams) plus out of band for anything that is remotely exploitable without authentication. Test each upgrade in staging with your real models first, because engine changes can also change memory use and output.

9. Control who can change the deployment

The admin plane is a higher-value target than the inference endpoint: someone who can load a model or change routing can affect every application. Put it behind SSO, use RBAC to separate the people who approve models from the people who deploy them, and keep an audit log of every download, deployment, permission change and token action. The LLM governance guide covers the approval process those roles support.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, installs inside your own infrastructure (your cloud, your VPC, on premises or air-gapped) and nothing phones home, which keeps prompts and model files on your network. Its catalog models come with security scoring and license documentation, and each quantization variant is benchmarked before it reaches the catalog, which gives the model review in step 3 a starting point. Deployed models are reached through an OpenAI-compatible endpoint with API tokens managed in the console, and RBAC, SSO, size-tier controls, approval workflows and a full audit log cover step 9.

It does not secure your applications. Prompt injection containment, output handling and what your apps do with responses (steps 4 and 5) stay with your developers, and network segmentation is still yours to design, since the platform runs on infrastructure and GPUs you provide. If you run a single model for a single internal tool, a hardened engine behind a reverse proxy may be all you need. For more models, teams and audit requirements, see how the private model platform handles it.

Security guidance changes quickly. Check the current OWASP list and each engine's security advisories before relying on specifics here.