Self-Hosted LLM: A Practical Guide for Enterprise Teams
A self-hosted LLM is a large language model that runs on infrastructure you control: your own servers, your cloud account, your VPC, or a network with no internet access at all. Prompts and responses never leave that environment, the model version only changes when you change it, and every request can be logged by your own systems.
The trade is simple. A public AI API gives you a model with zero setup, but your data travels to someone else's servers and the model can change under you. Self-hosting gives you privacy and control, in exchange for running the hardware and software yourself.
This guide covers when self-hosting is worth it, what you need to run one, and the steps to get a private model serving your applications.
What is a self-hosted LLM?
Most teams first meet LLMs through a hosted API: you send text to a provider's endpoint and get text back. The model, the GPUs and the logs all live with the provider.
With a self-hosted LLM, you take an open-weight model (one whose files you are allowed to download, such as Llama, Qwen, Mistral or Gemma), load it onto your own GPUs, and serve it through an inference server: a program that keeps the model in memory and answers requests over HTTP. Your applications call that server instead of the provider.
Three things change when you do this:
- Where the data goes. Prompts, documents and outputs stay inside your network.
- Who controls the model. The version you tested is the version in production until you decide to upgrade.
- Who sees the logs. Usage, latency and audit records live in your systems, not a vendor dashboard.
When self-hosting makes sense
Self-hosting is not automatically better. It earns its keep in a few common situations.
You handle sensitive data. Customer records, source code, legal documents, health or financial data. If a policy, a regulator or a customer contract says the data cannot leave your environment, a public API is off the table no matter how good the model is.
You need predictable behavior. Hosted models get updated, tuned and retired on the provider's schedule. If an application has been carefully tested against one model's output, a silent change can break it. A self-hosted model behaves the same way until you change it.
You need an audit trail. Security and compliance teams will ask who used which model, when, and for what. That is much easier to answer when every request passes through infrastructure you own.
You have steady, high volume. Per-token API pricing adds up when usage is constant. Dedicated GPUs cost the same whether they are busy or idle, so a heavily used model can become cheaper to run yourself. Low or spiky usage usually favors an API.
When a public API is the better choice
Be honest about the other side. If you need the single most capable frontier model, only want to try an idea for a few weeks, or have no one to own the infrastructure, a hosted API is often the right call. Many teams use both: an API for experiments and non-sensitive work, and a self-hosted model for anything that touches private data. We compare the two in detail in private LLM vs OpenAI API.
What you need to run an LLM yourself
1. A model
Open-weight models come in sizes measured in parameters: roughly, the number of values the model learned. Common sizes are 7-8B (billion), 13-14B, 30-34B and 70B and up. Bigger models are generally more capable and need more hardware.
Check two things before you pick one:
- The license. "Open" does not always mean "use for anything". Some licenses restrict commercial use, user counts or certain industries. Read it before you build on it.
- The task. A chat model, a code model, a vision model and an embedding model (which turns text into vectors for search) are different tools. Pick the type that matches the job.
2. GPU memory
The main constraint is GPU memory (VRAM): the whole model has to fit, plus working space for the conversations in flight. A useful rule of thumb for the model weights alone:
| Precision | Bytes per parameter | 7B model | 70B model |
|---|---|---|---|
| 16-bit (full) | 2 | ~14 GB | ~140 GB |
| 8-bit | 1 | ~7 GB | ~70 GB |
| 4-bit | ~0.6 | ~4 GB | ~42 GB |
Add headroom on top of that for the KV cache, the memory the model uses to keep track of each conversation. Longer prompts and more simultaneous users need more of it. Our guide to GPU memory for LLM inference shows how to calculate it.
3. Quantization
The 8-bit and 4-bit rows in that table come from quantization: storing the model's numbers with less precision so it takes less memory and usually runs faster. The cost is some loss in output quality, which is small for well-made 8-bit versions and larger as you go lower.
Quantization is what makes self-hosting practical for most teams. A 4-bit 70B model fits on hardware that could never hold the full-precision version. The right level depends on your task, so measure speed, memory and quality on your own hardware before you commit. LLM quantization explained covers what each level costs.
4. An inference server and an API
The inference server loads the model and answers requests. The most useful thing it can do for you is speak the same API as OpenAI: an OpenAI-compatible endpoint such as /v1/chat/completions. Then any existing code built on the OpenAI SDK, and most AI tooling, can switch to your model by changing one setting:
from openai import OpenAI
client = OpenAI(
base_url="https://llm.internal.example.com/v1", # your server, not the provider's
api_key="your-internal-token",
)
reply = client.chat.completions.create(
model="your-deployed-model",
messages=[{"role": "user", "content": "Summarize this contract clause..."}],
)
print(reply.choices[0].message.content)
In most cases the prompts stay the same too. The OpenAI-compatible API guide lists the differences worth testing before you switch.
5. Governance and monitoring
A model your whole company can call needs the same controls as any other internal service: who is allowed to use it (role-based access control, or RBAC), single sign-on, an audit log of requests, and dashboards for request volume, latency, token usage and errors. Skipping this is how a private model turns into an unmanaged one.
How to deploy a self-hosted LLM, step by step
- Write down the use case. What task, how many users, how sensitive the data is, and how fast responses need to be. This decides everything else.
- Shortlist models. Filter by task type and license, then by size that fits your budget for GPUs. Our guide to choosing an open source LLM for enterprise compares the main model families and their licenses.
- Pick a quantization level. Start with 8-bit if it fits, and test 4-bit if memory is tight. Compare outputs on real examples from your use case.
- Provision hardware. On premises, in your cloud account or in a private VPC. Size VRAM for the weights plus KV cache for your expected number of simultaneous users, and price it with the self-hosted LLM cost model. If the network has no internet access at all, follow the air-gapped LLM deployment checklist.
- Deploy the inference server with an OpenAI-compatible API, behind your network controls.
- Add access control, logging and monitoring before you open it to other teams. See LLM governance for the controls and LLM observability for what to measure.
- Point one application at it, compare results with what you had before, then roll out more widely.
Each step is straightforward on its own. The work is in doing all of them, keeping them up to date, and repeating the process every time a better model comes out.
Running it with AI Refinery
AI Refinery, a self-hosted LLM platform, packages these steps into one product that you install inside your own infrastructure: your cloud, your VPC or a fully air-gapped network. Nothing phones home.
- Model choice: a curated catalog of open-source models with security scoring, license documentation and quantization variants. Every quantization is benchmarked before it reaches the catalog, so you can see its speed, memory needs and quality cost before you commit hardware.
- Deployment: download a model into your private storage, pick a quantization level and launch an inference server with one click.
- Integration: every deployed model exposes an OpenAI-compatible
/v1/chat/completionsendpoint, so existing apps switch by changing their base URL. - Governance: RBAC, SSO, size-tier controls, approval workflows and a full audit log.
- Observability: request volume, latency, token usage and errors for every inference server, in one dashboard.
- Routing: with several models deployed, smart routing can send each request to the best model for the task, such as vision, text, embeddings or code.
You still provide the GPUs. AI Refinery removes the custom scripts and glue code around them.
If you are comparing against managed hosting, read the Hugging Face Inference Endpoints alternative comparison, and for cluster setups, deploy an LLM on Kubernetes.