Self-Hosted ChatGPT Alternative: Build a Private AI Chat

7 min read

A self-hosted ChatGPT alternative is a chat assistant your staff use in the browser, just like ChatGPT, except every part of it runs inside your own network. Prompts, uploaded files and chat history stay on servers you control, and nothing is sent to an outside AI provider.

You build one from four parts: an open-weight model, an inference server that runs it, a chat interface people log in to, and the authentication that ties it to your company accounts. None of these needs to be built from scratch. Open-source chat interfaces already talk to any server that speaks the OpenAI API, so most of the work is choosing the model and running it well.

This guide walks through each part, the honest trade-offs against ChatGPT Enterprise, and a rollout plan that does not end with 500 people on an untested system.

What a self-hosted ChatGPT alternative is made of

Part What it does Typical choice
Model Generates the answers An open-weight family such as Llama, Qwen, Mistral, Gemma or gpt-oss
Inference server Keeps the model in GPU memory and answers requests over HTTP Anything that exposes an OpenAI-compatible /v1/chat/completions endpoint
Chat UI The browser app: conversations, history, file uploads Open WebUI or LibreChat
Authentication Who can sign in and what they can use Your identity provider through SSO, OIDC or LDAP

The glue between the chat UI and the inference server is the OpenAI-compatible API: the same request and response format as OpenAI's own API. Because both popular chat UIs support it, you can swap the model behind the UI without users noticing anything except the answers. Our OpenAI-compatible API guide covers the format and the differences worth testing.

Choosing the model

Start from what your staff will actually ask. For an internal assistant that drafts emails, summarizes documents, answers policy questions and helps with spreadsheets and code, the useful questions are:

  • How hard are the tasks? Short drafting and summarization work well on 7-14B parameter models. Long documents, multi-step reasoning and code review benefit from the 30-70B tier.
  • Do people paste screenshots? Then you also need a vision-language model, one that reads images as well as text. Running a self-hosted vision language model covers model choice and sizing.
  • What does the license allow? Open-weight licenses differ. Some are plain Apache 2.0, others add user-count thresholds or acceptable use policies. Our guide to choosing an open source LLM for enterprise compares the main families.

Then check the hardware. A rough rule for the weights alone is parameters x bytes per weight. At 4-bit quantization (about 0.6 bytes per parameter in common formats):

  • 14B model: 14 x 0.6 = about 8.4 GB
  • 70B model: 70 x 0.6 = about 42 GB

On top of that comes the KV cache, the memory each active conversation uses. A chat assistant with many people typing at once needs a lot of it, so size for your peak number of simultaneous chats, not your total headcount. The GPU memory guide shows the full calculation.

A practical starting point: one general instruction-tuned model in the 30-70B range for most chats, plus a smaller, faster model for quick tasks if your GPUs allow it.

Pairing a chat UI with your endpoint

Both of the main open-source options connect to an OpenAI-compatible server through configuration only.

Open WebUI

In Open WebUI's connection guide, you go to the admin settings, open Connections, add an OpenAI connection, and enter the server URL and API key. Three details from those docs save time:

  • The only required endpoint is /v1/chat/completions. A /v1/models endpoint is recommended for model discovery; without it you type the model IDs in manually.
  • Saving a connection does not test it. Send a real chat before you announce anything.
  • If Open WebUI runs in Docker and the model server is on the same host, use host.docker.internal instead of localhost.

Its feature list includes SSO, OIDC and LDAP sign-in, SCIM provisioning, roles and groups with per-model access, and document upload with knowledge bases. Document features need an embeddings endpoint as well as chat.

Check the license before a company-wide rollout. Since v0.6.6, Open WebUI's license does not allow removing or changing the "Open WebUI" branding unless you have 50 or fewer users in a 30-day period or an enterprise license that permits it. Keeping the branding is allowed.

LibreChat

LibreChat is MIT-licensed and supports OAuth2, LDAP and email login. You add your server as a custom endpoint in librechat.yaml with four required fields: name, apiKey, baseURL and models.

endpoints:
  custom:
    - name: "Internal LLM"
      apiKey: "${INTERNAL_LLM_KEY}"
      baseURL: "https://llm.internal.example.com/v1"
      models:
        default: ["general-assistant"]
        fetch: false

LibreChat appends /chat/completions to the baseURL, so point it at the /v1 root. Keep the key in an environment variable rather than in the file.

Either UI is a reasonable choice. The license is often the deciding factor: Open WebUI's branding clause matters if you want your own logo, and LibreChat's MIT license has no such condition. Try both with ten users for a week and let them decide.

What you give up compared with ChatGPT Enterprise

Self-hosting solves the data question. It does not give you ChatGPT. Be clear about the gaps before you promise anything to leadership.

Model quality on the hardest tasks. The strongest closed models are trained and served at a scale no single company can match. Plan for open-weight models to be close on everyday drafting and summarization, and to fall behind on long, multi-step reasoning, hard coding problems and niche knowledge. Test with your own prompts rather than trusting a leaderboard.

The product around the model. A hosted assistant comes with features its vendor builds and maintains. With a self-hosted stack you get what your chat UI ships, and new features arrive when you upgrade it.

Someone else running it. Uptime, GPU capacity, upgrades and security patches become your team's job. When 300 people start chatting at 9am, the GPUs have to be there.

Scale on demand. If usage doubles, you need more GPUs. A hosted service absorbs that for you.

What you gain is just as concrete: data that never leaves your environment, a model version that changes only when you change it, logs in your own systems, and a cost set by GPU capacity rather than by seat count. For some teams the right answer is both: ChatGPT for public, non-sensitive work and the internal assistant for anything touching customer data, contracts or source code.

Rolling it out

  1. Write the use policy first. Which data staff may paste in, which tasks it is meant for, and that answers must be checked. People will ask.
  2. Pick two or three use cases with clear owners, such as support reply drafts, policy Q&A or meeting-note summaries.
  3. Deploy one model and one UI behind SSO, with an API token that belongs to the chat UI rather than to a person.
  4. Pilot with 10-30 people from the chosen teams for two to four weeks. Collect bad answers in a shared channel; they are your test set.
  5. Measure before you scale. Watch peak concurrent chats, latency and token usage. These numbers tell you whether you need a second GPU or a smaller model.
  6. Add governance before wider access. Role-based access, limits on which models each team can use, and an audit log. The LLM governance guide covers what auditors tend to ask for.
  7. Expand team by team, adding document search only after basic chat is stable. Retrieval over internal documents is its own project; RAG with a local LLM explains the pieces.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, covers the model and inference server parts of this stack. It installs inside your own infrastructure (your cloud, your VPC, on-premise or a fully air-gapped network), and nothing phones home. You pick a model from a curated catalog of text, embedding and vision-language models, choose a quantization variant, and launch an inference server with one click. Each variant shows its size on disk, GPU memory needed and measured tokens per second before you deploy it.

Every deployed model gets an OpenAI-compatible /v1/chat/completions endpoint, so you paste its URL and an API token from the console into Open WebUI or LibreChat. Usage, latency and errors for each inference server appear in one dashboard, and RBAC, SSO, size-tier controls, approval workflows and an audit log cover who can use which model.

It does not include a chat UI or document search. You pair it with an open-source chat interface such as Open WebUI or LibreChat, and with a vector database of your choice if you add retrieval later.

When it is not the right fit

If your staff need the best available model on hard reasoning and your data is not sensitive, ChatGPT Enterprise or a similar hosted product is the simpler and stronger choice. The same goes if you have no GPUs and no one to own them: a self-hosted assistant with nobody responsible for it degrades quickly. And for a team of five, a single open-source UI on one workstation may be all you need. Questions about your setup go to [email protected].

For developers, the equivalent is a self-hosted Copilot alternative.

For retrieval over your own documents, choose an embedding model with embedding models for RAG.