OpenAI-Compatible API: Switch to a Private LLM in Minutes

4 min read

An OpenAI-compatible API is a server that accepts the same requests as OpenAI's API and sends back responses in the same format. The paths match (/v1/chat/completions), the JSON has the same fields, and authentication uses the same Authorization: Bearer header.

That means code written for OpenAI can call a different model, including one running on your own hardware, by changing two settings: the base URL and the model name. No new SDK, no rewritten prompts.

This guide shows the change in the tools most teams use, then covers what is not identical, because that is where switching usually goes wrong.

What an OpenAI-compatible API actually matches

The OpenAI API became the format almost every tool supports, so most open-source inference servers copy it. A chat request looks the same wherever it goes:

curl https://llm.internal.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-14b-instruct",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain RAID 10 in two sentences."}
    ],
    "temperature": 0.2
  }'

And the reply has the familiar shape: choices[0].message.content holds the text, usage holds token counts, and stream: true returns server-sent events in the same chunked format as OpenAI.

Pointing your code at a different base URL

Python (official OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.internal.example.com/v1",
    api_key="your-token",
)

resp = client.chat.completions.create(
    model="qwen2.5-14b-instruct",
    messages=[{"role": "user", "content": "Explain RAID 10 in two sentences."}],
)
print(resp.choices[0].message.content)

The SDK also reads the OPENAI_BASE_URL and OPENAI_API_KEY environment variables. If your code creates the client with no arguments, you can switch it without touching the code at all:

export OPENAI_BASE_URL="https://llm.internal.example.com/v1"
export OPENAI_API_KEY="your-token"

Node.js / TypeScript

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://llm.internal.example.com/v1",
  apiKey: process.env.LLM_API_KEY,
});

const resp = await client.chat.completions.create({
  model: "qwen2.5-14b-instruct",
  messages: [{ role: "user", content: "Explain RAID 10 in two sentences." }],
});
console.log(resp.choices[0].message.content);

The Node SDK reads OPENAI_BASE_URL too.

LangChain

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://llm.internal.example.com/v1",
    api_key="your-token",
    model="qwen2.5-14b-instruct",
)

Most other frameworks and tools that "support OpenAI" have a base URL setting somewhere in their configuration. Look for base_url, api_base or endpoint.

Two habits make the switch painless. Keep the base URL, key and model name in configuration rather than in code. And put model names behind your own aliases (for example summarizer instead of gpt-4o-mini), so changing models is a config change too.

What is not the same

"Compatible" means the request and response formats match. It does not mean the model behind them behaves like OpenAI's. Check these before you move production traffic.

Model names. gpt-4o means nothing to your server. Use the name your server lists, usually at GET /v1/models.

Prompt behavior. A prompt tuned for one model will not always give the same results on another. Formatting instructions, JSON output and long system prompts are the usual places differences show up. Re-test your prompts; most need small changes at most.

Optional features. Tool calling (function calling), structured outputs with a JSON schema, logprobs, seed and multiple choices per request (n) depend on both the server and the model. Some servers ignore parameters they do not support, which is worse than an error, because nothing tells you. If your app relies on one of these, test it on purpose.

Other endpoints. Many servers implement chat completions but not every OpenAI endpoint. Embeddings, audio, images, the Assistants API and batch jobs may live elsewhere or not exist. Check which paths your server actually serves.

Context length. Every model has a maximum number of tokens it can handle in one request, and your server may be configured lower than the model's maximum to save memory. A request that worked on OpenAI can fail if it is longer than the limit on your server.

Token counts. Different models split text into tokens differently, so the same prompt produces a different usage number. Any budget or cost logic built on OpenAI token counts needs recalibrating.

Errors and limits. Rate limiting, timeouts and error messages are set by your server, not OpenAI. Make sure your retry logic handles whatever your server returns.

A safe way to switch

  1. Move the base URL, key and model name into configuration.
  2. Collect 30 to 50 real requests from production logs, with the outputs you got.
  3. Replay them against the new endpoint and compare. Look hardest at anything parsed by code: JSON, labels, extracted fields.
  4. Fix the prompts that need it. Test any tool calling or structured output explicitly.
  5. Send a small share of live traffic to the new endpoint, watch errors and latency, then increase it.

Step 5 is easy precisely because the API is compatible: the same code can talk to either backend, so splitting traffic is a configuration change.

Running your own OpenAI-compatible endpoint with AI Refinery

If the endpoint you are switching to is a model on your own infrastructure, AI Refinery, a self-hosted LLM platform, provides it. Every model you deploy from its catalog gets an OpenAI-compatible /v1/chat/completions endpoint, and you manage the API tokens your applications use from the console. The rest of your code stays as it is.

If you are still deciding whether to move off a hosted API at all, start with private LLM vs OpenAI API. For what it takes to run the model behind the endpoint, see the self-hosted LLM guide. To put one endpoint in front of several models, read what an LLM gateway does, and to give staff a private chat assistant on the same endpoint, see the self-hosted ChatGPT alternative guide.