LLM Gateway: What It Does and How to Run One Self-Hosted
An LLM gateway is a service that sits between your applications and your models and gives them one endpoint to call. Behind that endpoint it handles the work every team would otherwise repeat: checking API keys, enforcing rate limits, choosing which model gets the request, retrying on another model when one fails, logging the call and recording who should be charged for it.
If you run one model for one app, you do not need a gateway. Once you have several models, several teams and someone asking how many tokens the marketing team used last month, you do.
This guide covers what a gateway does, whether to build or adopt one, the main open-source options, and where it sits in a self-hosted stack.
What an LLM gateway does
Think of it as an API gateway that understands tokens and models. The common jobs:
- One endpoint for many models. Apps call one URL and name a model, or an alias like
summarizer. The gateway maps that name to a real server. Swapping the model behind an alias becomes a config change, not a code change in every app. - Authentication and API keys. Each app or team gets its own key. You can revoke one without touching the others, and every request is tied to an owner. Gateway auth is one item in a wider self-hosted LLM security checklist.
- Rate limits. Requests per minute and tokens per minute, per key or per team, so one batch job cannot starve an interactive app.
- Routing. Choosing a model based on the request: its type, its difficulty, the current load. We cover the strategies in LLM routing explained.
- Fallback. If a model times out, errors or rejects a long prompt, retry on another.
- Logging and metrics. A single place to record latency, tokens and errors for every call, which feeds your LLM observability dashboards.
- Cost attribution. Token counts per key, rolled up per team, so usage can be charged back or at least seen.
The format almost every gateway accepts is the OpenAI API, so apps written against the OpenAI SDK only need a new base URL. The OpenAI-compatible API guide shows that change in Python, Node.js and LangChain.
Cost attribution on your own hardware
With a hosted API, cost attribution is token count times price. On self-hosted GPUs there is no per-token price, so you split the fixed GPU cost by usage instead. A simple method is to share GPU hours in proportion to tokens:
team share of GPU hours = team tokens / all tokens x total GPU hours
Say one GPU server runs all month, 24 x 30 = 720 GPU hours. Team A used 40 million tokens and the whole company used 100 million. Team A's share is 40 / 100 x 720 = 288 GPU hours. Multiply by your internal hourly rate for the hardware and you have a chargeback figure.
Count output tokens with a higher weight if you want more precision, because generating output one token at a time takes much more GPU time per token than processing the prompt. Our self-hosted LLM cost guide covers working out that hourly rate.
Build vs buy
A minimal gateway is a reverse proxy with an API key check, which a platform team can write in a day. The work is in everything after that: token-aware rate limiting, streaming responses that do not break when proxied, retries that respect timeouts, a key management UI, usage reports by team, and keeping up with API changes.
Build your own when:
- Your needs are small and fixed, such as one key per app and a log line per request.
- You already run an API gateway (Kong, Envoy, NGINX) and only need a few rules for LLM traffic.
- Security policy requires every component in the request path to be code you own.
Adopt an existing gateway when you need per-key budgets, token-based rate limits, multi-provider fallback or a usage UI. These are solved problems, and the open-source options are mature enough to run inside a private network.
Open-source LLM gateway options
All three below can be self-hosted. Features change quickly, so check each project's docs for the current list before you decide.
LiteLLM proxy
The LiteLLM proxy gives you one OpenAI-compatible endpoint for 100+ LLM providers, including your own OpenAI-compatible servers. Its docs describe virtual keys per user, team or project, budgets and rate limits through settings such as max_budget, tpm_limit and rpm_limit (budgets require a database), spend tracking per key and team, and logging to tools including OpenTelemetry and Prometheus. Its router supports strategies such as simple-shuffle, least-busy and latency-based-routing, plus fallbacks, including fallback to a model with a longer context window. Some features fall under a commercial license, so check which ones you need.
Pointing it at a self-hosted model takes a few lines of config, per the OpenAI-compatible provider docs:
model_list:
- model_name: summarizer
litellm_params:
model: openai/qwen2.5-14b-instruct
api_base: https://llm.internal.example.com/v1
api_key: os.environ/LLM_API_KEY
Portkey AI Gateway
The open-source Portkey AI Gateway (MIT) supports fallbacks to another model on failed requests, automatic retries with exponential backoff, weighted load balancing and input and output guardrails. It runs locally with npx @portkey-ai/gateway or in Docker. The company also sells a managed platform; semantic caching is listed as an enterprise feature.
Kong AI Gateway
If you already run Kong, Kong AI Gateway adds LLM-specific features to it: one API across providers, routing and load balancing with failover, semantic caching and an AI rate limiting policy that can enforce limits based on request cost. Its data plane nodes run in your environment and are managed through Kong's Konnect control plane, so check whether that model works in your network before you plan an air-gapped install.
Where the gateway sits in a self-hosted stack
From the outside in, a typical private deployment looks like this:
- Applications: internal tools, a chat UI, RAG pipelines, batch jobs.
- Corporate network controls: SSO at the edge for user-facing apps, firewall rules, TLS.
- LLM gateway: keys, limits, routing, fallback, logs, cost attribution.
- Inference servers: one per model, each exposing an OpenAI-compatible endpoint, on your GPUs.
- Observability and audit: metrics and logs from the gateway and the servers flowing into your monitoring.
The gateway is the one place every request crosses, which makes it the right place for policy and the wrong place for anything slow. Keep it stateless where you can, run at least two replicas, and measure the latency it adds to time to first token.
How OpenAI-compatible endpoints plus routing cover part of this
Many teams do not need a separate gateway at first. If every model already exposes an OpenAI-compatible endpoint, apps share one client library and one request format. If the platform issues API tokens per application, you have authentication and ownership. Add routing by task type and per-server monitoring, and you have covered:
- one request format across models
- per-app API tokens
- sending each request to a suitable model
- request, latency, token and error visibility
What remains are per-key rate limits, budgets, multi-provider fallback and cross-provider spend reports. If you need those, put a gateway such as LiteLLM in front of the endpoints. The two layers do not conflict: the gateway treats each self-hosted endpoint as one more OpenAI-compatible provider.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, covers the inner layers of that stack inside your own infrastructure: your cloud, your VPC, on premises or air-gapped. Every model you deploy from its catalog exposes an OpenAI-compatible /v1/chat/completions endpoint, and you manage the API tokens your applications use in the console. On Professional and Enterprise plans, smart model routing reads each request and sends it to the best available model for the task: vision, text, embeddings or code. Observability covers request volume, latency, token usage and errors for every inference server, and RBAC, SSO, size-tier controls and a full audit log handle governance.
It is not a full gateway product: per-key rate limiting, token budgets and fallback to hosted providers such as OpenAI are not part of it. If you need those, or you route traffic across several clouds and vendors, run a dedicated gateway like LiteLLM or Portkey in front of AI Refinery's endpoints. And if your only goal is one proxy in front of hosted APIs, with no self-hosted models at all, a gateway alone is the simpler choice.
Comparing gateway products? Our LiteLLM alternative post sorts them by category.