Azure OpenAI Alternative: Run Open Models in Your Own Tenant

7 min read

The most practical Azure OpenAI alternative for teams that want data to stay in their own Azure tenant is to run an open-weight model on GPU virtual machines in your own subscription and expose it through an OpenAI-compatible endpoint. Your application keeps the same request format, the model never changes unless you change it, and no Microsoft-operated service processes the prompts. The cost is that you now own GPU capacity and operations, and that no open model matches the very best hosted GPT models on every task.

Before you switch, be clear about what Azure OpenAI already gives you. Many teams look for an alternative because of a privacy worry that the service has already addressed. The good reasons to move are narrower and more specific, and this post goes through them, the options, and the code change.

What Azure OpenAI already does for your data

Microsoft's data privacy page for Azure OpenAI states that prompts and completions are not used to train, retrain or improve the base models. You can also choose where processing happens: provisioned deployments come in Global, Data Zone (US or EU) and Regional types, and the resource can be locked to private endpoints inside your virtual network.

If your concern is "OpenAI trains on our data", Azure OpenAI is already the answer and you do not need to self-host.

What it does not give you is a service where only your own systems ever see the data. Two details on the same Microsoft page matter for strict environments:

  • Abuse monitoring. When the system detects warning signs, a sample of prompts and completions can be selected for review, with human reviewers involved when needed. Eligible customers can apply for modified abuse monitoring, which needs Microsoft's approval.
  • Stateful features. The Responses API, Assistants threads, files, vector stores, batch jobs, fine-tuning and stored completions keep data in the service, separate from the abuse-monitoring settings.

For most companies this is acceptable. For a bank's internal investigations team, a defense contractor or a hospital research group, "a sample may be reviewed unless we are approved for an exception" can be the sentence that blocks the project.

Reasons teams look for an Azure OpenAI alternative

These are the reasons that hold up in practice, roughly in the order we hear them.

  1. A policy that no third-party service may process certain data, including the cloud provider's AI service. The data can live in Azure storage, but inference must run on compute you control.
  2. Capacity. Standard deployments are limited by tokens-per-minute quota per region, per model and per deployment type. Guaranteed throughput means provisioned throughput units (PTUs), which have model-specific minimum deployment sizes and are billed per PTU per hour whether you use them or not.
  3. Model lifecycle. Each model version has a date on Microsoft's model retirement schedule. When it arrives you must move to a newer model, retest prompts and accept any change in behavior. A self-hosted model file stays exactly as it is until you replace it.
  4. Cost at steady volume. Per-token billing is cheap for spiky, low-volume work and expensive for a pipeline that runs all day. Our self-hosted LLM cost model shows where the break-even point usually sits.
  5. Air-gapped or sovereign networks where no cloud AI endpoint is reachable at all.

If none of these apply to you, stay on Azure OpenAI. The rest of this post assumes at least one does.

The options compared

Option Where inference runs Models Who runs the GPUs
Azure OpenAI Microsoft-operated service, region or zone you choose OpenAI models on the service Microsoft
Open models in Microsoft Foundry Microsoft-operated serverless or managed compute Catalog models, including Hugging Face models Microsoft
Self-hosted on Azure GPU VMs or AKS VMs in your own subscription and virtual network Any open-weight model you are licensed to run You
Another hosted API The vendor's cloud That vendor's models The vendor

Open models in Foundry are worth a look if you only need a model that Azure OpenAI does not offer. They do not change who operates the service. Self-hosting is the only option in the table where the inference process runs on machines you administer.

Running open models in your own Azure subscription

The work splits into three parts: choosing a GPU size, choosing a model that fits it, and putting the endpoint on a private network.

GPU sizes that fit LLM inference

Azure's H100 sizes cover most inference needs:

VM size GPUs GPU memory Good for
Standard_NC40ads_H100_v5 1 x H100 NVL 94 GB One 70B model at 4-bit, or several smaller models
Standard_NC80adis_H100_v5 2 x H100 NVL 188 GB 70B at 8-bit, or more concurrent users
Standard_ND96isr_H100_v5 8 x H100 640 GB Very large models and high-throughput serving

GPU VM quota is assigned per VM family and region, and new subscriptions often start with none. Request it before you plan a go-live date.

Picking a model that fits

Size the model from memory, not from parameter count alone. A Llama 3.3 70B model at 4-bit (Q4_K_M) needs about 50 GB for one user at an 8K context, which fits on a single 94 GB H100 NVL with room left for more users' KV cache. The numbers behind that, including how the KV cache grows with context, are in our post on Llama 70B VRAM requirements.

Many workloads that run on a GPT model today do not need a 70B model. Classification, extraction, summarization of internal documents and routing often work well on 8B to 14B models, which fit on a much smaller GPU. Test on your own prompts before deciding, and check each model's license for commercial use.

Keeping the endpoint private

Put the inference VMs in a private subnet with no public IP. Expose the endpoint through an internal load balancer, reach it from your application subnets or over your existing VPN or ExpressRoute, and restrict it with network security groups. If you already run AKS, GPU node pools are a reasonable home for inference; deploying an LLM on Kubernetes walks through it.

Changing the client code

Azure OpenAI uses its own client class with an API version and a deployment name. A self-hosted OpenAI-compatible endpoint uses the standard client with a base URL and a model name.

# Before: Azure OpenAI
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://my-resource.openai.azure.com",
    api_key=AZURE_OPENAI_KEY,
    api_version="2024-10-21",
)
resp = client.chat.completions.create(
    model="my-gpt-deployment",
    messages=[{"role": "user", "content": "Summarize this contract clause..."}],
)

# After: open model in your own subscription
from openai import OpenAI

client = OpenAI(
    base_url="https://llm.internal.example.com/v1",
    api_key=INTERNAL_API_TOKEN,
)
resp = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Summarize this contract clause..."}],
)

Plain chat completions, streaming and system prompts carry over. Features tied to the Azure service, such as its content filters, Assistants and stored completions, do not, so list what your application uses before you start. The OpenAI-compatible API guide covers the differences in more detail.

What you give up

Be honest with yourself about these before you commit:

  • Quality on the hardest tasks. Open models are close on many business tasks and behind on some, especially long multi-step reasoning. Measure the gap on your own evaluation set.
  • Operations. Someone has to patch the VMs, watch GPU memory, restart failed servers and plan upgrades.
  • Idle cost. A reserved H100 costs the same at 3 a.m. as at noon.
  • Built-in content filtering. Azure OpenAI applies content filters by default. On your own endpoint, any filtering is yours to add.

When Azure OpenAI is still the right choice

Stay on Azure OpenAI when your application depends on frontier GPT quality and you cannot find an open model that passes your tests, when traffic is small or unpredictable, or when nobody on the team wants to own GPU infrastructure. Plenty of companies run both: Azure OpenAI for the hardest general tasks and a self-hosted model for sensitive or high-volume work, behind one LLM gateway.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, installs inside your own infrastructure, which can be your Azure subscription and virtual network. You pick a model from its curated catalog of open-source text, embedding and vision-language models, choose a benchmarked quantization variant with its GPU memory need listed, and launch an inference server with one click. Each deployed model exposes an OpenAI-compatible /v1/chat/completions endpoint, and RBAC, SSO, approval workflows and an audit log come built in. You provide the GPUs.

It is not a way to run GPT models privately: OpenAI's hosted models are not open-weight and are not in the catalog. Plans are sized by the largest model you run; see the private LLM platform plans, or write to [email protected] with your workload and we will tell you plainly whether it fits.

A short decision checklist

  • Can you name the specific reason in the list above that rules out Azure OpenAI?
  • Have you tested at least two open models on 50 to 100 real prompts from your application?
  • Do you have GPU VM quota in the region you need?
  • Does your application use any Azure-only features that must be replaced?
  • Is someone assigned to run the inference servers after launch?

If you checked all five, a self-hosted model in your own tenant is a sound move. If you could not check the first, keep the service you have.