Amazon Bedrock Alternative: Private LLM Hosting in Your VPC

7 min read

If you want an Amazon Bedrock alternative for private model hosting, you have two real choices inside AWS: SageMaker AI endpoints, where AWS still manages the serving stack but you pick the instances and the model, or self-hosting open-weight models on EC2 or EKS GPU instances inside your own VPC. Self-hosting gives you the most control: the model is a file you pinned, inference runs on instances you administer, and no AWS AI service sees the prompts. It also makes you responsible for GPU capacity and operations.

Bedrock is a strong service, and for many teams the right answer is to keep it. This post covers what Bedrock already guarantees, the specific reasons people move off it, how the three options compare, and how to size a self-hosted setup on AWS.

What Bedrock already guarantees

AWS's Bedrock data protection page says model providers cannot access your prompts, completions or Bedrock service logs, because their software runs in AWS-owned accounts. AWS's security reference architecture for generative AI adds that Bedrock does not use your content to train models. You can reach Bedrock without crossing the internet through interface VPC endpoints, and Bedrock now offers an OpenAI-compatible Chat Completions API.

The catalog is wide. The models at a glance page lists Anthropic, Meta Llama, Mistral, Qwen, DeepSeek, Amazon Nova and OpenAI models, including the open-weight gpt-oss family. If your concern is only "will the model provider see our data", Bedrock already answers it.

The detail to read closely is the data retention page. In the default mode the model's own retention policy applies and AWS may keep data for safety and abuse-prevention purposes; for certain models, prompts and completions can be retained inside the AWS boundary for up to 30 days. Retention settings vary by model and region, so check the current page for the ones you use.

Reasons teams look for an Amazon Bedrock alternative

These are the reasons that hold up when you test them:

  1. Policy says no managed AI service may process the data, even one inside your cloud provider. The data may sit in S3, but inference has to run on compute you administer.
  2. Model lifecycle. Bedrock moves model versions through Active, Legacy and End-of-Life states, and after end-of-life, requests to that version fail. The model lifecycle page promises at least 12 months from launch to end-of-life, which is a planning horizon, not a guarantee that your prompts keep working forever.
  3. The exact model you want is not offered, or not in your region. Custom Model Import covers many architectures (Llama, Mistral, Mixtral, Qwen, GPT-OSS and others) with size limits, but not every model or every region.
  4. Cost at steady volume. Per-token pricing suits bursty or small workloads. A pipeline that processes documents around the clock is often cheaper on GPUs you keep busy; our self-hosted LLM cost model shows how to find the break-even point.
  5. Disconnected or air-gapped environments where no AWS service endpoint is reachable.

If none of these apply, stay on Bedrock.

Bedrock vs SageMaker vs self-hosting

Amazon Bedrock SageMaker AI endpoints Self-hosted on EC2 or EKS
What you manage Prompts and settings Model choice, instance type, scaling Everything from the OS up
Where inference runs AWS-managed service Instances AWS manages, attached to your VPC Instances in your account and VPC
Billing Per token (or provisioned throughput) Per instance while the endpoint is in service Per instance, plus your team's time
Models Bedrock catalog plus imported models Any model you can package Any open-weight model you are licensed to run
Model changes On the Bedrock lifecycle When you redeploy When you redeploy
Works air-gapped No No Yes, with the right setup

SageMaker AI real-time endpoints sit in the middle. You choose the instance type and the model, the endpoint can be placed in your VPC, and SageMaker can serve some containers through OpenAI-compatible APIs. AWS still runs the hosting layer, and endpoints are charged while they are in service whether or not they receive traffic. If your blocker is model choice or lifecycle rather than who operates the service, SageMaker may be enough.

Sizing a self-hosted setup on AWS

EC2 GPU instances for inference

Two instance families cover most LLM inference on AWS:

Instance GPUs GPU memory Good for
g6e.xlarge 1 x NVIDIA L40S 48 GB 8B to 32B models at 4- to 8-bit
g6e.48xlarge 8 x NVIDIA L40S 384 GB Several models, or one large model split across GPUs
p5.4xlarge 1 x NVIDIA H100 80 GB One 70B model at 4-bit
p5.48xlarge 8 x NVIDIA H100 640 GB Very large models, high concurrency

The G6e line has sizes between those two; check the instance page for the GPU count of each. GPU instances need service quota in your region, so request it early, and confirm the instance type is offered in the region you plan to use.

Matching the model to the GPU

Start from memory. Llama 3.3 70B at 4-bit (Q4_K_M) needs about 50 GB for one user at an 8K context, so it fits on a single 80 GB H100 but not on one 48 GB L40S. A 14B model at 8-bit is around 15 GB and fits on an L40S with plenty of room for concurrent users. Our post on Llama 70B VRAM requirements has the full tables by quantization and context length.

Many tasks people run on a large Bedrock model work well on a smaller open model. Test before you buy capacity for the biggest one.

EC2 or EKS

One or two models with steady traffic run fine on plain EC2 instances behind an internal Application Load Balancer. Once you have several models, several teams or a need for rolling upgrades, EKS with a GPU node group pays for itself; deploying an LLM on Kubernetes covers the GPU operator, model storage and autoscaling.

Keeping it private

Place the GPU instances in private subnets without public IPs, put the endpoint behind an internal load balancer, and allow traffic only from your application security groups. Model weights can be copied from S3 through a gateway VPC endpoint, so the instances need no internet access at all once the image is built.

Changing the client code

If your application already calls Bedrock's OpenAI-compatible Chat Completions API, moving to a self-hosted OpenAI-compatible endpoint is a base URL, a key and a model name. If it calls the native Bedrock Converse or InvokeModel APIs through boto3, you will rewrite the call site once, to the standard OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.internal.example.com/v1",
    api_key=INTERNAL_API_TOKEN,
)
resp = client.chat.completions.create(
    model="qwen3-32b-instruct",
    messages=[{"role": "user", "content": "Classify this support ticket..."}],
)

Bedrock-specific features such as Guardrails, Knowledge Bases and Agents do not come with you. List which ones your application depends on before you plan the move. The OpenAI-compatible API guide covers what transfers cleanly.

What you give up by leaving Bedrock

  • Access to closed models such as Claude, which are not available as open weights.
  • Per-token pricing for low or unpredictable traffic. Idle GPUs still cost money.
  • A managed service. Patching, monitoring, restarts and upgrades become your team's job.
  • Bedrock's managed extras, including Guardrails and Knowledge Bases, which you replace with your own components.

Many teams end up with both: Bedrock for closed models and light traffic, a self-hosted model for sensitive or high-volume work, and an LLM gateway in front so applications do not care which one answers.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, installs in your own infrastructure, which can be your AWS account and VPC. You choose a model from its curated catalog of open-source text, embedding and vision-language models, pick a benchmarked quantization variant with its GPU memory need and measured tokens per second listed, and launch an inference server with one click. Each model gets an OpenAI-compatible /v1/chat/completions endpoint, with RBAC, SSO, approval workflows and an audit log built in. You bring the EC2 GPU capacity.

It does not give you private access to closed models like Claude, and it is not a managed AWS service. If you need either, Bedrock is the better fit. Plans are sized by the largest model you run; see the self-hosted LLM plans, or write to [email protected] and describe your workload.

Decision checklist

  • You can name which reason above rules out Bedrock (and SageMaker, if model choice was the only issue).
  • You tested at least two open models on real prompts from your application.
  • You have GPU service quota for the instance type, in the region you need.
  • You listed the Bedrock-only features your application uses and how you will replace each.
  • Someone owns the inference servers after launch.

If all five hold, self-hosting in your VPC is a sound move. If the first does not, Bedrock is probably still the right place for this workload.