Amazon Bedrock Alternative: Private LLM Hosting in Your VPC
If you want an Amazon Bedrock alternative for private model hosting, you have two real choices inside AWS: SageMaker AI endpoints, where AWS still manages the serving stack but you pick the instances and the model, or self-hosting open-weight models on EC2 or EKS GPU instances inside your own VPC. Self-hosting gives you the most control: the model is a file you pinned, inference runs on instances you administer, and no AWS AI service sees the prompts. It also makes you responsible for GPU capacity and operations.
Bedrock is a strong service, and for many teams the right answer is to keep it. This post covers what Bedrock already guarantees, the specific reasons people move off it, how the three options compare, and how to size a self-hosted setup on AWS.
What Bedrock already guarantees
AWS's Bedrock data protection page says model providers cannot access your prompts, completions or Bedrock service logs, because their software runs in AWS-owned accounts. AWS's security reference architecture for generative AI adds that Bedrock does not use your content to train models. You can reach Bedrock without crossing the internet through interface VPC endpoints, and Bedrock now offers an OpenAI-compatible Chat Completions API.
The catalog is wide. The models at a glance page lists Anthropic, Meta Llama, Mistral, Qwen, DeepSeek, Amazon Nova and OpenAI models, including the open-weight gpt-oss family. If your concern is only "will the model provider see our data", Bedrock already answers it.
The detail to read closely is the data retention page. In the default mode the model's own retention policy applies and AWS may keep data for safety and abuse-prevention purposes; for certain models, prompts and completions can be retained inside the AWS boundary for up to 30 days. Retention settings vary by model and region, so check the current page for the ones you use.
Reasons teams look for an Amazon Bedrock alternative
These are the reasons that hold up when you test them:
- Policy says no managed AI service may process the data, even one inside your cloud provider. The data may sit in S3, but inference has to run on compute you administer.
- Model lifecycle. Bedrock moves model versions through Active, Legacy and End-of-Life states, and after end-of-life, requests to that version fail. The model lifecycle page promises at least 12 months from launch to end-of-life, which is a planning horizon, not a guarantee that your prompts keep working forever.
- The exact model you want is not offered, or not in your region. Custom Model Import covers many architectures (Llama, Mistral, Mixtral, Qwen, GPT-OSS and others) with size limits, but not every model or every region.
- Cost at steady volume. Per-token pricing suits bursty or small workloads. A pipeline that processes documents around the clock is often cheaper on GPUs you keep busy; our self-hosted LLM cost model shows how to find the break-even point.
- Disconnected or air-gapped environments where no AWS service endpoint is reachable.
If none of these apply, stay on Bedrock.
Bedrock vs SageMaker vs self-hosting
| Amazon Bedrock | SageMaker AI endpoints | Self-hosted on EC2 or EKS | |
|---|---|---|---|
| What you manage | Prompts and settings | Model choice, instance type, scaling | Everything from the OS up |
| Where inference runs | AWS-managed service | Instances AWS manages, attached to your VPC | Instances in your account and VPC |
| Billing | Per token (or provisioned throughput) | Per instance while the endpoint is in service | Per instance, plus your team's time |
| Models | Bedrock catalog plus imported models | Any model you can package | Any open-weight model you are licensed to run |
| Model changes | On the Bedrock lifecycle | When you redeploy | When you redeploy |
| Works air-gapped | No | No | Yes, with the right setup |
SageMaker AI real-time endpoints sit in the middle. You choose the instance type and the model, the endpoint can be placed in your VPC, and SageMaker can serve some containers through OpenAI-compatible APIs. AWS still runs the hosting layer, and endpoints are charged while they are in service whether or not they receive traffic. If your blocker is model choice or lifecycle rather than who operates the service, SageMaker may be enough.
Sizing a self-hosted setup on AWS
EC2 GPU instances for inference
Two instance families cover most LLM inference on AWS:
| Instance | GPUs | GPU memory | Good for |
|---|---|---|---|
| g6e.xlarge | 1 x NVIDIA L40S | 48 GB | 8B to 32B models at 4- to 8-bit |
| g6e.48xlarge | 8 x NVIDIA L40S | 384 GB | Several models, or one large model split across GPUs |
| p5.4xlarge | 1 x NVIDIA H100 | 80 GB | One 70B model at 4-bit |
| p5.48xlarge | 8 x NVIDIA H100 | 640 GB | Very large models, high concurrency |
The G6e line has sizes between those two; check the instance page for the GPU count of each. GPU instances need service quota in your region, so request it early, and confirm the instance type is offered in the region you plan to use.
Matching the model to the GPU
Start from memory. Llama 3.3 70B at 4-bit (Q4_K_M) needs about 50 GB for one user at an 8K context, so it fits on a single 80 GB H100 but not on one 48 GB L40S. A 14B model at 8-bit is around 15 GB and fits on an L40S with plenty of room for concurrent users. Our post on Llama 70B VRAM requirements has the full tables by quantization and context length.
Many tasks people run on a large Bedrock model work well on a smaller open model. Test before you buy capacity for the biggest one.
EC2 or EKS
One or two models with steady traffic run fine on plain EC2 instances behind an internal Application Load Balancer. Once you have several models, several teams or a need for rolling upgrades, EKS with a GPU node group pays for itself; deploying an LLM on Kubernetes covers the GPU operator, model storage and autoscaling.
Keeping it private
Place the GPU instances in private subnets without public IPs, put the endpoint behind an internal load balancer, and allow traffic only from your application security groups. Model weights can be copied from S3 through a gateway VPC endpoint, so the instances need no internet access at all once the image is built.
Changing the client code
If your application already calls Bedrock's OpenAI-compatible Chat Completions API, moving to a self-hosted OpenAI-compatible endpoint is a base URL, a key and a model name. If it calls the native Bedrock Converse or InvokeModel APIs through boto3, you will rewrite the call site once, to the standard OpenAI client:
from openai import OpenAI
client = OpenAI(
base_url="https://llm.internal.example.com/v1",
api_key=INTERNAL_API_TOKEN,
)
resp = client.chat.completions.create(
model="qwen3-32b-instruct",
messages=[{"role": "user", "content": "Classify this support ticket..."}],
)
Bedrock-specific features such as Guardrails, Knowledge Bases and Agents do not come with you. List which ones your application depends on before you plan the move. The OpenAI-compatible API guide covers what transfers cleanly.
What you give up by leaving Bedrock
- Access to closed models such as Claude, which are not available as open weights.
- Per-token pricing for low or unpredictable traffic. Idle GPUs still cost money.
- A managed service. Patching, monitoring, restarts and upgrades become your team's job.
- Bedrock's managed extras, including Guardrails and Knowledge Bases, which you replace with your own components.
Many teams end up with both: Bedrock for closed models and light traffic, a self-hosted model for sensitive or high-volume work, and an LLM gateway in front so applications do not care which one answers.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, installs in your own infrastructure, which can be your AWS account and VPC. You choose a model from its curated catalog of open-source text, embedding and vision-language models, pick a benchmarked quantization variant with its GPU memory need and measured tokens per second listed, and launch an inference server with one click. Each model gets an OpenAI-compatible /v1/chat/completions endpoint, with RBAC, SSO, approval workflows and an audit log built in. You bring the EC2 GPU capacity.
It does not give you private access to closed models like Claude, and it is not a managed AWS service. If you need either, Bedrock is the better fit. Plans are sized by the largest model you run; see the self-hosted LLM plans, or write to [email protected] and describe your workload.
Decision checklist
- You can name which reason above rules out Bedrock (and SageMaker, if model choice was the only issue).
- You tested at least two open models on real prompts from your application.
- You have GPU service quota for the instance type, in the region you need.
- You listed the Bedrock-only features your application uses and how you will replace each.
- Someone owns the inference servers after launch.
If all five hold, self-hosting in your VPC is a sound move. If the first does not, Bedrock is probably still the right place for this workload.