Self-Hosted LLM Cost: A Break-Even Model vs API Pricing
The self-hosted LLM cost is mostly fixed: you pay for a GPU every hour whether it serves one request or a million. A per-token API is the opposite: you pay nothing when idle and a fixed price for every token. So the comparison always comes down to one question: how many tokens will you actually push through the hardware each month?
Below that volume the API is cheaper. Above it, self-hosting wins, often by a wide margin. This post builds the cost model piece by piece, works through a break-even example with every step of the arithmetic shown, and explains why utilization matters more than the GPU price.
All prices below were checked on 3 November 2026 on the pages linked. They change often, so plug in current numbers before you decide anything.
The full self-hosted LLM cost model
Monthly cost has four parts:
monthly cost = hardware + power + operations + software
Hardware. Either cloud GPU hours or a purchase spread over its useful life. For cloud, multiply the hourly rate by the hours you keep it running (730 hours is a full month). For a purchase, divide the price by the months you expect to use it; 36 months is a common planning figure.
Power. Only for hardware you own. Watts x hours / 1,000 gives kWh, multiplied by your electricity price. Include the host server and cooling as well as the GPU.
Operations. The person-hours to patch, monitor, upgrade models and answer "why is it slow". This is the line item teams forget, and as you will see it can be the largest one.
Software. Anything you license to run the stack, such as a platform or support contract.
Two numbers then turn cost into a comparison: the throughput your setup delivers (tokens per second at your real concurrency, which you measure) and your utilization (the share of that capacity you actually use).
The prices used in this example
Cloud GPU, one H100 SXM 80 GB:
- RunPod: $3.99 per hour on Secure Cloud.
- Lambda: H100 SXM from $3.99 to $4.29 per GPU per hour, depending on instance size.
Owned GPU, one RTX PRO 6000 Blackwell Max-Q (96 GB, 300 W maximum power according to NVIDIA's specifications):
- GPUPrix listed the lowest current US price at $17,063, against a 12-month median of $10,701. Prices for this card have moved a lot, so get a quote.
Power and labor:
- EIA: average US commercial electricity price of 14.47 cents per kWh (August 2026, preliminary).
- BLS: median pay for network and computer systems administrators of $99,130 a year (May 2025). That is salary only; a fully loaded cost is higher.
API prices per million tokens, from Google's Gemini API pricing page, standard tier:
gemini-3.1-flash-lite: $0.25 input, $1.50 output.gemini-3.1-pro-preview: $2.00 input, $12.00 output (prompts up to 200k tokens).
These are reference points, not like-for-like quality matches. A cheap API tier and a premium tier bracket most of what teams compare against.
A worked break-even example
Step 1: one blended API price
Chat and document workloads send more tokens in than they get back. Assume 3 input tokens for every output token. The blended price per million tokens is:
- Flash-Lite: (3 x $0.25 + 1 x $1.50) / 4 = $2.25 / 4 = $0.5625
- Pro: (3 x $2.00 + 1 x $12.00) / 4 = $18.00 / 4 = $4.50
Step 2: monthly cost of the cloud GPU
Running all month: $3.99 x 730 hours = $2,912.70.
Step 3: break-even volume
Break-even tokens per month = monthly cost / blended price per million.
- Against Flash-Lite: $2,912.70 / $0.5625 = 5,178 million tokens, about 5.2 billion tokens a month.
- Against Pro: $2,912.70 / $4.50 = 647 million tokens, about 650 million tokens a month.
To see what that means for the hardware, divide by the seconds in a month (730 x 3,600 = 2,628,000):
- 5,178,133,333 / 2,628,000 = about 1,970 tokens per second, every second, all month.
- 647,266,667 / 2,628,000 = about 246 tokens per second, all month.
If your measured throughput at realistic concurrency is comfortably above the figure for the API you would otherwise use, and you have the traffic to fill it, self-hosting is cheaper. If not, it is not.
Step 4: the owned-hardware version
For one RTX PRO 6000 in a server you already have:
- Hardware: $17,063 / 36 months = $473.97
- Power: assume 600 W for GPU plus host and cooling. 0.6 kW x 730 h = 438 kWh; 438 x $0.1447 = $63.38
- Operations: assume one tenth of an administrator's time. $99,130 x 0.1 / 12 = $826.08
- Total: $473.97 + $63.38 + $826.08 = $1,363.43 a month, before the host server and any software license.
Break-even: $1,363.43 / $0.5625 = 2,424 million tokens against Flash-Lite, and $1,363.43 / $4.50 = 303 million tokens against Pro.
Two things stand out. Power is the smallest line. And the ops estimate is larger than the GPU itself, which is why tooling that cuts ops time matters as much as cheaper hardware. A 96 GB card also fits a 70B model at 4-bit (70 x 0.6 = about 42 GB of weights) with room left for the KV cache; the GPU memory guide shows how to check that for your context length.
Why utilization dominates
A GPU's cost per million tokens is its hourly cost divided by the tokens it actually serves. Take a hypothetical setup that sustains 1,000 tokens per second under load (measure your own; it varies widely with model size, quantization and concurrency). At full load for a month it serves 1,000 x 2,628,000 = 2,628 million tokens.
| Utilization | Tokens served per month | Cost per million tokens ($2,912.70 / tokens) |
|---|---|---|
| 100% | 2,628 million | $1.11 |
| 30% | 788 million | $3.69 |
| 10% | 263 million | $11.08 |
The same GPU at the same price is cheaper than the Pro tier at full load and more than twice as expensive at 10%. Nothing about the hardware changed, only how busy it was.
Office-hours traffic is the usual cause. Ten hours a day for 22 working days is 220 of the month's 730 hours, about 30%. If the GPU runs all month for that, you pay for 510 idle hours. A cloud GPU you start and stop with the working day costs 220 x $3.99 = $877.80 instead of $2,912.70. Owned hardware cannot do that, which is why owned GPUs only pay off when you can keep them busy: batch jobs overnight, several teams sharing one model, or several applications behind one endpoint.
Quantization moves the same lever from the other side. A smaller quantized model serves more tokens per second on the same GPU, which lowers cost per token at any utilization. LLM quantization explained covers the quality cost.
When the API is cheaper
Take 200 staff, each making 40 requests a day of about 2,000 tokens (input plus output), on 22 working days:
200 x 40 x 2,000 x 22 = 352 million tokens a month.
- On Flash-Lite: 352 x $0.5625 = $198
- On Pro: 352 x $4.50 = $1,584
- On a cloud H100 during office hours only: $877.80
Against the premium tier, self-hosting during office hours already saves money. Against the cheap tier, the API costs less than a quarter as much, and no amount of tuning closes that gap at this volume.
So the API is usually cheaper when:
- Volume is low or unpredictable, so a GPU would sit idle most of the time.
- A low-cost API tier meets your quality bar.
- You have no one to own the infrastructure, so the ops line would be a new hire rather than a fraction of one.
- You are still experimenting and do not yet know which model or volume you need.
Cost is rarely the only reason to self-host. If data cannot leave your environment, the API is not an option at any price, and the question becomes how to run your own model efficiently. The private LLM vs OpenAI comparison covers the non-cost trade-offs, and LLM observability shows how to measure the utilization this whole model depends on.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, targets the two numbers that drive this calculation: ops time and utilization. It installs in your own cloud, VPC, on-premise or air-gapped network, and you launch an inference server from its catalog with one click instead of maintaining your own scripts. Each catalog variant shows its GPU memory needed and measured tokens per second before you deploy, so you can estimate throughput before buying hardware. The observability dashboard tracks request volume, latency, token usage and errors per inference server, which gives you the real utilization figure. Its plans are sized by model rather than by token; see pricing. You still provide the GPUs.
When it is not the right fit
If your volume is in the low hundreds of millions of tokens a month, your data is not sensitive and a low-cost API tier passes your quality tests, the API is cheaper and simpler, and a platform adds cost you do not need. The same applies if usage is too spiky to keep a GPU busy and you cannot schedule it on and off.
Managed GPU hosting is the other side of this comparison; see the Hugging Face Inference Endpoints alternative post. If you are on a cloud provider's AI service today, the Azure OpenAI alternative and Amazon Bedrock alternative posts cover moving that workload into your own subscription.