GPU Memory for LLM Inference: How Much VRAM You Need
To estimate GPU memory for LLM inference, add three things: the model weights, the KV cache for the conversations in flight, and about 10% extra for working buffers. The weights are easy (parameters times bytes per parameter). The KV cache is the part people forget, and at long context lengths or with many users it can be bigger than the model itself.
As a quick guide for one user at an 8K-token context:
| Model size | 4-bit | 8-bit | 16-bit |
|---|---|---|---|
| 8B | ~7 GB | ~11 GB | ~19 GB |
| 14B | ~11 GB | ~18 GB | ~33 GB |
| 32B | ~23 GB | ~40 GB | ~73 GB |
| 70B | ~49 GB | ~85 GB | ~157 GB |
The rest of this post shows where those numbers come from, so you can work out your own.
Part 1: the model weights
Every parameter takes a fixed amount of memory depending on its precision:
weights (GB) = parameters (billions) x bytes per parameter
| Precision | Bytes per parameter |
|---|---|
| FP16 / BF16 (unquantized) | 2 |
| 8-bit (e.g. Q8_0) | ~1.06 |
| 4-bit (e.g. Q4_K_M) | ~0.6 |
So an 8B model is about 16 GB at 16-bit and about 5 GB at 4-bit. A 70B model is about 140 GB at 16-bit and about 42 GB at 4-bit. The 4-bit and 8-bit figures are a little above 0.5 and 1 because quantized formats also store scaling values. Our LLM quantization guide explains what you give up at each level.
Part 2: the KV cache
While a model reads your prompt and writes its answer, it keeps two vectors (a "key" and a "value") for every token, in every layer. That store is the KV cache, and it grows with every token of context:
KV cache (bytes) = 2 x layers x KV heads x head size x bytes per value x tokens
The layer and head numbers are in each model's config.json. Take Llama 3.1 8B: 32 layers, 8 KV heads, head size 128, cache stored at 16-bit (2 bytes):
2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KB per token
At an 8K-token context that is 1 GB. At the model's full 128K context it is 16 GB, twice the size of the model's own 16-bit weights.
Llama 3.1 70B has 80 layers with the same head layout, which works out to 320 KB per token: 2.5 GB at 8K, and 40 GB at 128K.
Users multiply it
The KV cache is per conversation. If 20 people use that 8B model at the same time with 8K-token contexts, the cache needs 20 GB, more than the 4-bit weights several times over. This is the most common reason a model that "fits" during testing runs out of memory in production.
Two settings control it. A shorter maximum context reduces the cache in direct proportion, so do not serve 128K if your prompts never pass 8K. Some inference servers can also store the cache at 8-bit, which roughly halves it with a small quality cost. For how GQA, paged attention and prefix caching change these numbers, read KV cache explained.
Part 3: overhead
Inference servers need some extra GPU memory for temporary buffers during computation and for the GPU runtime itself. Budget about 10% of weights plus cache, and at least 1 GB. Some servers reserve a fixed share of the GPU up front and fill it with KV cache, so the memory reported as "used" may be higher than the model strictly needs.
Worked example: GPU memory for LLM serving
Say you want to serve Llama 3.1 8B at 8-bit to 10 simultaneous users, each with up to 8K tokens of context.
weights: 8 x 1.06 = 8.5 GB
KV cache: 10 users x 1 GB = 10.0 GB
overhead: 10% of 18.5 GB = 1.9 GB
total ~ 20.4 GB
That fits on a 24 GB GPU with a little room. At 4-bit the weights drop to about 5 GB and the total to about 17 GB. At 32K context per user, the cache alone becomes 40 GB and you need a 48 GB card or a shorter limit.
Which GPUs fit which models
Common GPU memory sizes and what they hold comfortably for a handful of users at moderate context:
| GPU memory | Example GPUs | Fits comfortably |
|---|---|---|
| 16 GB | RTX 4080, RTX 5070 Ti | 8B at 4-bit or 8-bit |
| 24 GB | RTX 4090, NVIDIA L4 | 8B at 16-bit, 14B at 8-bit |
| 32 GB | RTX 5090 | 32B at 4-bit |
| 48 GB | NVIDIA L40S, RTX 6000 Ada | 32B at 8-bit (70B at 4-bit only with a very short context) |
| 80 GB | NVIDIA A100 80GB, H100 | 70B at 4-bit with room for users |
| 141 GB+ | NVIDIA H200, AMD MI300X | 70B at 8-bit |
Larger models, or 70B at 16-bit, need several GPUs. Inference servers can split a model across GPUs in one machine (tensor parallelism), which works well over fast links between the cards. Splitting across machines is possible but slower. To compare specific cards by memory, bandwidth and FP8 support, see the best GPU for LLM inference.
If a model does not fit, some servers will run the remaining layers on the CPU from system RAM. It works, but generation slows down sharply, often by many times. Treat it as a way to try a model, not a way to serve one.
A sizing checklist
- Choose the model and precision, and calculate the weights.
- Decide the longest context you will actually allow, not the model's maximum.
- Estimate peak simultaneous requests, not total users.
- Calculate the KV cache for that context and that concurrency.
- Add 10% overhead, then pick a GPU with at least 10-20% to spare.
- Load test with realistic prompts before you buy more of the same.
How AI Refinery helps with sizing
Every model variant in the catalog of AI Refinery, a self-hosted LLM platform, shows the GPU memory it needs, calculated from its weights and the KV cache at the model's full context window, along with its size on disk and benchmarked tokens per second. Because that figure assumes the full context, serving with a shorter context needs less, exactly as the KV cache formula above predicts.
AI Refinery plans are also sized by model: up to 14B parameters, up to 70B, or unlimited on Enterprise. Compare them on the pricing section. For the bigger picture of running models on your own hardware, read the self-hosted LLM guide, and to turn the hardware into a monthly figure, see self-hosted LLM cost.
For per-model sizing, see Llama 70B VRAM requirements, Qwen VRAM requirements, DeepSeek R1 hardware requirements, gpt-oss hardware requirements and Gemma VRAM requirements.