GPU Memory for LLM Inference: How Much VRAM You Need

5 min read

To estimate GPU memory for LLM inference, add three things: the model weights, the KV cache for the conversations in flight, and about 10% extra for working buffers. The weights are easy (parameters times bytes per parameter). The KV cache is the part people forget, and at long context lengths or with many users it can be bigger than the model itself.

As a quick guide for one user at an 8K-token context:

Model size 4-bit 8-bit 16-bit
8B ~7 GB ~11 GB ~19 GB
14B ~11 GB ~18 GB ~33 GB
32B ~23 GB ~40 GB ~73 GB
70B ~49 GB ~85 GB ~157 GB

The rest of this post shows where those numbers come from, so you can work out your own.

Part 1: the model weights

Every parameter takes a fixed amount of memory depending on its precision:

weights (GB) = parameters (billions) x bytes per parameter
Precision Bytes per parameter
FP16 / BF16 (unquantized) 2
8-bit (e.g. Q8_0) ~1.06
4-bit (e.g. Q4_K_M) ~0.6

So an 8B model is about 16 GB at 16-bit and about 5 GB at 4-bit. A 70B model is about 140 GB at 16-bit and about 42 GB at 4-bit. The 4-bit and 8-bit figures are a little above 0.5 and 1 because quantized formats also store scaling values. Our LLM quantization guide explains what you give up at each level.

Part 2: the KV cache

While a model reads your prompt and writes its answer, it keeps two vectors (a "key" and a "value") for every token, in every layer. That store is the KV cache, and it grows with every token of context:

KV cache (bytes) = 2 x layers x KV heads x head size x bytes per value x tokens

The layer and head numbers are in each model's config.json. Take Llama 3.1 8B: 32 layers, 8 KV heads, head size 128, cache stored at 16-bit (2 bytes):

2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KB per token

At an 8K-token context that is 1 GB. At the model's full 128K context it is 16 GB, twice the size of the model's own 16-bit weights.

Llama 3.1 70B has 80 layers with the same head layout, which works out to 320 KB per token: 2.5 GB at 8K, and 40 GB at 128K.

Users multiply it

The KV cache is per conversation. If 20 people use that 8B model at the same time with 8K-token contexts, the cache needs 20 GB, more than the 4-bit weights several times over. This is the most common reason a model that "fits" during testing runs out of memory in production.

Two settings control it. A shorter maximum context reduces the cache in direct proportion, so do not serve 128K if your prompts never pass 8K. Some inference servers can also store the cache at 8-bit, which roughly halves it with a small quality cost. For how GQA, paged attention and prefix caching change these numbers, read KV cache explained.

Part 3: overhead

Inference servers need some extra GPU memory for temporary buffers during computation and for the GPU runtime itself. Budget about 10% of weights plus cache, and at least 1 GB. Some servers reserve a fixed share of the GPU up front and fill it with KV cache, so the memory reported as "used" may be higher than the model strictly needs.

Worked example: GPU memory for LLM serving

Say you want to serve Llama 3.1 8B at 8-bit to 10 simultaneous users, each with up to 8K tokens of context.

weights:   8 x 1.06              =  8.5 GB
KV cache:  10 users x 1 GB       = 10.0 GB
overhead:  10% of 18.5 GB        =  1.9 GB
total                            ~ 20.4 GB

That fits on a 24 GB GPU with a little room. At 4-bit the weights drop to about 5 GB and the total to about 17 GB. At 32K context per user, the cache alone becomes 40 GB and you need a 48 GB card or a shorter limit.

Which GPUs fit which models

Common GPU memory sizes and what they hold comfortably for a handful of users at moderate context:

GPU memory Example GPUs Fits comfortably
16 GB RTX 4080, RTX 5070 Ti 8B at 4-bit or 8-bit
24 GB RTX 4090, NVIDIA L4 8B at 16-bit, 14B at 8-bit
32 GB RTX 5090 32B at 4-bit
48 GB NVIDIA L40S, RTX 6000 Ada 32B at 8-bit (70B at 4-bit only with a very short context)
80 GB NVIDIA A100 80GB, H100 70B at 4-bit with room for users
141 GB+ NVIDIA H200, AMD MI300X 70B at 8-bit

Larger models, or 70B at 16-bit, need several GPUs. Inference servers can split a model across GPUs in one machine (tensor parallelism), which works well over fast links between the cards. Splitting across machines is possible but slower. To compare specific cards by memory, bandwidth and FP8 support, see the best GPU for LLM inference.

If a model does not fit, some servers will run the remaining layers on the CPU from system RAM. It works, but generation slows down sharply, often by many times. Treat it as a way to try a model, not a way to serve one.

A sizing checklist

  1. Choose the model and precision, and calculate the weights.
  2. Decide the longest context you will actually allow, not the model's maximum.
  3. Estimate peak simultaneous requests, not total users.
  4. Calculate the KV cache for that context and that concurrency.
  5. Add 10% overhead, then pick a GPU with at least 10-20% to spare.
  6. Load test with realistic prompts before you buy more of the same.

How AI Refinery helps with sizing

Every model variant in the catalog of AI Refinery, a self-hosted LLM platform, shows the GPU memory it needs, calculated from its weights and the KV cache at the model's full context window, along with its size on disk and benchmarked tokens per second. Because that figure assumes the full context, serving with a shorter context needs less, exactly as the KV cache formula above predicts.

AI Refinery plans are also sized by model: up to 14B parameters, up to 70B, or unlimited on Enterprise. Compare them on the pricing section. For the bigger picture of running models on your own hardware, read the self-hosted LLM guide, and to turn the hardware into a monthly figure, see self-hosted LLM cost.

For per-model sizing, see Llama 70B VRAM requirements, Qwen VRAM requirements, DeepSeek R1 hardware requirements, gpt-oss hardware requirements and Gemma VRAM requirements.