Llama 70B VRAM Requirements by Quantization and Context
The Llama 70B VRAM requirements come down to one number per quantization level, plus a KV cache that grows with context length and users. At 4-bit (Q4_K_M) the weights are about 42.5 GB, so one user at an 8K-token context needs roughly 50 GB of GPU memory. That fits on a single 80 GB card with room for several more users, and does not fit on a single 48 GB card.
This post works through the numbers for Llama 3.1 and 3.3 70B, which share the same architecture and therefore the same memory profile. For the general method behind them, see our guide to GPU memory for LLM inference.
VRAM by quantization level
Weights are the fixed cost. These are the approximate GGUF file sizes for Llama 3.3 70B Instruct, plus the KV cache for one user at 8K tokens (about 2.5 GB) and 10% for working buffers:
| Quantization | Weights | Total, 1 user at 8K | Quality note |
|---|---|---|---|
| Q2_K | ~26 GB | ~32 GB | Visible quality loss; last resort |
| Q3_K_M | ~34 GB | ~41 GB | Usable for light tasks |
| Q4_K_M | ~42.5 GB | ~50 GB | The usual default |
| Q5_K_M | ~50 GB | ~58 GB | Small gain over Q4 |
| Q6_K | ~58 GB | ~66 GB | Close to Q8 |
| Q8_0 | ~75 GB | ~85 GB | Near-lossless |
| FP16 | ~141 GB | ~158 GB | Reference quality |
The file sizes come from the published GGUF builds, so they move by a gigabyte or two between uploaders. Check the file list on the model page you actually download. Our LLM quantization guide covers what each level costs in quality.
The KV cache for a 70B model
Llama 3.1 and 3.3 70B have 80 layers, 8 key-value heads and a head size of 128. At 16-bit cache precision that is 320 KB per token:
2 x 80 layers x 8 KV heads x 128 head size x 2 bytes = 327,680 bytes per token
| Context length | KV cache, one user |
|---|---|
| 8K tokens | ~2.5 GB |
| 32K tokens | ~10 GB |
| 128K tokens | ~40 GB |
The cache is per conversation, so multiply by the number of simultaneous users. Many inference servers can store the cache at 8-bit instead, which roughly halves these figures at a small quality cost. The model's own config.json holds the layer and head counts if you want to check them.
Putting users and context together
For Q4_K_M weights (42.5 GB), 16-bit cache and 10% overhead:
| Scenario | Total memory |
|---|---|
| 1 user, 8K context | ~50 GB |
| 4 users, 8K context | ~58 GB |
| 8 users, 8K context | ~69 GB |
| 16 users, 8K context | ~91 GB |
| 2 users, 32K context | ~69 GB |
| 1 user, 128K context | ~91 GB |
The last row is the one that surprises people. A 70B model that fits comfortably at 8K needs two cards once you serve its full 128K window, because the cache alone is as large as the 4-bit weights. If your documents are short, set a smaller maximum context and spend the memory on concurrency instead.
Which GPUs fit
- One 80 GB card (A100 80GB, H100): Q4_K_M with up to about 8 users at 8K. Q5_K_M with a handful. Q8_0 does not fit.
- One 48 GB card (L40S, RTX 6000 Ada, A6000): only Q3_K_M or lower, with one user and a short context. Not recommended for a shared service.
- Two 48 GB cards (96 GB total): Q4_K_M with 16 users at 8K, or Q8_0 for a single user. The model is split across the cards with tensor parallelism, so the link between them matters: NVLink is much faster than PCIe, and PCIe-only pairs lose some throughput.
- Four 24 GB cards (96 GB total): works for Q4_K_M, but each card holds a quarter of the model plus its own buffers, and the PCIe traffic between four cards costs speed.
- Two 80 GB cards: Q8_0 with real concurrency, or FP16 for one user with very little to spare.
Splitting a model across GPUs never gives you the full sum of their memory. Each card needs room for its own shard, activations and buffers, so plan on 5 to 10% less than the total.
Choosing a quantization for 70B
Start with Q4_K_M. It is the default for good reason: it cuts memory by about 70% compared to FP16 and, on most tasks, the difference is hard to measure. Step up to Q6_K or Q8_0 only if you have GPU memory left over and an evaluation showing your task benefits. Step down to Q3_K_M only when the card you have forces it, and test on your own prompts first, because the loss shows up unevenly: long reasoning and code suffer before simple chat does.
Compare against a smaller model before paying for 70B at all. A 32B model at Q8_0 uses about 40 GB and often matches an older 70B on narrow tasks. The open-source LLM for enterprise post covers how to choose by task and license, and Qwen VRAM requirements has the same sizing for the Qwen family.
How this shows up in AI Refinery
AI Refinery, a self-hosted LLM platform, benchmarks every quantization variant before it reaches the catalog and lists the GPU memory it needs, counting the weights plus the KV cache at the model's full context window. That is why its figure for a 70B model can look larger than the 8K numbers above: it reserves the full window rather than assuming short prompts. The Professional plan covers models up to 70B parameters, so this class of model is within it; see plans for details. The customer provides the GPUs.
If you are still working out whether a 70B model is worth the hardware, the self-hosted LLM cost model shows where owning GPUs breaks even against API pricing. For the broader picture, start from the home page: AI Refinery's private model catalog.
Figures are estimates. Measure on your own hardware before buying GPUs.