Llama 70B VRAM Requirements by Quantization and Context

5 min read

The Llama 70B VRAM requirements come down to one number per quantization level, plus a KV cache that grows with context length and users. At 4-bit (Q4_K_M) the weights are about 42.5 GB, so one user at an 8K-token context needs roughly 50 GB of GPU memory. That fits on a single 80 GB card with room for several more users, and does not fit on a single 48 GB card.

This post works through the numbers for Llama 3.1 and 3.3 70B, which share the same architecture and therefore the same memory profile. For the general method behind them, see our guide to GPU memory for LLM inference.

VRAM by quantization level

Weights are the fixed cost. These are the approximate GGUF file sizes for Llama 3.3 70B Instruct, plus the KV cache for one user at 8K tokens (about 2.5 GB) and 10% for working buffers:

Quantization Weights Total, 1 user at 8K Quality note
Q2_K ~26 GB ~32 GB Visible quality loss; last resort
Q3_K_M ~34 GB ~41 GB Usable for light tasks
Q4_K_M ~42.5 GB ~50 GB The usual default
Q5_K_M ~50 GB ~58 GB Small gain over Q4
Q6_K ~58 GB ~66 GB Close to Q8
Q8_0 ~75 GB ~85 GB Near-lossless
FP16 ~141 GB ~158 GB Reference quality

The file sizes come from the published GGUF builds, so they move by a gigabyte or two between uploaders. Check the file list on the model page you actually download. Our LLM quantization guide covers what each level costs in quality.

The KV cache for a 70B model

Llama 3.1 and 3.3 70B have 80 layers, 8 key-value heads and a head size of 128. At 16-bit cache precision that is 320 KB per token:

2 x 80 layers x 8 KV heads x 128 head size x 2 bytes = 327,680 bytes per token
Context length KV cache, one user
8K tokens ~2.5 GB
32K tokens ~10 GB
128K tokens ~40 GB

The cache is per conversation, so multiply by the number of simultaneous users. Many inference servers can store the cache at 8-bit instead, which roughly halves these figures at a small quality cost. The model's own config.json holds the layer and head counts if you want to check them.

Putting users and context together

For Q4_K_M weights (42.5 GB), 16-bit cache and 10% overhead:

Scenario Total memory
1 user, 8K context ~50 GB
4 users, 8K context ~58 GB
8 users, 8K context ~69 GB
16 users, 8K context ~91 GB
2 users, 32K context ~69 GB
1 user, 128K context ~91 GB

The last row is the one that surprises people. A 70B model that fits comfortably at 8K needs two cards once you serve its full 128K window, because the cache alone is as large as the 4-bit weights. If your documents are short, set a smaller maximum context and spend the memory on concurrency instead.

Which GPUs fit

  • One 80 GB card (A100 80GB, H100): Q4_K_M with up to about 8 users at 8K. Q5_K_M with a handful. Q8_0 does not fit.
  • One 48 GB card (L40S, RTX 6000 Ada, A6000): only Q3_K_M or lower, with one user and a short context. Not recommended for a shared service.
  • Two 48 GB cards (96 GB total): Q4_K_M with 16 users at 8K, or Q8_0 for a single user. The model is split across the cards with tensor parallelism, so the link between them matters: NVLink is much faster than PCIe, and PCIe-only pairs lose some throughput.
  • Four 24 GB cards (96 GB total): works for Q4_K_M, but each card holds a quarter of the model plus its own buffers, and the PCIe traffic between four cards costs speed.
  • Two 80 GB cards: Q8_0 with real concurrency, or FP16 for one user with very little to spare.

Splitting a model across GPUs never gives you the full sum of their memory. Each card needs room for its own shard, activations and buffers, so plan on 5 to 10% less than the total.

Choosing a quantization for 70B

Start with Q4_K_M. It is the default for good reason: it cuts memory by about 70% compared to FP16 and, on most tasks, the difference is hard to measure. Step up to Q6_K or Q8_0 only if you have GPU memory left over and an evaluation showing your task benefits. Step down to Q3_K_M only when the card you have forces it, and test on your own prompts first, because the loss shows up unevenly: long reasoning and code suffer before simple chat does.

Compare against a smaller model before paying for 70B at all. A 32B model at Q8_0 uses about 40 GB and often matches an older 70B on narrow tasks. The open-source LLM for enterprise post covers how to choose by task and license, and Qwen VRAM requirements has the same sizing for the Qwen family.

How this shows up in AI Refinery

AI Refinery, a self-hosted LLM platform, benchmarks every quantization variant before it reaches the catalog and lists the GPU memory it needs, counting the weights plus the KV cache at the model's full context window. That is why its figure for a 70B model can look larger than the 8K numbers above: it reserves the full window rather than assuming short prompts. The Professional plan covers models up to 70B parameters, so this class of model is within it; see plans for details. The customer provides the GPUs.

If you are still working out whether a 70B model is worth the hardware, the self-hosted LLM cost model shows where owning GPUs breaks even against API pricing. For the broader picture, start from the home page: AI Refinery's private model catalog.

Figures are estimates. Measure on your own hardware before buying GPUs.