LLM Quantization Explained: Speed, Memory and Quality

5 min read

LLM quantization stores a model's weights with fewer bits per number, for example 4 or 8 bits instead of 16. The model gets two to four times smaller, usually runs faster, and loses a little output quality. How much quality depends mostly on how far you go: 8-bit is very close to the original, 4-bit is a good trade for most uses, and below that the losses grow quickly.

Quantization is the reason a 70-billion-parameter model can run on one or two GPUs instead of a rack of them. This post explains what is happening, what the names like Q4_K_M mean, and how to pick a level for your own workload.

What LLM quantization does

A language model is mostly a very large list of numbers, its weights. A 7B model has about seven billion of them. Trained models usually store each one as a 16-bit floating-point value, which is 2 bytes. Seven billion times 2 bytes is about 14 GB, before the model has processed a single word.

Quantization replaces those 16-bit values with smaller ones. Weights are grouped into small blocks, and each block stores a shared scale plus a low-precision number for each weight. When the model runs, the values are expanded back on the fly. You lose some precision in every weight, but the overall behavior of the model mostly survives, because no single weight matters much on its own.

Precision Approx. bytes per weight 7B model 70B model
FP16 / BF16 2 ~14 GB ~140 GB
8-bit ~1.06 ~7.5 GB ~74 GB
6-bit ~0.82 ~5.8 GB ~58 GB
5-bit ~0.71 ~5 GB ~50 GB
4-bit ~0.6 ~4.2 GB ~42 GB

These are file sizes for the weights. Running the model takes more memory on top, mainly for the KV cache, which our post on GPU memory for LLM inference covers.

Reading quantization names like Q4_K_M

If you download models in the GGUF format used by llama.cpp and the tools built on it (see vLLM vs Ollama for how the main inference servers handle formats), you will see names like these:

  • F16 / BF16: not quantized. The reference version.
  • Q8_0: 8-bit. Hard to tell apart from F16 in normal use.
  • Q6_K, Q5_K_M: 6-bit and 5-bit. Small, usually unnoticeable losses.
  • Q4_K_M: 4-bit. The most common default and a good balance of size and quality.
  • Q3_K_M, Q2_K: 3-bit and 2-bit. Noticeably worse, mainly for squeezing a model onto hardware that is too small.

The number is roughly the bits per weight. K marks the newer "k-quant" scheme, which spends a few extra bits on the parts of the model that are most sensitive. The trailing S, M or L means small, medium or large: how many of those sensitive parts get the extra bits.

Other formats use other names. GPTQ and AWQ are 4-bit methods common on GPU servers, and FP8 is an 8-bit floating-point format supported natively by recent NVIDIA data center GPUs. The trade-offs are similar: fewer bits, less memory, some quality cost. GGUF vs AWQ vs GPTQ compares the formats and which engines load each one.

How quantization affects speed

Generating text happens one token at a time, and for each token the GPU has to read essentially all of the model's weights from memory. On most hardware that memory read, not the arithmetic, is the bottleneck. Halve the size of the weights and you roughly halve the data moved per token, so token generation gets faster.

Reading the prompt (the "prefill" phase) works differently. The model processes many prompt tokens at once, which makes it limited more by raw compute than by memory, so quantization helps it less and can even slow it slightly. If your workload is long documents with short answers, expect a smaller speed gain than with short prompts and long answers.

The biggest speed effect of all is indirect. A quantized model that fits entirely in GPU memory is many times faster than a larger version that spills layers into system RAM and runs partly on the CPU.

How quantization affects quality

Researchers usually measure quality loss with perplexity, a score of how well the model predicts real text, where lower is better. Across many models the pattern is consistent:

  • 8-bit is within measurement noise of the original.
  • 6-bit and 5-bit lose very little.
  • 4-bit (k-quant or AWQ) loses a small but measurable amount, usually acceptable.
  • 3-bit and below fall off quickly.

Two things move those lines. Small models suffer more. A 70B model at 4-bit usually beats a 14B model at 8-bit, because the larger model has more redundancy to absorb the lost precision. And some tasks suffer more: exact arithmetic, long chains of reasoning, code and strict output formats show damage sooner than summarizing or classifying text.

Perplexity is also an average. A model can score almost the same and still fail differently on your particular task. That is why the only test that counts is your own. Evaluating open source LLMs on your own data walks through building that test.

How to choose a quantization level

  1. Start from your memory budget. Work out how much GPU memory you have for weights after leaving room for the KV cache.
  2. Pick the largest model that fits at 4-bit or better. Model size usually matters more than the last few bits of precision. Choosing an open source LLM for enterprise covers which families to shortlist.
  3. Use 8-bit if it fits comfortably. There is no reason to give up quality you can afford.
  4. Avoid 3-bit and 2-bit unless the alternative is not running the model at all.
  5. Test on your own examples. Run 30 to 50 real inputs through two neighboring levels, for example Q4_K_M and Q8_0, and compare the outputs side by side. If people cannot tell them apart, take the smaller one.

Comparing quantizations in AI Refinery

The model catalog in AI Refinery, a self-hosted LLM platform, lists the quantization variants of each model, and every variant is benchmarked before it reaches the catalog. For each one you can see the size on disk, the GPU memory it needs and its measured tokens per second, so you can compare levels side by side before you download anything or buy hardware. You then pick the level when you deploy, and the inference server you launch serves it behind an OpenAI-compatible endpoint.