KV Cache Explained: Size, GQA, Quantization and Paging
With the KV cache explained in one sentence: it is the memory where a model keeps the attention keys and values of every token it has already seen, so it does not have to recompute them for each new token. It makes generation fast, it grows with every token of context and every concurrent request, and on a busy server it often uses more GPU memory than the model weights.
Its size per token is fixed by four numbers from the model's config.json:
KV cache per token (bytes) = 2 x layers x KV heads x head size x bytes per value
The rest of this post explains where that formula comes from, why two models of the same size can differ by 4x in cache, and the techniques inference engines use to fit more of it. Our GPU memory guide uses the same formula for sizing; this post is about the mechanics underneath.
KV cache explained: what it stores
In a transformer, each layer's attention step turns every token into three vectors: a query, a key and a value (Vaswani et al., 2017). To produce the next token, the newest token's query is compared against the keys of all earlier tokens, and the matching values are blended into the output.
The keys and values of earlier tokens never change. Without a cache, the model would recompute them for the whole sequence at every step, so generating 1,000 tokens would redo the work for token 1 a thousand times. With a cache, each token's keys and values are computed once and stored, per layer and per KV head.
That splits inference into two phases:
- Prefill processes the whole prompt at once and writes its keys and values into the cache. It is compute-heavy and sets your time to first token.
- Decode generates one token at a time. Each step reads the model weights and the entire KV cache for that request, then appends one more entry. It is limited by memory bandwidth, which is why a long conversation gets slightly slower per token as it goes.
How to calculate KV cache size
The formula has a 2 for keys plus values, then one entry per layer, per KV head, of head_dim numbers, each taking 2 bytes at 16-bit. All four inputs are in the model's config:
| Field in config.json | Meaning |
|---|---|
num_hidden_layers |
Layers |
num_key_value_heads |
KV heads |
head_dim (or hidden_size / num_attention_heads) |
Head size |
torch_dtype |
Usually bfloat16, so 2 bytes unless you quantize the cache |
Here are four real models, using the values in their published configs:
| Model | Layers | Query heads | KV heads | Head size | KV per token (16-bit) |
|---|---|---|---|---|---|
| Llama 2 7B | 32 | 32 | 32 | 128 | 512 KB |
| Mistral 7B v0.3 | 32 | 32 | 8 | 128 | 128 KB |
| Qwen3 32B | 64 | 64 | 8 | 128 | 256 KB |
| Gemma 3 27B | 62 | 32 | 16 | 128 | see below |
Working through Mistral 7B:
2 x 32 layers x 8 KV heads x 128 x 2 bytes = 131,072 bytes = 128 KB per token
128 KB x 32,768 tokens (its full context) = 4 GB per request
Multiply by concurrent requests. Ten users at full context need 40 GB of cache, against about 14.5 GB of 16-bit weights.
Why GQA and MQA shrink the cache
Llama 2 7B and Mistral 7B are almost the same size, yet Llama 2's cache is 4x larger per token. The difference is the KV head count.
Original multi-head attention gives every query head its own key and value head. Multi-query attention (MQA) keeps many query heads but shares a single key and value head among all of them (Shazeer, 2019), cutting the cache by the number of heads. It saves a lot of memory but can cost quality. Grouped-query attention (GQA) is the middle ground: query heads are split into groups, and each group shares one KV head (Ainslie et al., 2023). Mistral 7B has 32 query heads in 8 groups, so 8 KV heads and a quarter of the cache.
Most current open models use GQA, often with 8 KV heads. It is why models with much larger context windows than Llama 2 are practical at all.
Two other designs go further:
- Multi-head latent attention (MLA), used by DeepSeek V2, V3 and R1, compresses keys and values into a smaller latent vector before caching. DeepSeek reported a 93.3% smaller KV cache than its earlier 67B model (DeepSeek-AI, 2024), though that comparison also reflects other changes between the two models.
- Sliding-window layers, used by Gemma 3, limit some layers to attending over the most recent tokens only. Gemma 3 uses five local layers with a 1,024-token window for every global layer (Gemma Team, 2025). For Gemma 3 27B that is roughly 10 global layers whose cache grows with context and 52 local layers whose cache stops at 1,024 tokens. At a 128K context, our estimate is about 10 GB of cache instead of the roughly 60 GB the same model would need with every layer global. Whether your inference engine actually reclaims that memory depends on the engine and version, so check before relying on it.
Context length is the biggest lever
The cache grows linearly with context, so the maximum context you allow is the most direct control you have. Qwen3 32B at 256 KB per token:
| Context per request | Cache per request | Requests in 40 GB of free memory |
|---|---|---|
| 4K tokens | 1 GB | ~40 |
| 8K tokens | 2 GB | ~20 |
| 32K tokens | 8 GB | ~5 |
| 40K tokens (model default) | 10 GB | ~4 |
Most inference servers reserve memory for the maximum context you configure. If your prompts and answers stay under 8K tokens, serving 40K wastes capacity you could spend on more concurrent users. Set the limit from your logs, not from the model card. The Qwen VRAM requirements post has the full sizing for the Qwen family, and Llama 70B VRAM requirements shows the same trade-off at 70B.
KV cache quantization
The cache does not have to stay at 16-bit. Storing it at 8-bit halves it, which doubles the context or concurrency that fits.
- vLLM supports an FP8 KV cache with
--kv-cache-dtype fp8. Its quantized KV cache docs recommend calibrated scales for the best accuracy; without calibration the scales default to 1.0. - llama.cpp's server takes
--cache-type-kand--cache-type-v(for exampleq8_0) per its server README. The default is f16.
Research methods go lower. KIVI, for example, quantizes the cache to 2 bits with separate treatment for keys and values (Liu et al., 2024). For production, 8-bit is the safe step: test it on your own prompts, especially long-context retrieval and code, where small errors in attention show up first. Weight quantization is a separate choice with its own trade-offs, covered in the LLM quantization guide.
Paged attention: less waste, more users
Early serving systems reserved one contiguous block of memory per request, sized for the maximum possible length. A request that stopped after 300 tokens still held space for 8,000, and memory fragmented as requests came and went.
PagedAttention borrows the idea of virtual memory pages from operating systems (Kwon et al., 2023). The cache is split into small fixed-size blocks that are allocated as the sequence grows and can live anywhere in GPU memory. The paper reports near-zero KV cache waste and 2-4x higher throughput than the systems it compared against at the same latency. It is the core of vLLM, and similar block-based allocation is now standard across serious inference engines.
The practical effect: on a paged engine, the number of concurrent requests you can serve depends on the tokens actually in flight, not on worst-case reservations. Your capacity estimate from the table above is a floor, not a ceiling, when most requests are short.
Prefix caching: reuse instead of recompute
Many requests start the same way: a long system prompt, a fixed instruction block, few-shot examples, or the same document asked about repeatedly. Their keys and values for that shared prefix are identical, so there is no reason to compute and store them twice.
- vLLM's automatic prefix caching hashes full cache blocks and reuses matching ones across requests. It also supports a per-request
cache_saltso tenants in a shared deployment do not share cached blocks. - SGLang's RadixAttention keeps cached prefixes in a radix tree with least-recently-used eviction, reusing partial overlaps automatically (Zheng et al., 2023).
Prefix caching saves prefill time, which shows up as a faster time to first token on repeated prefixes, and saves memory when many in-flight requests share one prefix. To benefit, put the stable parts of your prompt first and the variable parts last. A timestamp at the top of a system prompt breaks the match for everything after it. The vLLM vs SGLang comparison covers how to benchmark the two on prefix-heavy workloads.
A checklist for KV cache sizing
- Read
num_hidden_layers,num_key_value_headsandhead_dimfrom the model's config and compute bytes per token. - Set the maximum context from your real prompt and answer lengths.
- Multiply by peak concurrent requests, not total users.
- Try an 8-bit KV cache if it does not fit, and evaluate quality on your own prompts.
- Order prompts so the shared part comes first, and enable prefix caching.
- Load test and watch cache usage under peak traffic before adding GPUs.
How AI Refinery reports it
AI Refinery, a self-hosted LLM platform, benchmarks every catalog model variant and lists the GPU memory it needs, counting the weights plus the KV cache at the model's full context window, alongside its measured tokens per second. Because it assumes the full window, a deployment serving shorter contexts needs less memory than that figure, exactly as the formula above predicts. You can see how the catalog and plans fit together on the private model catalog page.
Model configs change between releases. Read the config.json of the exact model version you deploy.