# KV Cache VRAM Calculator: How Much Memory Does Context Length Need? (2026)

**The KV cache is the memory a transformer uses to remember every token in the context, and its size grows linearly with context length:** bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For Qwen3 32B at FP16 that is 256 KiB per token, or about 8 GB for a 32k-token context, on top of the model weights. This is why a model that loads fine can still run out of memory when you raise the context.

This post gives the formula, the per-token numbers for popular models (read from their `config.json`), and how much context fits on 24, 32, 48 and 80 GB cards.

## The formula

```text
kv_bytes_per_token = 2 × num_hidden_layers × num_key_value_heads × head_dim × bytes_per_value
kv_total_bytes     = kv_bytes_per_token × context_tokens × concurrent_sequences
```

- The **2** is for keys and values.
- `num_key_value_heads` is smaller than the number of attention heads in models with grouped-query attention (GQA), which is why modern models have much smaller caches than older ones.
- `bytes_per_value` is 2 for FP16/BF16, 1 for 8-bit KV cache, about 0.5 for 4-bit.
- **Concurrent sequences multiply the cache.** If your server keeps four parallel slots, budget four times the context.

## Per-token KV cache for popular models (FP16)

Architecture values are from each model’s `config.json` on Hugging Face (read 2026-10-23). GB here means GiB (1024³ bytes).

| Model | Layers | KV heads | head_dim | KV per token | 8k ctx | 32k ctx | 128k ctx |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.1 8B | 32 | 8 | 128 | 128 KiB | 1.0 GB | 4.0 GB | 16 GB |
| Qwen3 8B | 36 | 8 | 128 | 144 KiB | 1.1 GB | 4.5 GB | 18 GB\* |
| Qwen3 14B | 40 | 8 | 128 | 160 KiB | 1.25 GB | 5.0 GB | 20 GB\* |
| Mistral Small 3 24B | 40 | 8 | 128 | 160 KiB | 1.25 GB | 5.0 GB | — (32k max) |
| Qwen3 32B | 64 | 8 | 128 | 256 KiB | 2.0 GB | 8.0 GB | 32 GB\* |
| Llama 3.3 70B | 80 | 8 | 128 | 320 KiB | 2.5 GB | 10 GB | 40 GB |
| Qwen2.5 72B | 80 | 8 | 128 | 320 KiB | 2.5 GB | 10 GB | 40 GB\* |

\* Beyond the model’s native window. Qwen3 dense models are documented with a native 32,768-token context, extendable with YaRN rope scaling; check the model card before going past the native length.

The formula does not apply as-is to models with sliding-window layers (for example Gemma 3) or latent attention (DeepSeek V3-style MLA); their caches are smaller than this formula suggests.

## How much context fits on each card

Total memory ≈ weights + KV cache + overhead. Weights are the Q4_K_M sizes on our [/run](https://cloudgpu.app/run) pages; overhead is an **assumed 1.5 GB** for CUDA context, buffers and the runtime. The result is a ceiling for a single sequence; runtimes reserve extra buffers, so plan below it.

**Qwen3 32B at Q4_K_M (≈ 19.5 GB weights, 256 KiB/token):**

| Card class | Left for KV after weights + 1.5 GB | Context ceiling (1 sequence) | Practical guidance |
| --- | --- | --- | --- |
| 24 GB (RTX 4090 / 3090) | ≈ 3 GB | ≈ 12k tokens | about 8k; tight |
| 32 GB (RTX 5090) | ≈ 11 GB | ≈ 45k (native max 32k) | 16k–32k comfortable |
| 48 GB (RTX 4090 48G) | ≈ 27 GB | ≈ 110k | full native 32k, or several parallel slots |

**Llama 3.3 70B at Q4_K_M (≈ 41.5 GB weights, 320 KiB/token):**

| Card class | Left for KV | Context ceiling (1 sequence) | Practical guidance |
| --- | --- | --- | --- |
| 48 GB (RTX 4090 48G / A6000) | ≈ 5 GB | ≈ 16k | about 8k; tight |
| 80 GB (A100 / A800) | ≈ 37 GB | ≈ 121k | 32k comfortable, with room for parallel requests |

The “practical guidance” column matches the per-model notes on our [Qwen3 32B page](https://cloudgpu.app/run/qwen3-32b) and [Llama 3.3 70B page](https://cloudgpu.app/run/llama-3-3-70b).

## A small calculator

```python
def kv_gb(layers, kv_heads, head_dim, ctx, bytes_per_value=2, seqs=1):
    return 2 * layers * kv_heads * head_dim * bytes_per_value * ctx * seqs / 1024**3

def max_ctx(vram_gb, weights_gb, layers, kv_heads, head_dim,
            bytes_per_value=2, overhead_gb=1.5, seqs=1):
    free = vram_gb - weights_gb - overhead_gb
    per_tok = 2 * layers * kv_heads * head_dim * bytes_per_value / 1024**3
    return max(0, int(free / (per_tok * seqs)))

# Qwen3 32B (64 layers, 8 KV heads, head_dim 128), Q4 weights ≈ 19.5 GB
print(kv_gb(64, 8, 128, 32768))          # 8.0
print(max_ctx(32, 19.5, 64, 8, 128))     # ≈ 45k (native limit is 32k)
print(max_ctx(24, 19.5, 64, 8, 128, bytes_per_value=1))  # 8-bit KV: ≈ 24k
```

Read `num_hidden_layers`, `num_key_value_heads` and `head_dim` (or `hidden_size / num_attention_heads`) from the model’s `config.json`.

## Three ways to fit more context

**1\. Quantise the KV cache.** Halving bytes per value halves the cache.

```bash
# Ollama: 8-bit KV cache (requires flash attention)
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve
```

```bash
# vLLM: FP8 KV cache
vllm serve Qwen/Qwen3-32B-AWQ --kv-cache-dtype fp8 --max-model-len 32768
```

Quality impact is usually small at 8-bit and more noticeable at 4-bit; test on your own prompts.

**2\. Set the context you need, not the maximum.** In Ollama, pass `"options": {"num_ctx": 16384}` per request (or set `OLLAMA_CONTEXT_LENGTH`). In vLLM, `--max-model-len` caps the context and `--gpu-memory-utilization` sets how much VRAM the cache may take.

**3\. Watch parallelism.** Ollama’s `OLLAMA_NUM_PARALLEL` and vLLM’s batching both allocate cache per sequence. If memory runs out under load but not in single tests, this is usually why.

## Troubleshooting order when you run out of memory

Say you run Qwen3 32B (Q4_K_M) with Ollama on a 24 GB RTX 4090, set `num_ctx` to 32768, and get out-of-memory errors or a sudden slowdown (layers spilling to CPU). The formula explains it: about 19.5 GB of weights plus about 8 GB of FP16 KV cache is more than 24 GB. Try these, least effort first:

1. **Lower the context.** Back to 8192 and the KV cache is about 2 GB; total about 23 GB — it runs, but tight.
2. **Quantise the KV cache.** With `OLLAMA_FLASH_ATTENTION=1` and `OLLAMA_KV_CACHE_TYPE=q8_0`, the same memory holds roughly twice the context — about 24k by the formula (a ceiling; leave headroom).
3. **Check parallelism.** Set `OLLAMA_NUM_PARALLEL=1` to rule out several slots sharing the memory.
4. **Change cards.** Still short? Move to a 32 GB RTX 5090 or a 48 GB RTX 4090 48G. Switching means starting a new instance and downloading the model again.

After each change, check real usage with `nvidia-smi` and note the number; next time you size a card, you will have your own data.

## Picking a card from the numbers

Once you have weights + KV + overhead, round up to the next VRAM class and check [Deploy](https://cloudgpu.app/deploy) for live stock. GPUs are billed per minute; current rates are on the [pricing page](https://cloudgpu.app/pricing). For a side-by-side of those cards, see [RTX 4090 vs RTX 5090 vs A100](https://cloudgpu.app/compare/rtx-4090-vs-5090-vs-a100).

## FAQ

**How do I calculate the KV cache size for a model?**

Multiply 2 × layers × KV heads × head dimension × bytes per value (2 for FP16) to get bytes per token, then multiply by context length and the number of parallel sequences. The values are in the model’s config.json.

**How much VRAM does a 32k context need for Qwen3 32B?**

About 8 GB of FP16 KV cache (256 KiB per token × 32,768 tokens), on top of about 19.5 GB of Q4_K_M weights, so a 32 GB card is the comfortable choice.

**Why does my model run out of memory only at long context?**

The weights are fixed, but the KV cache grows with every token of context and with each parallel request. Lower num_ctx / max-model-len, reduce parallel slots, or quantise the KV cache.

**Does an 8-bit KV cache halve memory?**

Yes, the cache itself is halved compared with FP16. In Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention on; in vLLM use --kv-cache-dtype fp8.

**Does the formula work for every model?**

It works for standard and grouped-query attention. Models with sliding-window layers or latent attention use less memory than it predicts.

---

Once you have your own numbers, the [RTX 4090 vs RTX 5090 vs A100 comparison](https://cloudgpu.app/compare/rtx-4090-vs-5090-vs-a100) lines up the cards by VRAM class. *Originally published at* [*cloudgpu.app*](https://cloudgpu.app/blog/kv-cache-vram-context-length-calculator-2026)*.*

> CloudGPU (cloudgpu.app) has no partnership with, authorization from, or endorsement by the model vendor. Model names are trademarks of their owners. Context ceilings are estimates under the stated assumptions, not measurements. Live prices are on [/pricing](https://cloudgpu.app/pricing) and live stock on [/deploy](https://cloudgpu.app/deploy).