AI · 3 min read

The KV cache: why LLM inference runs out of memory, not compute

Generating text one token at a time would be hopelessly slow without the key/value cache. It is also the thing that eats your GPU memory. A quick tour with the numbers.

On this page

When you serve a language model, the weights are a fixed cost. What actually decides how many users fit on one GPU, and how long a context you can afford, is the KV cache.

Decoding one token at a time

A decoder-only transformer generates autoregressively: each step produces one token, appends it, and runs again.

  • Attention for the new token needs keys and values for every previous token
  • Recomputing them each step makes generation quadratic in length
  • Instead, we compute each token’s K and V once and keep them: the KV cache

So the first step (the prefill) processes the whole prompt in parallel and is compute-bound. Every step after that (the decode) does a tiny amount of maths but reads the whole cache from memory.

How big is the cache?

Per sequence, in bytes:

def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, bytes_per=2):
    return 2 * layers * kv_heads * head_dim * seq_len * bytes_per

# Llama-3-8B style: 32 layers, 8 KV heads (GQA), head_dim 128, fp16
gb = kv_cache_bytes(32, 8, 128, 32_768) / 1e9
print(f"{gb:.1f} GB per 32k-token sequence")   # ~4.3 GB

The weights of that model are about 16 GB in fp16. Four users at full 32k context need as much memory for the cache as for the whole model.

Tricks that shrink it

  • Grouped-query attention: many query heads share a few KV heads (8 instead of 32 is a 4× cut)
  • Quantised cache: store K and V in 8-bit or 4-bit instead of fp16
  • Paged attention: allocate the cache in fixed blocks, like virtual memory, so nothing is wasted on padding
  • Prefix caching: reuse the cache for a shared system prompt across requests

Paged attention is the reason modern inference servers fit far more concurrent requests than naive implementations: fragmentation used to waste a large share of the cache memory.

What it means when you deploy

Decode speed is bounded by memory bandwidth: every token reads the weights plus the live cache. That is why a GPU with huge compute but modest bandwidth disappoints on chat workloads.

  • Set the max context you actually need; it caps concurrency
  • Batch requests: weights are read once per step for the whole batch
  • Turn on prefix caching when prompts share a long preamble

Takeaways

  • The KV cache trades memory for speed; without it decoding is quadratic
  • Its size grows linearly with context length and with concurrent users
  • GQA, quantisation and paging are how servers fit more requests per GPU

Keep reading