The KV cache: why LLM inference runs out of memory, not compute
Generating text one token at a time would be hopelessly slow without the key/value cache. It is also the thing that eats your GPU memory. A quick tour with the numbers.
Tag
Tag
Generating text one token at a time would be hopelessly slow without the key/value cache. It is also the thing that eats your GPU memory. A quick tour with the numbers.