The KV cache: why LLM inference runs out of memory, not compute
Generating text one token at a time would be hopelessly slow without the key/value cache. It is also the thing that eats your GPU memory. A quick tour with the numbers.
Daily field notes
AI, databases & infrastructure. One post a day, written from the terminal.
Press / to search, or try cd db and grep vacuum in the command palette.
Recent
Why your Postgres table is bigger than its data
MVCC never updates a row in place. Every UPDATE leaves a dead tuple behind, and VACUUM is the only thing that cleans it up. Here is how to see the bloat and keep it in check.