Notes on the KV Cache

The KV cache is one of those things that people mention casually, as if it should be obvious. It took me a while to feel like I actually understood what it was and why it mattered. This is my best attempt at explaining it to myself.

When a transformer generates a new token, it needs to look back at everything that came before. To do that, each previous token has a set of key and value vectors that the attention mechanism uses. Without a cache, the model would have to recompute those vectors for the entire sequence at every step. That would be wildly wasteful because the earlier tokens do not change.

So the KV cache stores them. Every time the model generates a token, its key and value get added to the cache. On the next step the model only needs to compute new keys and values for the newest token, and it can reuse everything else. This is what makes long generations possible without the cost blowing up.

That was the part I understood pretty quickly. The part that took longer was realizing how much memory this thing actually uses.

The KV cache grows linearly with sequence length and linearly with the number of layers and heads. For a big model with a long context, the cache can be larger than the model weights themselves. That is wild. You are carrying around something bigger than the model just to answer one request. And you need one cache per active request, not one per model.

Once I saw that, a lot of things clicked. Why context length is expensive. Why running many parallel users is a memory problem, not just a compute problem. Why paged attention and other cache management schemes matter. The cache is basically a big allocation problem that lives on top of the model.

I have also been thinking about how much of the recent progress in serving efficiency is really progress in cache management. Sharing prefixes across users. Evicting parts of the cache that will not be needed. Storing the cache in compressed or lower-precision forms. None of these are new algorithms in the model. They are just smarter ways to handle memory.

I do not think I am done being surprised by the KV cache. Every time I think I have wrapped my head around it, someone shows me a new trick that reframes it. For now, my rough mental model is that the KV cache is the real state of an inference server, and the model is just the function that operates on it.