What Is the KV Cache?
The KV cache stores reusable attention state from tokens the model has already processed. It can make decoding much more efficient, but it also becomes part of the runtime memory budget.
Why a KV cache exists
Autoregressive language models generate output one token at a time. During decoding, each new token attends to tokens that came before it. Recomputing all of the earlier attention state from scratch at every step would repeat a large amount of work.
The KV cache keeps reusable attention information from already processed tokens. The runtime can then reuse that state while it computes the next token. This is one reason the prefill phase and the token-by-token decoding phase have different computational behavior.
What the K and V mean
In transformer attention, tokens are projected into query, key and value representations. The cache retains the keys and values needed from previous positions. A new decoding step creates the current token's state and can attend against the cached prior state instead of rebuilding it all.
The cache is runtime state. It is not the model's learned knowledge, and it is not a replacement for the context window. The context window describes the token budget available to a run; the KV cache is one implementation mechanism used to make attention over active tokens practical.
Why longer context can increase memory
More cached token positions generally mean more stored key/value state. The exact memory relationship depends on the model architecture, cache precision, runtime and parallel workload. This is why changing context settings can change runtime memory even though the model weights themselves have not changed.
A useful conventional estimate
For a conventional transformer cache, a planning formula is:
KV bytes ≈ 2 × layers × KV heads × head dimension × context tokens × bytes per value × parallel sequencesThe factor of two represents keys plus values. This formula is useful when the architecture exposes the required dimensions and uses a conventional cache layout. It is an estimate, not a universal law for every modern architecture.
Where the simple formula stops being exact
Grouped-query attention changes the number of KV heads relative to query heads. Some runtimes use lower-precision caches. Sliding-window attention can limit which positions remain active. Architectures using mechanisms such as MLA can organize attention state differently. Parallel sequences also multiply the state that must be retained.
For those reasons, a generic calculator should keep its assumptions visible rather than presenting a single number as a guaranteed VRAM requirement.
The context window is a token budget. The KV cache is runtime attention state. Model weights are learned parameters. All three affect planning, but they describe different things.
Use the estimate, then measure the runtime
Use the Model Size & Memory Explorer when you want to compare weight precision and a conventional KV estimate. Use the VRAM Calculator when you already know a specific model artifact and want a hardware-oriented planning budget.
Then verify the real workload. E—DOCEO Lab #003 is a useful caution: Gemma 3 1B stayed at 100% GPU placement from 1K through 32K configured context in that experiment, but the prompt itself was short. That result tests context allocation for a short generation task; it does not prove the same memory or throughput behavior for a fully populated 32K prompt.