Context Length in Local LLMs
A model's maximum context is a capability limit, not a target you must allocate for every run. On local hardware, context length is also a memory and performance decision.
What context length means
Context length is the amount of token history a model can have available for a request. That history can include the system prompt, your messages, previous assistant messages, retrieved documents, tool output and generated text.
It is measured in tokens, not characters or words. The mapping between text and tokens depends on the tokenizer and language, so a context value such as 4096 should not be translated into one fixed word count.
Long context is useful when a task genuinely needs more material at once: working with a long document, a large codebase, a lengthy conversation or retrieved reference material. It is not automatically better for every prompt.
Model maximum is not the same as active context
A model may advertise a very large maximum context window. That describes a supported limit under appropriate conditions. Your runtime can allocate a smaller context for an actual session.
This distinction matters on local hardware. In our Qwen3 4B test, ollama show qwen3:4b reported a model context length of 262144, while the tested Ollama sessions used active contexts of 4096, 2048 and 1024.
Model context limit = what the model supports. Active context = what the runtime allocates for the current configuration. Do not treat them as the same number.
Why longer context uses more memory
Autoregressive LLM inference keeps information from previous tokens so it does not have to recompute the entire sequence for every new token. A major part of this state is commonly called the KV cache (key-value cache).
As the allocated context grows, the runtime needs more memory for context-related state. The exact amount depends on the model architecture, cache data types, runtime and configuration, so there is no universal rule such as “every 1,000 tokens costs X MB.”
This is visible in runtime controls. llama.cpp, for example, exposes KV-cache data types and whether the KV cache is offloaded, while Ollama explicitly documents that increasing context length increases the memory required to run a model.
Longer context can therefore change more than capacity. On a memory-constrained GPU it can affect how much of the runtime remains GPU-resident and may change generation speed.
How much context should you allocate?
Start with the task rather than the model's headline maximum.
| Workload | Practical approach |
|---|---|
| Short questions and simple chat | Start small; increase only if history is being truncated |
| Coding with several files | Allocate enough for the relevant code and instructions |
| RAG / document work | Budget for retrieved chunks plus prompt and output |
| Agents with large tool traces | Expect a substantially larger context requirement |
| Very long documents | Test memory and quality before allocating the model maximum |
A smaller context is not a quality setting by itself. If the complete information needed for the task fits, allocating unused context can simply consume memory that could otherwise help model placement.
Measured example on 3 GB VRAM
Our Lab #002 used the same Qwen3 4B Q4_K_M model and changed the active context on a GTX 1060 3GB. Ollama reported:
| Active context | Runtime size | Processor placement | Median generation |
|---|---|---|---|
| 4096 | 3.5 GB | 57% CPU / 43% GPU | 12.16 tok/s |
| 2048 | 3.2 GB | 55% CPU / 45% GPU | 13.25 tok/s |
| 1024 | 3.1 GB | 53% CPU / 47% GPU | 13.49 tok/s |
This is one measured system, not a universal scaling formula. But it demonstrates the mechanism that matters for local planning: changing active context changed the reported runtime size and CPU/GPU placement, while the model itself stayed the same.
See the complete measurements and methodology →
Check active context instead of guessing
With Ollama, inspect the loaded model:
ollama ps
The CONTEXT column shows the allocated context and PROCESSOR shows the CPU/GPU placement. This is more useful for diagnosing a real local configuration than assuming that the model's advertised maximum is currently allocated.
If a model is unexpectedly offloading to CPU on a tight GPU, context length is one of the variables worth testing — alongside model size, quantization, other GPU memory use and runtime settings.