How Much VRAM Do You Need for Local LLMs?
VRAM planning starts with model weights, but it does not end there. Context and runtime overhead make a single model-size table an approximation, not a guarantee.
Model weights are the starting point
A simple mental model is:
weight memory ≈ parameter count × average bits per weight ÷ 8
For example, a hypothetical 8-billion-parameter model stored at an idealized 4 bits per parameter would contain roughly 4 GB of raw weight data. But that is not a promise that the complete inference process needs only 4 GB of VRAM.
Real quantization formats store block metadata and may use mixed representations. The runtime also needs memory for inference state, and context-related memory grows with the configuration.
Why “4-bit = exactly 4 bits per parameter” is too simple
llama.cpp exposes many quantization schemes rather than one universal “4-bit” format. Its tensor-encoding documentation distinguishes formats such as Q4_0 and K-quant variants, and the quantization tool describes different resulting sizes and quality trade-offs.
That means the formula above is useful for order-of-magnitude planning, while the actual GGUF file and runtime measurements are better inputs for a final decision.
Use arithmetic to shortlist models. Use the actual model file plus runtime measurements to decide whether a configuration truly fits.
Context length also consumes memory
The context window is the amount of token history available to the model during a request. Increasing it can increase runtime memory requirements. Ollama's documentation explicitly warns that context length affects memory usage and provides configuration controls for it.
This is one reason two users running the “same model” can report different VRAM usage: they may be using different context settings, runtime versions, backends or offloading strategies.
What if the whole model does not fit in VRAM?
Some local runtimes can split work between GPU and CPU/system memory. This can make models usable on hardware whose VRAM is smaller than the complete GPU-side requirement.
But “it runs” and “it runs entirely on the GPU” are different statements. Partial offload can change performance substantially. When comparing hardware, record whether the model is fully GPU-resident, partially offloaded or CPU-only.
Measured example: a 2.5 GB model on a 3 GB GPU
E—DOCEO Lab #002 gives a concrete example of why file size is not enough. On our Windows test system, ollama ls reported Qwen3 4B at 2.5 GB, while ollama ps reported a 3.5 GB runtime at active context 4096. On a GTX 1060 with 3072 MiB of VRAM, Ollama placed the runtime at 57% CPU / 43% GPU rather than fully on the GPU.
Reducing active context changed the reported runtime and placement on the same machine:
| Active context | Runtime size | GPU placement | Median generation |
|---|---|---|---|
| 4096 | 3.5 GB | 43% GPU | 12.16 tok/s |
| 2048 | 3.2 GB | 45% GPU | 13.25 tok/s |
| 1024 | 3.1 GB | 47% GPU | 13.49 tok/s |
These are measurements from one documented system, not universal sizing rules. They do demonstrate the practical point: context configuration and runtime overhead can change whether, and how much of, a model is GPU-resident.
Open the complete Qwen3 4B / GTX 1060 3GB test →
How the E—DOCEO VRAM calculator will work
Our calculator should not return a magic number from only “32B + Q4.” It will separate:
- estimated weight memory;
- quantization/format assumptions;
- context configuration;
- runtime overhead allowance;
- full-GPU vs partial-offload scenarios.
We will validate the estimator against documented model files and reproducible runtime measurements before enabling the result field.