CPU vs GPU for Local LLMs
A local LLM does not have to be either 'CPU-only' or 'GPU-only.' Modern runtimes can place different parts of a model on different devices, which makes memory capacity as important as raw compute.
Running an LLM on the CPU
Local inference can run without a discrete GPU. CPU execution is useful when the machine has enough system RAM but little or no compatible VRAM, when a model is too large for the GPU, or when maximum generation speed is not the priority.
The practical advantage is capacity: desktop systems often have much more RAM than VRAM. The trade-off is that token generation for modern LLM workloads is often slower than a well-supported GPU configuration.
CPU performance also varies widely. Memory bandwidth, instruction-set support, core architecture, model quantization and runtime implementation all matter. “Runs on CPU” therefore says very little about the actual tokens per second you will see.
Why GPUs can accelerate local inference
Transformer inference repeatedly performs large tensor operations. GPUs are designed for highly parallel numerical work and offer high memory bandwidth, which can make them effective for these workloads when the runtime has a mature backend for the hardware.
But compute capability alone is not enough. The model weights and runtime state need memory. A fast GPU with insufficient VRAM may be unable to keep the full model on-device.
For local LLMs, ask two separate questions: Can this backend use my GPU? and Can the configuration fit in its VRAM?
Partial GPU offload: the middle ground
CPU and GPU are not necessarily an either/or choice. Runtimes can place part of a model on the GPU and leave the rest in system memory for CPU-side execution.
llama.cpp exposes controls for the number of layers stored in VRAM and supports model splitting strategies. Ollama surfaces the resulting placement in the PROCESSOR column of ollama ps.
Partial offload is valuable because it can make a model usable when it does not fully fit in VRAM. It should not, however, be described as equivalent to full GPU residency. Data movement and CPU-side work can materially change performance.
Why VRAM and RAM both matter
A fully GPU-resident configuration is constrained primarily by available VRAM. A CPU-only or partially offloaded configuration also needs sufficient system RAM.
The model file size is only a starting point. Runtime memory can include model weights, context-related state such as the KV cache, temporary buffers and backend overhead. Quantization reduces weight memory, while context length can increase runtime memory.
That is why hardware selection should be made from the complete configuration — model, quantization, context, runtime and backend — rather than parameter count alone.
What should you measure?
For a useful local benchmark, record more than “it works”:
- exact model and quantization;
- active context length;
- runtime and version;
- CPU/GPU placement;
- generation tokens per second;
- whether the first run includes model-loading time;
- available RAM/VRAM and other workloads using the GPU.
Generation speed and total request time answer different questions. A cold request can spend seconds loading a model before generation begins, while a warm request may reuse an already loaded runtime.
Two measured configurations on the same GTX 1060 3GB
Our first two Lab tests show why placement matters.
Gemma 3 1B Q4_K_M, active context 4096, was reported by Ollama as 100% GPU on the test machine and produced a median generation rate of 66.99 tok/s across three runs.
Qwen3 4B Q4_K_M, also at active context 4096, was reported as 57% CPU / 43% GPU and produced a median generation rate of 12.16 tok/s across three runs.
These results should not be read as a model-quality comparison or as a universal CPU/GPU multiplier: the models differ substantially in size and output behavior. They are useful as concrete examples of two placement states observed on the same constrained GPU.
Lab #001 — Gemma 3 1B on GTX 1060 3GB →
Lab #002 — Qwen3 4B context and offload test →