Does Context Length Slow Gemma 3 1B on a 3GB GPU?

Result

On this test machine, Gemma 3 1B Q4_K_M remained at 100% GPU placement according to ollama ps at every tested active context from 1,024 through 32,768 tokens. Generation throughput also stayed tightly grouped: the median was 64.85 tok/s at 1,024 and 64.55 tok/s at 32,768, a difference of about 0.46%.

The important qualification is that the benchmark prompt itself was short. This experiment measures the effect of allocating different context sizes for the same short generation task; it does not measure attention over prompts containing tens of thousands of populated tokens.

Test environment

CUDA Version: 12.9 is the compatibility version reported by the NVIDIA driver. This test did not establish that a separate CUDA Toolkit was installed.

Method

We tested six num_ctx values: 1024, 2048, 4096, 8192, 16384 and 32768. Before each configuration in the main sequence, the model was stopped and allowed to unload. Each configuration then received three requests through Ollama's local POST /api/generate endpoint with the same prompt, stream: false and temperature: 0.

Explain in about 150 words what a local large language model is and give three practical reasons for running one locally.

Generation throughput was calculated from Ollama's returned values:

generation tok/s = eval_count / (eval_duration / 1e9)

The first request at each context also served as the post-load observation point for ollama ps and nvidia-smi. Runs 2 and 3 were warm requests with load times around 2–3 ms. The separate 32K extension followed the same request configuration and recorded three runs.

Results across 1K–32K context

*Total GPU memory reported by nvidia-smi under Windows/WDDM, not isolated model VRAM.

The main sequence began at 563 MiB / 3072 MiB aggregate GPU memory before loading a model and returned to 552 MiB after the final unload. The separate 32K test began at 554 MiB. At 32K, the first post-run snapshot was 1604 MiB / 3072 MiB, an observed aggregate increase of about 1050 MiB from that run's baseline.

The first request at each setting included model-load overhead. For example, the 32K first request reported a 2.859 s load and 5.967 s total request time, while runs 2 and 3 reported approximately 0.002 s load and totals of 3.016 s and 3.012 s. For that reason, generation tok/s — not total request time — is the primary comparison here.

Memory and processor placement

The most useful placement result is simple: Ollama reported 100% GPU at all six context settings, including the model's reported maximum of 32768. There was no observed transition to partial CPU/GPU offload on this machine.

Memory reporting was less simple. Aggregate nvidia-smi use generally rose at larger contexts, but not monotonically. ollama ps runtime SIZE was also non-monotonic: it rose from 874–877 MB at 1K–4K to 952 MB at 8K, then showed 889 MB at 16K and 906 MB at 32K.

This is why the VRAM Calculator is deliberately presented as a planning tool rather than a promise of exact runtime residency.

What this test tells us

For this small, fully GPU-resident model and short prompt, allocating a larger context window did not materially reduce generation throughput. The median changed from 64.85 tok/s at 1K to 64.55 tok/s at 32K while placement remained 100% GPU.

That result should not be generalized to larger models. In Lab #002, Qwen3 4B on the same 3 GB GPU was only partially GPU-resident, and changing context from 4096 to 1024 changed both reported runtime size and CPU/GPU placement. Model size, architecture and available memory determine whether context allocation pushes a runtime across a placement boundary.

It also does not mean that processing a genuinely 32K-token prompt costs the same as processing a short prompt. This test held prompt content constant. A separate long-prompt benchmark would be needed to measure prompt-processing cost as populated sequence length grows.

Reproduce one configuration

$body = @{
    model  = "gemma3:1b"
    prompt = "Explain in about 150 words what a local large language model is and give three practical reasons for running one locally."
    stream = $false
    options = @{
        temperature = 0
        num_ctx = 32768
    }
} | ConvertTo-Json -Depth 5
$r = Invoke-RestMethod     -Method Post
-Uri http://localhost:11434/api/generate     -ContentType "application/json"
-Body $body
ollama ps
nvidia-smi
[math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2)

To reproduce the series, stop the model between context configurations and test 1024, 2048, 4096, 8192, 16384 and 32768. Record the raw API response and placement instead of relying only on a stopwatch.

Limitations

This is a single-machine, single-day experiment using Ollama 0.34.2, Gemma 3 1B Q4_K_M and Windows/WDDM. Three generation runs per context provide a practical comparison, not a statistically exhaustive benchmark.

The prompt contained only 35 prompt tokens, so the experiment isolates the effect of changing the allocated context setting much more than the computational cost of processing a long populated context. Output length also varied slightly between the first and subsequent runs (195 versus 192 tokens), despite temperature: 0.

GPU memory snapshots are aggregate system values. Other desktop applications were present, and WDDM does not give these snapshots the meaning of isolated per-model VRAM consumption. ollama ps SIZE is recorded as an observed runtime field and is not treated as a direct measure of KV-cache allocation.

Documentation

• Ollama — Generate a response
• Ollama — Context length