Tutorial · Ollama · Local AI

Run a Local AI Model with Ollama

Move from an installed Ollama runtime to a repeatable model lifecycle: pull → run → inspect → stop → remove.

UPDATED SEP 25, 2026 · BEGINNER
PREREQUISITE
This tutorial assumes Ollama is installed and the ollama command works. If not, start with Install Ollama on Windows.

Before you start: choose for your hardware

A model name is not a hardware requirement. The active runtime also depends on quantization, context and the inference configuration. Before pulling a large model, use the hardware-aware model selection workflow and the Model Size & Memory Explorer.

E—DOCEO's measured Qwen3 4B experiment is a useful warning: a 2.5 GB downloaded model was reported by Ollama as a 3.5 GB active runtime at context 4096 on the test machine, with partial CPU/GPU placement. Download size alone did not describe the loaded configuration.

1. Pull a model

Use the model name you selected from the Ollama model library:

PowerShell
ollama pull MODEL_NAME

Then confirm it is installed:

PowerShell
ollama ls

Treat this as the first checkpoint: the model should appear in the local list before you diagnose inference problems.

2. Run local inference

Start an interactive session:

PowerShell
ollama run MODEL_NAME

Use a short prompt first. A small diagnostic prompt separates basic runtime health from long-context or workload-specific problems.

Diagnostic prompt
Explain in one sentence what local inference means.

A completed response confirms that the runtime can load the selected model and perform inference. It does not prove that the configuration is optimal for your hardware.

3. Inspect what Ollama actually loaded

While the model is active, open another terminal and run:

PowerShell
ollama ps

Record the model, active context, runtime size and processor placement shown by your version. These values are more useful for diagnosis than the downloaded file size alone.

DO NOT OVER-INTERPRET PLACEMENT
A CPU/GPU percentage reported by the runtime describes its placement decision. It is not automatically the same thing as percentage of elapsed time, token generation work or VRAM consumption. See Model Offloading.

4. Stop a loaded model

When you are finished with the active runtime:

PowerShell
ollama stop MODEL_NAME

Run ollama ps again to verify the runtime state rather than assuming the model unloaded immediately.

5. Remove a downloaded model

First list local models so you can verify the exact name:

PowerShell
ollama ls

Then remove the model you no longer need:

PowerShell
ollama rm MODEL_NAME

Removing a downloaded model and stopping a loaded model solve different problems: one manages persistent model storage; the other manages the active runtime.

Troubleshooting by layer

If something fails, identify the layer before changing settings:

  1. CLI: does ollama --help work?
  2. Storage: does ollama ls show the expected model?
  3. Runtime: does ollama run MODEL_NAME produce a response?
  4. Placement: what does ollama ps report?
  5. Integration: does the local API answer?

This sequence prevents an API configuration problem from being mistaken for a GPU problem, or a model-fit problem from being mistaken for a broken installation.

Next steps

Continue with Managing Local Models with Ollama for the lifecycle concepts, then use the Ollama API to call the same local runtime from PowerShell or your own application.

Sources