How AI Model Inference Works
Inference is the process of using a trained model to produce outputs. It is different from training — and its speed and memory use depend on more than model size alone.
Inference uses a model that has already been trained
Inference is the process of running input through a trained model to produce an output. Training changes learned parameters. Inference normally uses those parameters without retraining them for every prompt.
For a language model, the visible result is usually a sequence of generated tokens. The application around the model may add system instructions, conversation history, retrieved documents or tool results before inference begins.
Training learns or updates model parameters. Inference uses the trained parameters to compute outputs for new input.
A simplified prompt-to-token pipeline
A practical mental model is:
- Text is converted into tokens by the model's tokenizer.
- Tokens are represented numerically and processed through the model's layers.
- The model produces scores for possible next tokens.
- A decoding strategy selects the next token from those scores.
- The selected token becomes part of the sequence and the process continues until a stopping condition is reached.
This is deliberately simplified. Architectures and runtimes differ, but it separates the model computation from the user interface and from the decoding policy that turns model outputs into a sequence.
Prefill and token-by-token decoding are different phases
Many autoregressive language-model runtimes can be understood as having a prefill phase followed by decoding. During prefill, the existing input sequence is processed. During decoding, new tokens are generated iteratively.
That distinction helps explain why two performance numbers can describe the same run. Prompt-processing throughput and generation throughput measure different work. A long input can increase time spent before the first generated token even when generation speed later remains similar.
The KV cache avoids recomputing all prior attention state
In many transformer implementations, key and value tensors from previous positions are retained in a KV cache. This allows subsequent decoding steps to reuse attention state instead of recomputing the complete history from scratch.
The cache is useful, but it consumes memory. Its size can depend on architecture, context length, precision, batch or parallel settings and runtime implementation. Some architectures use techniques that change this relationship, so a generic KV-cache formula should be treated as an estimate rather than a universal hardware requirement.
Weights are only part of runtime memory
Inference needs access to model weights, but a running model can also require memory for context state, temporary buffers and runtime overhead. Depending on the runtime and hardware, work may execute on a GPU, CPU or a combination of both.
This is why a downloaded model file fitting inside nominal VRAM does not guarantee that the complete workload will fit fully on the GPU. E—DOCEO's Qwen3 4B experiment demonstrated this distinction on a 3GB GTX 1060: the downloaded model and the running workload were not the same memory quantity.
Measure the workload, not just the specification
Useful inference measurements include time to first output, prompt-processing throughput, generation tokens per second, total latency, memory use and hardware placement. Which metric matters depends on the workload: interactive chat, batch extraction and coding-agent execution can value different things.
Specifications explain constraints. Reproducible runtime measurements show what actually happened under a defined environment.