Methodology

Measure first.
Explain second.

DOCEO LAB records are designed to make the tested configuration and the limits of each result visible.

1. Record the environment

Before a benchmark, we record the relevant operating system, processor, memory, GPU, VRAM, driver, runtime and model version. Configuration details that can affect the result—such as context size and quantization—belong in the test record.

2. Keep the procedure reproducible

Where practical, the exact commands, prompt and runtime options are published. When a test changes one variable, the other important variables are kept constant.

3. Separate cold and warm runs

Model loading can dominate a first request. Load duration and generation duration are therefore treated separately when the runtime exposes them. A cold run is not silently mixed with a warm run as if they measured the same thing.

4. Prefer raw measurements to scores

We publish observable values such as tokens per second, context size, processor placement, runtime size and GPU-memory snapshots. E—DOCEO does not create an arbitrary overall score or declare a universal winner from one setup.

5. Use medians when repeated runs are available

For repeated generation tests, the median is used as a compact summary because it is less sensitive to a single unusually fast or slow run. Individual runs remain visible when they matter to interpretation.

6. State limitations

Results apply to the recorded test environment. Windows WDDM GPU-memory figures are treated as observed aggregate snapshots rather than exact per-model VRAM requirements. A result on one GPU, driver or runtime version should not be presented as universal.

Verification labels

VERIFIED means the procedure described by the record was actually reproduced on the stated E—DOCEO test environment. It does not mean the result has been independently reproduced on every platform or hardware configuration.

The first public test environment and raw measurements are available in DOCEO LAB.