The problem
If you use AI for checks, reports or decisions, you want the same question to give the same answer: to test, to verify outcomes, and to explain why something was decided. That is why people set the "temperature" low or pass a fixed seed. Yet that turns out to be no guarantee. Providers themselves call a seed "best-effort": an attempt, not a promise. This phenomenon is called nondeterminism.
How rounding can cause a difference
A language model computes with billions of numbers, and computers round as they go. The order in which additions happen decides exactly how that rounding falls. Compute kernels split their work differently depending on how much work there is at once. A tiny difference can produce a different word early in the answer, and from there the rest of the answer runs differently. The research company Thinking Machines Lab showed in 2025 how the order of calculations can lead to different outcomes. llama.cpp, the engine underneath Ollama, warns about it too: with the prompt cache on, outcomes are not guaranteed to be bit-for-bit identical.
What we saw ourselves
In our own local AI gateway we ran the same call again and again, with fixed settings and a fixed seed:
- Right after loading the model, the answer was literally the same every time.
- As a second call on the already loaded model, the answer was different, but that one was also the same every time.
- The difference: on the second call, two pieces of the question came from the model's cache. The rest was therefore computed in different blocks and rounded differently.
So the answer depends not only on the question and the seed, but also on what the model did just before.
The difference often comes down to one word. If the model picks a different word early in the answer, because of the cache or because parts of the prompt interact, everything after it goes differently. That makes AI less predictable and less reproducible than it seems. Almost back to the punch card, where one shifted character also changed everything.
What it means
- Checks fluctuate. A check that approves the same answer one time and rejects it the next is not necessarily unreliable: it may simply run on a model that did something slightly different just before.
- Testing compares apples with oranges. Two test runs are only comparable if they start in the same state.
- Multiple measurements are not automatically independent. If you measure three times, you may be comparing outcomes from three different states.
Our approach
- Deterministic first. Anything ordinary code can calculate or look up is done by code; there this plays no role.
- Never a single measurement. For important questions we look at a cross-section of several answers. Another model checks the answers. We now also know that those measurements must start in the same state.
- Testing in the same state. For comparative tests we reload the model, so every run starts equally.
- Record and measure. Every finding goes into a log, with measurements and sources. That keeps the AI explainable.
- Honest about what is still open: how to start every important call in the same state, cheaply. Reloading works, but takes time.
What to watch out for
- A fixed seed is no guarantee of the same answer.
- Do not let decisions that matter depend on a single AI answer.
- If you want outcomes to be repeatable, record the state too: which model, which settings, and what happened just before.
Would you like advice on the behaviour of AI models? Request a free advice conversation.
Sources
Updated on