noun · AI concept
Non-determinism
also called randomness
Ask the same question twice, get two different answers. That is how it is built, not a glitch.
The same input can produce different outputs across runs because tokens are sampled, and in some setups because of hardware and batching effects.
The server's sampler draws each next token from a probability distribution instead of looking it up. With any randomness in sampling, the same input can branch into different outputs, and even settings meant to remove it do not always guarantee identical results.
Real-life example
You run a prompt that extracts dates from emails and it works. Run it again and one date is formatted differently. Nothing is broken, the model just chose a slightly different path, and your code assumed it never would. Plan for a range of outputs, not one.
Where the analogy breaks
-
It is not a bug to be fixed. The variety is part of how the model writes, and it is often useful.
-
Setting temperature to zero helps but does not promise identical results. Small numeric differences in serving can still change the output.
-
Testing needs a different mindset. One passing run proves little, so run many times and check the rate of success.