noun · AI process
Sampling
also called decoding
Picking the next chunk of text from a list of options, like drawing from a hat where likelier choices have more tickets.
The step that chooses the next token from the model's probabilities, using a rule such as greedy selection or random draws shaped by temperature and top-p.
After the model produces probabilities for every token, a decoding rule selects one: always the top choice, or a random draw shaped by settings such as temperature and top-p. This happens once per token, so small choices compound across a reply.
Real-life example
You ask a model for a tagline twice. The first time it picks the most likely first word, the second time it picks the second most likely. Everything after that follows from the first choice, so the two taglines read very differently even though the model and prompt were the same.
Where the analogy breaks
-
Always picking the most likely token is not always best. It can make text repetitive and flat, which is why some randomness is added.
-
Every token is its own draw. An early unusual choice shapes everything after it, so small differences snowball.
-
Several settings affect it at once. Changing temperature and top-p together makes results harder to reason about, so change one at a time.