noun · AI feature
Streaming
also called token streaming
Words appear as the AI writes them, instead of all at once at the end. It does not finish sooner, but it feels faster.
Delivering a model's output incrementally as it is generated, instead of returning it only after the full response is complete.
The server sends each token, or a small group of them, as soon as it is generated rather than waiting for the full reply. The client appends them as they arrive, which cuts the perceived wait but not the total generation time.
Real-life example
Your app waits twelve seconds and then dumps a full answer, and users think it froze. You switch on streaming, and the first words show up after about a second and keep flowing. The answer finishes at the same moment as before. It just no longer feels stuck.
Where the analogy breaks
-
Streaming does not make the answer finish sooner. The same text still takes the same time, you simply see it earlier.
-
Partial output is hard to use directly. Half a JSON object cannot be parsed, so structured results usually need the full reply first.
-
Stopping early may save work. If the reader cancels, many setups stop generating, but check how your provider counts partial replies.