noun · AI concept
Token
also called tokens
A Lego brick of text. The AI does not read letters or words, it reads small chunks, and it counts everything in chunks.
The unit a model reads and writes: whether a word, part of a word or a symbol, and the measure behind context windows, output caps and billing.
A tokenizer splits text into pieces from a fixed vocabulary, and each piece becomes an ID the model works with. Limits and prices are counted in these tokens, and the same text can produce different counts in different models.
Real-life example
You paste the same paragraph into two tools and one says 120 tokens while the other says 135. Neither is wrong. They split the text differently. A plain English word is often one or two tokens, but a long rare word, a block of code or a sentence in another language can break into many more.
Playground
Type anything and watch a real tokenizer cut it into tokens. The count is what limits, and bills, your request.
- Words
- 7
- Tokens
- ~9
- Characters per token
- 4.1
What the model sees
Unbelievably, the cat sat on the mat.
Each coloured block is one token. A space usually belongs to the word after it.
The model never sees letters. It sees the numeric IDs under each block, which is why counting letters in a word is harder for it than it looks.
Loading the real tokenizer… until then this is a rough estimate.
Where the analogy breaks
-
A token is not a word. Common words are often one token, but rare words, numbers and code split into several pieces.
-
Token counts differ between models. Each model family has its own vocabulary, so any count you see in a different tool is an estimate.
-
Not every language costs the same. Text in languages that are less common in the training data often needs more tokens for the same meaning, which uses up space and money faster.