noun · AI component
Tokenizer
also called token splitter
The paper shredder at the front door. It cuts your text into small standard pieces before the AI sees anything.
The component that splits text into tokens using a fixed vocabulary and converts them to IDs for the model, and back to text for the reply.
A fixed algorithm and vocabulary, built before training, that maps text to a sequence of token IDs and back. The model only ever sees the IDs, so the tokenizer decides what counts as one unit.
Real-life example
You ask a model how many times the letter r appears in a long word and it may get it wrong. It never saw the letters one by one. The tokenizer had already turned the word into two or three chunks, and counting letters inside a chunk is not something it can simply look at.
Where the analogy breaks
-
Tokenizers are not interchangeable. A model works only with the tokenizer it was trained with, so other tools give estimates.
-
Small changes in text can change the split. Extra spaces, capital letters and line breaks can turn into different tokens.
-
It makes some tasks harder than they look. Counting letters or reversing a word means working inside chunks the model cannot easily see into.