noun · AI mechanism
Attention
also called self-attention
How the AI decides which earlier words matter for the word it is working on right now.
The mechanism in a transformer that lets each token weigh and combine information from other tokens in the input.
For each token the model computes weights over the other tokens in the input and uses them to blend their information. It lets a word like "it" pull in meaning from the noun it refers to, wherever that noun sits.
Real-life example
You write "The trophy did not fit in the suitcase because it was too big." To read that sentence correctly, a model has to connect "it" to the trophy and not the suitcase. Attention is the step where it looks back across the sentence and gives the trophy more weight.
Where the analogy breaks
-
It is not human attention. It is a calculation, and the model does not choose to focus or get distracted.
-
More input makes it more expensive. In the standard form the work grows faster than the length of the text, which is one reason long contexts are costly.
-
The weights are not a full explanation. They show one part of what happened, not why the model gave its answer.