adjective · AI concept
Multimodal
also called vision model
An AI that handles more than words. It can take in pictures, sound or video as well as text, and sometimes produce them too.
Able to process, and in some cases generate, more than one type of data, such as text, images, audio and video, in a single model.
Inputs such as images or audio are converted into tokens or embeddings and fed through the same model alongside text. That lets one model reason across them, though each image costs context space and fine details can still be missed.
Real-life example
You photograph a restaurant receipt and ask a model to total it. It reads the lines and gives you the sum. It did not look at the photo the way you would. The picture was turned into tokens, which also use up part of the context window, and a smudged digit can still trip it up.
Where the analogy breaks
-
It does not see like a person. An image is converted into numeric pieces, so small text, fine detail and exact positions can be misread.
-
Images and audio use up context. A single picture can take as much room as many words, so they count against the window and the bill.
-
Input and output can differ. A model may accept images yet only answer in text, so check which types go in and which come out.