Glossary
Multimodal model
A multimodal model processes more than one kind of input. “Modality” here means the form of the information: text, image, sound or video. A pure language model receives only text. A multimodal model takes in, for example, a photo and a question about it and answers with text. To make this possible, the image is first translated into numbers too, which the model can then compute with together with the text. Many multimodal chatbots are language models at their core: they predict the next text piece and additionally get image information in their input.
An example: Google’s Gemma 4 models take in text and images, some sizes also sound, and generate text. You can show them a photo of a sign and ask what it says.
Not to be confused with a model that can generate anything: Multimodal first describes what goes in. A model that understands images as input does not therefore generate images; Gemma 4, for example, accepts images but answers only with text.
Where you’ll come across it: In chatbots that let you upload photos, screenshots or documents, in voice assistants that process speech directly, and in announcements of new models that advertise “understands images and audio.”
Introduced in What an AI Model Actually Is.