Handling More Than Text
Multimodal means an LLM can handle not only text but also other kinds of data, such as images, audio, and video. It can be used to read text in a photo, read numbers off a chart, or summarize a meeting recording. The kinds of data handled differ by model, and the kinds accepted as input and the kinds it can output are often listed separately.
Reading and Creating Are Different
Even when a model is described as supporting images, being able to read images and being able to create them are separate abilities. Anthropic says its models can read and analyze images but cannot create or edit them. Creating images or audio is often the job of dedicated models.
The ability to read also has limits. Anthropic says errors are more likely with low-quality, rotated, or very small images, and that the model may not count many small objects accurately. When you have a model read tables or handwriting, test it on a few pages of your own material. The Image reading condition on the pages for choosing by use marks whether a model can read images.
Images Are Counted in Tokens Too
Inside the model, images are also handled by splitting them into tokens. Anthropic counts an image by dividing it into small patches, and Google counts it by dividing it into tiles of a fixed size. In both cases, larger images use more tokens.
Sending many images increases input tokens and raises the cost. If you do not need to read fine text, shrinking images before sending them makes it cheaper. The exact counting method differs by model, so check each developer's documentation.