Multimodal Search

Multimodal search is searching with more than one kind of input, or across more than one kind of content. A query can combine a photo with a typed question, a voice question about what a camera sees, or a circled part of the screen. Answers can draw on text, images and video together. Google Lens, Circle to Search and AI Mode are the best-known examples, and ChatGPT and Gemini handle image questions in the same way.

How it works

Two technologies make it possible:

• Multimodal models, sometimes called vision-language models, that read images and text together. They can identify objects, read text in images and describe relationships between objects in a scene.
• Multimodal embedding models, which place images, text and sometimes video and audio in one vector space. A photo of a chair and the words “mid-century oak armchair” end up close together, so either can retrieve the other.

Google’s milestones, as summarised in MLforSEO’s image SEO guide: in April 2025 AI Mode gained multimodal understanding by combining Lens with a custom Gemini model and using query fan-out to ask about the whole image and the objects within it. In May 2025 Google announced Search Live for talking with Search about what your camera sees, and in September 2025 it added visual search in AI Mode with a “visual search fan-out” that also looks at secondary objects in an image.

How it’s used in SEO and marketing

• Image SEO: original, high-quality images, descriptive file names and alt text, and surrounding text that names the product or entity shown.
• Product discovery: shoppers photograph items and ask where to buy them or what matches, so clear product images and complete product data matter.
• Video: chapters, transcripts and descriptive titles help systems understand and cite video content.
• Auditing at scale: multimodal models can check thumbnails, flag images whose content doesn’t match the page and draft alt text for review.
• Keeping key facts in HTML text as well as inside images, since text in images can be misread.

Example

Someone photographs a plant with yellowing leaves and asks AI Mode what is wrong. The system identifies the species from the image, generates sub-queries about that plant’s watering, light and pests, and cites a care guide whose photos show the same symptom next to clear text explaining the cause.

Limitations and things to watch

• Processing images is costly; systems resize and tile them, and small text in images can be lost.
• Vision models still misread stylised text, rotated images and fine spatial detail.
• Google does not publish exactly how Search uses image embeddings, so focus on clarity and relevance rather than guesses about ranking factors.
• Features and availability change quickly by country and device.

Learn it on MLforSEO

• Image SEO in 2026: The Complete Guide to Image Optimisation, Advanced Tactics and Multimodal Search
• How to do a YouTube SEO Audit
• Image Clustering by Color with Three Machine Learning Approaches
• Introduction to Machine Learning for SEO