Multimodal AI

Models that see, hear, and read at once.

Multimodal AI describes models that take in and reason across more than one kind of data — text, images, audio, and video — in a single system, aligning those modalities so the model can answer about an image, transcribe and act on speech, or ground a response in a chart.

Multimodal is a description of inputs and outputs, not a promise that every kind of signal is supported. The GPT-4 technical report, for example, describes image and text inputs with text output (OpenAI, GPT-4 Technical Report). For a document-reading application, write down which evidence the system must use: the words in a table, the table's layout, or a chart beside it. Then test those cases separately instead of assuming a model that accepts an image will interpret every detail correctly.

Tormod Ree in episode 70 describes combining cameras and microphones in a meeting room; episode 5 with Chip Huyen and Vivienne Zhang covers evaluation challenges for generative and multimodal systems. What is multimodal AI? explains the term, while can a language model control a robot? follows perception into action. The edge AI definition covers a separate decision: where that processing happens.

8 episodes

Explainers on this topic

Terms on this topic

Guests on this topic

Tormod Ree Alex Ratner Dan Klein Paige Bailey Olga Beregovaya Logan Kilpatrick Chip Huyen Vivienne Zhang Sara Hooker Craig Wiley Vikram Chatterji