Skip to content

AI, decoded

What is multimodal AI?

Multimodal AI processes, and sometimes generates, more than one kind of data — text, images, audio, video — and connects information across them. Supported inputs and outputs vary by model and endpoint. A single system, sometimes one model and sometimes several combined components, can read a chart and answer questions about it, transcribe speech and act on it, or describe a video, and some multimodal models can now generate images as well. A hard part isn't handling each modality; it's alignment — getting the model to connect what it sees, hears, and reads into one coherent understanding. Google DeepMind's Logan Kilpatrick told Chain of Thought that a successful domain-specific computer vision project once took roughly nine to twelve months; models now handle segmentation and object detection.

· Updated · Chain of Thought

Level 1: How AI works · 1.5 From chatbot to agent

Multimodal AIModel Architecture

  • 5 episodes
  • 5 guest clips
  • 6 glossary terms
Text, image, audio and video streams converging into a single model, beside the line: one model, many inputs.

In detail

One system, many kinds of input

A text-only model reads and writes words. A multimodal model takes in several kinds of data — words, pixels, sound, frames of video — and reasons over them together. That’s what lets a single system answer a question about a photo, pull the key figure out of a chart, or follow a spoken instruction, though that system may still combine specialized components. Some multimodal models can now generate images as well as read them.

Alignment is a hard part

Accepting an image and accepting text is the easy half. The difficulty is cross-modal alignment: making the model relate the thing it sees to the thing it reads so they form one understanding rather than two disconnected ones. When alignment is weak, the model describes an image fluently but gets the details wrong, or answers about what it expected to see instead of what’s actually there.

Update, October 3, 2026: Combining modalities also raises a system design problem. At 3:39 in episode 70, Tormod Ree describes cameras and microphones supplying signals to help interpret who is present, who is talking and how people interact. Ree identifies combining signals across multiple devices as a hard problem. At 6:27, Ree locates the differentiation in the harness and specialized data. These passages describe a system combining sensor signals; they do not establish that it uses one general multimodal model.

Worked example: read a chart, then check the answer

In this fictional example, a chart shows 20 support tickets on Monday and 30 on Tuesday, and the question is “By what percentage did tickets increase?”

StepWhat to establish
Read the imageThe bars show Monday: 20 and Tuesday: 30; the unit is tickets
Connect text and image“Increase” compares Tuesday with Monday, the starting value
Calculate(30 − 20) ÷ 20 × 100 = 50%
Check the evidenceVerify the labels, values and baseline against the source chart

If the image is cropped so the values can’t be read, a plausible “50%” doesn’t demonstrate visual understanding; ask for a clearer chart or the data. Reading both values correctly also doesn’t guarantee the calculation is right, so evaluate those failure points separately.

Why it matters, and where it breaks

Multimodal models open up tasks that pure-text systems can’t touch — visual question answering, document understanding, video analysis, voice interfaces. They also widen the surface for failure: a model can hallucinate about an image as confidently as about text, and evaluating “did it get the picture right” is harder than grading words. The capability is real; so is the need to measure it carefully.

The input side matters for the person using the model too. Intangible founder Bharat Vasan argued in episode 30 at 49:50 that directing an image or video model by typing is like a film director forced to give every instruction by text. Text suits text-based reasoning and research, he says at 50:19, but for visual work “text is just not the right way” to get what is in your head onto the screen. His company builds a 3D tool for showing a model what you mean, so this is also his product thesis. The episode aired in season two, which Galileo sponsored while Conor worked there; Conor points listeners to Galileo’s YouTube channel at 0:06.

Disclosure: Episodes 12, 20 and 30 aired in season two, which Galileo sponsored while Conor worked there; their show notes carry a Galileo block. Episode 47 was sponsored by Galileo, the Season 3 presenting sponsor (read at 49:58).

Multimodal AI: where you once needed a separate pipeline per input type, one model can now take several and connect them Before: a separate system for each kind of input, a text model, a vision model and a speech model, each its own pipeline; Logan Kilpatrick recalls that one domain-specific computer vision problem took on the order of nine to twelve months to solve in a successful case. Now: text, images, audio and video can flow into one model that connects them, answering about a photo, following a spoken instruction, and doing segmentation, bounding boxes and object detection; some such models also generate images. Which inputs and outputs are supported varies by model and endpoint. Where the inputs meet is a hard part, alignment: connecting what the model sees to what it reads into one understanding. Underneath, where it breaks: weak alignment gives a fluent description with the details wrong, a model can hallucinate about an image as confidently as about text, and checking whether it got the picture right is harder than grading words, so measure it. Many inputs can now meet in one model Words, pixels, sound and video, connected in one model instead of a pipeline per input. BEFORE: ONE SYSTEM PER INPUT text → text model image → vision model speech → speech model Logan Kilpatrick: one domain-specific vision problem took nine to twelve months, in a successful case. NOW: ONE MODEL CAN TAKE MANY INPUTS text image audio video alignment one model can answer about a photo, follow a spoken instruction, segment, box, detect some also generate images Not every model takes all four. Supported inputs and outputs vary by model and endpoint. A hard part: connecting what it sees to what it reads, so it's one understanding, not two. WHERE IT BREAKS A fluent description of the picture, with the details wrong. A model can hallucinate about an image as confidently as about text, and checking "did it get the picture right" is harder than grading words. The capability is real; measure it.
Many pipelines become one model, and alignment is where it can break. The nine-to-twelve-month figure and the vision capabilities are Logan Kilpatrick’s, from episode 12. Download the image

Hear it from the guest

“The amount of time it would take to solve one of these, like, domain specific computer vision problems was on the order of nine to twelve months in a successful case.”
Logan Kilpatrick, Google DeepMind EP 12 · around 19:10 · read the transcript
“The models can do image segmentation now, and they can do bounding boxes and object detection.”
Logan Kilpatrick, Google DeepMind EP 12 · around 19:53 · read the transcript
“I think the real differentiation isn't so much in the model for us.”
Watch on YouTube
“Now all you really need to do is just say reproduce this image in a target language.”
“You want the thing that's in your head as a human being to be the thing that you see on the screen, and text is just not the right way of doing that.”

Quotes lightly edited to remove filler words.

From the conversation

This explainer is drawn from these episodes — each carries its full transcript.

Concepts in this explainer

Multimodal AIGenerative AIText-to-ImageText-to-VideoComputer VisionObject Detection