In detail
One system, many kinds of input
A text-only model reads and writes words. A multimodal model takes in several kinds of data — words, pixels, sound, frames of video — and reasons over them together. That’s what lets a single system answer a question about a photo, pull the key figure out of a chart, or follow a spoken instruction, though that system may still combine specialized components. Some multimodal models can now generate images as well as read them.
Alignment is a hard part
Accepting an image and accepting text is the easy half. The difficulty is cross-modal alignment: making the model relate the thing it sees to the thing it reads so they form one understanding rather than two disconnected ones. When alignment is weak, the model describes an image fluently but gets the details wrong, or answers about what it expected to see instead of what’s actually there.
Update, October 3, 2026: Combining modalities also raises a system design problem. At 3:39 in episode 70, Tormod Ree describes cameras and microphones supplying signals to help interpret who is present, who is talking and how people interact. Ree identifies combining signals across multiple devices as a hard problem. At 6:27, Ree locates the differentiation in the harness and specialized data. These passages describe a system combining sensor signals; they do not establish that it uses one general multimodal model.
Worked example: read a chart, then check the answer
In this fictional example, a chart shows 20 support tickets on Monday and 30 on Tuesday, and the question is “By what percentage did tickets increase?”
| Step | What to establish |
|---|---|
| Read the image | The bars show Monday: 20 and Tuesday: 30; the unit is tickets |
| Connect text and image | “Increase” compares Tuesday with Monday, the starting value |
| Calculate | (30 − 20) ÷ 20 × 100 = 50% |
| Check the evidence | Verify the labels, values and baseline against the source chart |
If the image is cropped so the values can’t be read, a plausible “50%” doesn’t demonstrate visual understanding; ask for a clearer chart or the data. Reading both values correctly also doesn’t guarantee the calculation is right, so evaluate those failure points separately.
Why it matters, and where it breaks
Multimodal models open up tasks that pure-text systems can’t touch — visual question answering, document understanding, video analysis, voice interfaces. They also widen the surface for failure: a model can hallucinate about an image as confidently as about text, and evaluating “did it get the picture right” is harder than grading words. The capability is real; so is the need to measure it carefully.
The input side matters for the person using the model too. Intangible founder Bharat Vasan argued in episode 30 at 49:50 that directing an image or video model by typing is like a film director forced to give every instruction by text. Text suits text-based reasoning and research, he says at 50:19, but for visual work “text is just not the right way” to get what is in your head onto the screen. His company builds a 3D tool for showing a model what you mean, so this is also his product thesis. The episode aired in season two, which Galileo sponsored while Conor worked there; Conor points listeners to Galileo’s YouTube channel at 0:06.
Disclosure: Episodes 12, 20 and 30 aired in season two, which Galileo sponsored while Conor worked there; their show notes carry a Galileo block. Episode 47 was sponsored by Galileo, the Season 3 presenting sponsor (read at 49:58).
Hear it from the guest
“The amount of time it would take to solve one of these, like, domain specific computer vision problems was on the order of nine to twelve months in a successful case.”
“The models can do image segmentation now, and they can do bounding boxes and object detection.”
“I think the real differentiation isn't so much in the model for us.”
“Now all you really need to do is just say reproduce this image in a target language.”
“You want the thing that's in your head as a human being to be the thing that you see on the screen, and text is just not the right way of doing that.”
Quotes lightly edited to remove filler words.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
-
Gemini 3 & Robot Dogs: Inside Google DeepMind's AI Experiments | Paige Bailey -
The Making of Gemini 2.0: DeepMind's Approach to AI Development and Deployment | Logan Kilpatrick -
11 Cameras, Dozens of Mics: The AI That Reads the Room | Tormod Ree, Neat -
Breaking the Language Barrier: Smartling's AI Translation Pipeline | Olga Beregovaya -
Taste Is The New Moat: Why Customer Obsession Wins in the AI Era | Bharat Vasan, Intangible
Concepts in this explainer
Multimodal AIGenerative AIText-to-ImageText-to-VideoComputer VisionObject Detection