GPUs were built for graphics rendering and adapted for machine learning later. Google designs TPUs as ASICs for machine-learning workloads, particularly large matrix computations. Google’s architecture description gives each TensorCore matrix-multiply units alongside a vector unit and a scalar unit; the details differ by generation. Because Google controls the chip design, the compiler stack (XLA), and the models that run on top of both, it can tune the whole pipeline together in a way that is harder when you build on someone else’s hardware, which is what most AI labs do with NVIDIA.
That vertical integration is part of why Google can price its model tiers so far apart. Pairing custom silicon with smaller model architectures pushes the cost per token of inference down far enough to serve a lot of users cheaply. TPUs also underpin the large training runs, where thousands of chips are networked across data centers to train frontier models like Gemini.
The arithmetic a TPU is built around
A teaching calculation: multiply the matrix with rows [1, 2] and [3, 4] by the vector [5, 6]. Each output is a row of multiply-and-add steps: 1 × 5 + 2 × 6 = 17, and 3 × 5 + 4 × 6 = 39, giving [17, 39]. A neural network layer is this operation at enormous scale, which is why a chip built around large multiply-and-accumulate units can be efficient for it. CPUs and GPUs compute the same result; the example shows the workload, not a TPU’s internal layout or speed.
Hear it from the guest
“But I think the only reason we're able to put long context into production, like at the scale that it is, is because of TPUs.”
Quotes lightly edited to remove filler words.
Sources
- Google Cloud: Introduction to Cloud TPU — Describes TPUs as ASICs for machine-learning workloads built for large matrix operations, explains the XLA compiler, and compares when TPUs or GPUs fit better; checked October 25, 2026.
- Google Cloud: TPU architecture — Describes each TensorCore as one or more matrix-multiply units plus a vector unit (activations, softmax) and a scalar unit (control flow, memory addresses); generations differ; checked November 17, 2026.
- Jouppi et al.: In-Datacenter Performance Analysis of a Tensor Processing Unit — Describes the first TPU, an inference ASIC deployed in Google datacenters since 2015 and built around a 65,536-unit matrix multiply unit; checked October 25, 2026.
- Gemini Team: Gemini: A Family of Highly Capable Multimodal Models — States that Gemini models were trained on TPUv5e and TPUv4, with Gemini Ultra trained on a large TPUv4 fleet across multiple datacenters; checked October 25, 2026.
Go deeper
From the conversation
-
Gemini 3 & Robot Dogs: Inside Google DeepMind's AI Experiments | Paige Bailey -
The Making of Gemini 2.0: DeepMind's Approach to AI Development and Deployment | Logan Kilpatrick -
The AI Framework Era Is Over: Why Context Is the Moat | Jerry Liu of LlamaIndex -
Context Poisoning is Killing Your AI Agents: How to Stop it | Michel Tricot, Airbyte