Skip to content

AI Glossary · Concepts

What is TPU (Tensor Processing Unit)?

TPU (Tensor Processing Unit)

A TPU is a custom AI accelerator chip that Google designs specifically for training and running neural networks, built as an alternative to general-purpose GPUs. Google trains its own models, including Gemini, on TPUs, which gives it a stack from the silicon up that doesn't depend on third-party GPU supply.

Also known as: tensor processing unit, tensor processing units, tpus

· Updated · Chain of Thought

  • 2 explainers
  • 4 episodes
  • 1 guest clip
  • 4 sources

GPUs were built for graphics rendering and adapted for machine learning later. Google designs TPUs as ASICs for machine-learning workloads, particularly large matrix computations. Google’s architecture description gives each TensorCore matrix-multiply units alongside a vector unit and a scalar unit; the details differ by generation. Because Google controls the chip design, the compiler stack (XLA), and the models that run on top of both, it can tune the whole pipeline together in a way that is harder when you build on someone else’s hardware, which is what most AI labs do with NVIDIA.

That vertical integration is part of why Google can price its model tiers so far apart. Pairing custom silicon with smaller model architectures pushes the cost per token of inference down far enough to serve a lot of users cheaply. TPUs also underpin the large training runs, where thousands of chips are networked across data centers to train frontier models like Gemini.

The arithmetic a TPU is built around

A teaching calculation: multiply the matrix with rows [1, 2] and [3, 4] by the vector [5, 6]. Each output is a row of multiply-and-add steps: 1 × 5 + 2 × 6 = 17, and 3 × 5 + 4 × 6 = 39, giving [17, 39]. A neural network layer is this operation at enormous scale, which is why a chip built around large multiply-and-accumulate units can be efficient for it. CPUs and GPUs compute the same result; the example shows the workload, not a TPU’s internal layout or speed.

Hear it from the guest

“But I think the only reason we're able to put long context into production, like at the scale that it is, is because of TPUs.”
Logan Kilpatrick, Google DeepMind EP 12 · around 25:46 · read the transcript

Quotes lightly edited to remove filler words.

Sources

Go deeper

From the conversation