Why does AI run on GPUs?

In detail

The work is parallel math

A neural network turns inputs into outputs through layers of multiplication and addition, mostly on large grids of numbers. Each output value can be computed largely independently of its neighbors, so the work splits into thousands of small, identical jobs.

That is the job a GPU is designed for. NVIDIA’s CUDA programming guide puts the difference plainly: a CPU is designed to run a serial sequence of operations as fast as possible and can run a few tens of these threads in parallel, while a GPU is designed to run thousands of threads in parallel, giving up single-thread speed for much greater total throughput. The GPU spends more of its transistors on arithmetic and fewer on caching and flow control. For matrix math, that trade pays off.

Gamers paid for the hardware first

The GPU began as fixed-function hardware for rendering 3D graphics. By 2003 parts of the graphics pipeline had become programmable, and in 2006 NVIDIA released CUDA so that any computation could use the GPU’s throughput, according to the same guide.

Carly Taylor, field CTO for gaming at Databricks, traces the economics. Players kept demanding more polygons and more realistic scenes, which pushed GPU makers to keep improving and drove the cost of each calculation down. Her suggested yardstick is the cost of a matrix calculation on a GPU, which kept falling. The data center GPUs that came out of that curve are now used mostly for generative AI, not for games.

One chip is no longer enough

Memory sets the first limit. A model has to fit in the memory of the chips serving it, which is why accelerator makers lead with memory capacity. AMD’s Anush Elangovan says the 288 GB on AMD’s 350 series puts a 500-billion-parameter model on a single GPU. That is AMD’s own framing, and the switching-cost explainer compares it with NVIDIA’s parts.

Training runs passed that limit long ago. Cisco’s Jeetu Patel describes the progression: models once fit in one GPU, then needed a server of four to eight GPUs, then a rack, then a row of racks. Now a single training run can need hundreds of thousands of GPUs. That can be more than one data center holds or more power than one site can draw, so separate buildings have to work as one cluster. His summary is that the GPU is the core asset and the network is the force multiplier. Without fast links between chips, the chips sit idle waiting for data.

GPUs are not the only option

An AI accelerator is any chip built to run this math faster or with less energy than a general-purpose processor. Google’s TPU is the best-known alternative. Google Cloud’s TPU architecture documentation describes it as an application-specific integrated circuit built for large matrix operations, using matrix-multiply units arranged as systolic arrays. Google’s 2017 paper on the first TPU reported that on its own production inference workloads, the chip averaged about 15 to 30 times the speed of a contemporary GPU or CPU, and 30 to 80 times the performance per watt. Those comparisons are against 2015-era hardware, not today’s GPUs.

The chip comes with its software. Paige Bailey of Google DeepMind says the JAX framework is loved partly because it uses TPUs so well and scales from one TPU to many, across data centers, which training Gemini requires. Logan Kilpatrick, then at Google DeepMind, credits TPUs with making long context practical to serve at scale. Both are describing their employer’s hardware.

Where it falls short

Not every part of an AI system belongs on an accelerator. AMD’s Sharon Zhou points out that an agent’s model calls run on GPUs, while many of the tools it calls run on CPUs, so the two loads have to be balanced. Google’s documentation makes a similar point from the other side: a program dominated by non-matrix operations, such as reshaping or concatenating data, will not keep the TPU’s matrix unit busy.

The hardware also does not decide the bill on its own. The energy cost of the chips is covered in how much electricity AI data centers use, and the software that keeps teams on one vendor is covered in the switching-cost explainer linked above.

Disclosure: Episode 47 is sponsored by Galileo, the Season 3 presenting sponsor; the read is at 49:58. Episodes 12, 26 and 28 were sponsored by Galileo, where Conor then worked; he introduces himself as Galileo’s Head of Developer Awareness at 0:05 in episode 28 and cites Galileo’s agent leaderboard at 1:08 in episode 12.