When people in AI say “compute,” they usually mean how much number-crunching something takes and the hardware that does it, mostly GPUs and other AI accelerators in a data center. It is measured in rough terms such as GPU-hours or total arithmetic operations, and it ends up as an electricity bill and a cloud invoice.
A bakery is a useful comparison. Developing a new recipe takes weeks of test batches: that is training compute, a large one-time cost to build the model. Selling loaves every morning is inference compute: a smaller cost per answer, paid again for every customer, which adds up with popularity. A model used by millions of people can spend more on inference over its life than it did on training. Newer reasoning models also spend extra work per question while they “think,” which is called test-time compute.
Why it matters: compute is a budget, and it shapes most trade-offs in AI. Research on scaling laws found that model quality improves predictably as training compute, data and model size grow together. DeepMind’s Chinchilla work showed that a smaller model trained on more data can beat a larger one for the same training compute, and is then cheaper to run. For a business, compute explains why bigger models cost more per token, why responses can be slow at peak times, and why AI companies compete for chips and power.
In episode 23 at 11:05, Poolside’s Eiso Kant says frontier labs work within “a compute budget” for both inference and training, which is why a model too large for customers to afford is not worth serving. Episode 23 aired in season two, which Galileo sponsored while Conor worked there (Conor mentions Galileo at 19:01). In episode 44 at 11:39, Cisco’s Jeetu Patel argues there is “not enough power, compute, network bandwidth, and data center capacity” for AI demand. That is his view of the market, and Cisco sells networking for those data centers.
The word has an ordinary meaning too, so this glossary does not link it automatically.
Hear it from the guest
“So we're all in a compute budget, compute budget and inference time that people can use.”
“There is just not enough power, compute, network bandwidth, and data center capacity to go satiate the needs of AI.”
Quotes lightly edited to remove filler words.
Sources
- Kaplan et al.: Scaling Laws for Neural Language Models — Finds language-model loss improves as a power law with model size, dataset size and the compute used for training, and uses this to allocate a fixed compute budget; checked October 24, 2026.
- Hoffmann et al.: Training Compute-Optimal Large Language Models (Chinchilla) — Trains Chinchilla with the same compute budget as the larger Gopher model but with 70B parameters and four times more data, and notes the smaller model needs less compute for fine-tuning and inference; checked October 24, 2026.
