Every multiplication or addition on decimal numbers inside a neural network counts as a floating-point operation. The spelling carries the meaning. “FLOPs”, lowercase s, is a count of operations, used to describe how much work a job takes. “FLOPS”, or FLOP/s, is a rate per second, used to describe how fast a chip is. The prefixes scale up quickly: a teraflop is a trillion, a petaflop a thousand trillion.
Everyday analogy. FLOPs are like the total distance of a road trip, while FLOPS is the car’s speed. A training run’s FLOP count, divided by the effective FLOPS of the cluster, gives roughly how long it will take. Real systems rarely run at their advertised peak, because chips spend time waiting on memory and on communication during distributed training.
Measuring training. Kaplan et al.’s scaling-laws paper estimates training compute as about six floating-point operations per parameter per training token. That rule of thumb links model size and data size to cost, and it underlies much of the work on scaling laws. Peak FLOPS also depends on number format: a chip does many more operations per second in FP8 or FP4 than in FP32.
In policy. Training compute is used as a legal threshold. Article 51 of the EU AI Act, shown in this reproduction of the regulation’s text, presumes that a general-purpose AI model has high-impact capabilities when its cumulative training compute exceeds 10^25 floating-point operations, and lets the Commission amend that threshold. Editorial synthesis: FLOP counts are easy to measure and hard to game, but they are only a proxy for capability, since better data and algorithms can deliver more capability per FLOP.
On the show. In episode 28, at 3:07, AMD’s Anush Elangovan says of AMD’s newest accelerator, “Like 20 petaflops of FP4 is mind blowing.” That is a vendor peak figure in a low-precision format. Chain of Thought’s second season was sponsored by Galileo, where Conor was head of developer awareness when he recorded the episode, as he says in the episode intro.
Why it matters. FLOPs are the common unit behind compute budgets, chip comparisons and regulatory thresholds.
Hear it from the guest
“Like 20 petaflops of FP4 is mind blowing.”
Quotes lightly edited to remove filler words.
Sources
- Kaplan et al.: Scaling Laws for Neural Language Models — 2020 OpenAI paper that estimates non-embedding training compute as C ≈ 6NBS, about six floating-point operations per parameter per training token, and reports compute in PF-days; checked October 24, 2026
- EU Artificial Intelligence Act, Article 51 (text published by the Future of Life Institute) — Reproduction of Regulation (EU) 2024/1689 Article 51, which presumes a general-purpose AI model has high-impact capabilities when its cumulative training compute exceeds 10^25 floating-point operations; checked October 24, 2026
