A computer stores each number in a fixed number of bits. More bits mean finer detail and a wider range, but more memory and slower math. Classic scientific computing uses 32-bit floats (FP32). Neural networks turn out to tolerate much less detail for most of their arithmetic, so AI accelerators now support 16-bit formats (FP16 and BF16), 8-bit FP8, and even 4- and 6-bit formats.
Everyday analogy. Rounding prices to the nearest dollar is fine for a grocery budget but not for a bank ledger. Mixed precision applies that logic inside a model. Most multiplications run in a small format, while steps where rounding errors would pile up, such as adding up weight updates, stay in higher precision.
The formats. The 2017 mixed-precision paper showed that networks can train in FP16 if a full-precision master copy of the weights is kept and the loss is scaled up so tiny gradients do not round to zero. BF16, developed at Google, keeps FP32’s exponent range with fewer digits of detail, so, per Google’s TPU documentation, it has the same dynamic range as FP32 in half the memory and usually needs no loss scaling. FP8 halves the size again. Frameworks such as PyTorch’s AMP choose a format per operation automatically.
On the show. In episode 28, at 5:06, AMD’s Anush Elangovan describes the new data types in AMD’s CDNA 4 architecture: “We have FP4, we also have FP6”, and says FP6 can run at FP4-like throughput as an intermediate step down from FP8. That is AMD’s description of its own hardware. Chain of Thought’s second season was sponsored by Galileo, where Conor was head of developer awareness when he recorded the episode, as he says in the episode intro.
Why it matters. Halving the bits per number roughly halves memory and can multiply chip throughput, which cuts the cost of training and inference. Lower precision can also hurt accuracy, so it needs testing. Precision during training is related to, but different from, quantization, which usually shrinks a finished model’s weights for serving.
Hear it from the guest
“We have FP4, we also have FP6”
Quotes lightly edited to remove filler words.
Sources
- Micikevicius et al.: Mixed Precision Training — 2017 NVIDIA and Baidu paper that trained networks with half-precision numbers while keeping a single-precision master copy of the weights and scaling the loss; checked October 24, 2026
- Micikevicius et al.: FP8 Formats for Deep Learning — 2022 NVIDIA, Arm and Intel proposal of two 8-bit floating-point formats (E4M3 and E5M2) for training and inference; checked October 24, 2026
- Google Cloud TPU documentation: The bfloat16 numerical format — Google's explanation that bfloat16 has the same dynamic range as float32 while using half the memory; checked October 24, 2026
- PyTorch documentation: Automatic Mixed Precision package (torch.amp) — Reference for PyTorch's automatic mixed precision, which runs some operations in float16 or bfloat16 and keeps others in float32; checked October 24, 2026
