A single GPU would take years to train a modern large language model, and the model often would not fit in its memory anyway. Distributed training divides the job. This page covers the main strategies together, because real training runs usually combine them.
Data parallelism. Every GPU holds a full copy of the model and trains on a different slice of each batch. After each step, the GPUs average their gradients so all copies stay identical. Picture several accountants each auditing a different stack of invoices, then pooling their corrections. PyTorch’s DistributedDataParallel is the standard version. Sharded variants such as ZeRO and PyTorch’s FSDP split the optimizer state and weights across GPUs too, so each holds only part of the model.
Model parallelism. When the model itself is too big for one GPU, its pieces go on different GPUs. Tensor parallelism, used in Megatron-LM, splits the large matrix multiplications inside each layer. Pipeline parallelism, as in GPipe, puts different groups of layers on different GPUs and streams small batches through them like an assembly line. Hugging Face’s guide walks through when to use each.
On the show. In episode 49, at 11:25, Intercom’s Fergal Reid says the company’s AI infrastructure team built tooling that makes it easy for a scientist “to run a distributed training job on those those large gpu’s” when the team trained its own models. That is Reid’s account of Intercom’s setup. The episode’s sponsor read names Galileo.
Why it matters. Splitting the work means GPUs must constantly exchange data, so the network between chips matters almost as much as the chips themselves. Editorial synthesis: that communication cost is one reason AI factories are designed as whole systems and why training at frontier scale is limited to organizations that can buy large, tightly connected clusters. See also scaling laws and FLOPs.
Hear it from the guest
“To run a distributed training job on those large gpu's”
Quotes lightly edited to remove filler words.
Sources
- PyTorch documentation: Distributed Data Parallel — Design notes for PyTorch DistributedDataParallel, which keeps a model replica per process and synchronizes gradients between them; checked October 24, 2026
- Hugging Face Transformers documentation: Parallelism methods — Guide to training across GPUs with data, tensor, pipeline and model parallelism; checked October 24, 2026
- Shoeybi et al.: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism — NVIDIA paper that split transformer layers across GPUs (intra-layer or tensor model parallelism) to train models with billions of parameters; checked October 24, 2026
- Huang et al.: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism — Google paper that partitioned a network's layers across accelerators and pipelined micro-batches through them; checked October 24, 2026
- Rajbhandari et al.: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models — Microsoft paper that removes memory redundancy in data-parallel training by partitioning optimizer states, gradients and parameters across devices; checked October 24, 2026
