Picture an image classifier trained on batches of 32 photos. After a layer computes its outputs, batch normalization subtracts that batch’s average and divides by its spread, then applies a learned scale and shift so the network can still choose the range it needs. Keeping each layer’s inputs on a predictable scale often lets teams use a higher learning rate and reach a good model in fewer steps.
The catch is that a deployed model often sees one photo at a time, so there is no batch to average. During training, the layer also keeps running averages of the statistics, and at prediction time it uses those stored values instead. In PyTorch this is the switch between training and evaluation mode. Serving a model still in training mode is a common bug: its answers can change with whatever else happens to be in the batch.
Why it works is still debated. The original paper credited reducing “internal covariate shift”; later research argues the main effect is making the training problem smoother. Batch statistics also get noisy with very small batches. Transformers, the architecture behind most large language models, usually use layer normalization instead, which normalizes across one example’s own values rather than across a batch.
Sources
- Ioffe and Szegedy: Batch Normalization — The 2015 paper that introduced the technique, normalizing layer inputs with mini-batch statistics so deep networks can train with higher learning rates; checked October 24, 2026.
- PyTorch: BatchNorm2d — Documents learnable scale and shift parameters and running estimates of mean and variance kept during training and used during evaluation; checked October 24, 2026.
- Santurkar et al.: How Does Batch Normalization Help Optimization? — Argues that batch normalization's benefit comes mainly from smoothing the optimization problem rather than from reducing internal covariate shift, the original explanation; checked October 24, 2026.
- Ba, Kiros and Hinton: Layer Normalization — Introduces layer normalization, which computes statistics across one example's features instead of across a batch; checked October 24, 2026.