Think of a game of telephone with 100 people. By the end, a whispered message has either faded to nothing or been exaggerated into nonsense. Deep networks have a similar problem during backpropagation, the step that sends the error back from the output to every layer so each can adjust. The signal is multiplied layer by layer; if those multipliers are mostly below 1 it vanishes, and if mostly above 1 it explodes.
With vanishing gradients, the early layers barely change, so the network can’t learn long-range patterns; this is a big reason older recurrent networks struggled to remember the start of a long sentence. With exploding gradients, updates become huge and the loss jumps to enormous values or “NaN” (not a number), and training fails to reach convergence.
Much of modern deep learning design is a set of fixes for this. Activation functions such as ReLU avoid squashing the signal; careful weight initialization keeps multipliers near 1; LSTM cells in recurrent networks give information a protected path; residual connections, introduced in ResNet and used in every transformer, let the signal skip layers. Gradient clipping caps how large an update can be, and a modest learning rate with warm-up helps too. When a large training run “spikes” or diverges, engineers check these first.
Sources
- Google Machine Learning Glossary: vanishing gradient problem — Describes gradients of early hidden layers becoming surprisingly flat so those layers train slowly or not at all, and names LSTM as one mitigation; checked October 24, 2026.
- Google Machine Learning Glossary: exploding gradient problem — Describes very large gradients causing huge weight updates and unstable training, and names gradient clipping as a mitigation; checked October 24, 2026.
- Pascanu, Mikolov and Bengio: On the difficulty of training recurrent neural networks — Analyzes the vanishing and exploding gradient problems in recurrent networks and proposes gradient norm clipping for exploding gradients; checked October 24, 2026.
- He et al.: Deep Residual Learning for Image Recognition — Introduces residual connections that let networks over 100 layers deep train successfully; checked October 24, 2026.