Picture standing on a foggy hillside, trying to reach the valley floor. You can’t see the bottom, but you can feel which way the ground slopes under your feet. So you take a step downhill, check the slope again, and take another. Gradient descent trains a model the same way. The “height” is the model’s error, measured by its loss function, and the “position” is the current setting of its parameters. The gradient says which direction is uphill for each parameter, so the update moves the other way.
A concrete case: a model that predicts delivery times is off by several minutes on average. For each weight, training works out whether raising or lowering it would shrink that error, then moves every weight a little in the helpful direction. After thousands of rounds over the training data, the predictions get closer. For a neural network, backpropagation is the bookkeeping that computes those directions for millions or billions of weights at once; gradient descent is the step that uses them. The backpropagation entry walks through one such step with real numbers.
The size of each step is set by the learning rate, a hyperparameter people choose. Too large and training overshoots and becomes unstable; too small and it crawls. In practice, models rarely compute the slope over the whole dataset at once. Stochastic and mini-batch gradient descent estimate it from a small batch of examples per step, which is far cheaper, and optimizers such as Adam adjust the step size as they go.
Why it matters: this loop is where most of a model’s training compute goes, and it explains common training language. An epoch is one pass over the data. “The loss stopped going down” means the steps no longer help. A falling training loss does not guarantee the model will work on new examples; that is the overfitting risk. For simple linear models, Google’s course notes, the loss surface has one lowest point, so gradient descent reliably finds it. Neural networks have far bumpier surfaces, and training settles for a good-enough low point rather than a guaranteed best one.
Sources
- Google Machine Learning Crash Course: Gradient descent — Defines gradient descent as an iterative technique that repeatedly calculates the loss, finds the direction that reduces it, and updates weights and bias until further steps stop helping; checked October 24, 2026.
- Sebastian Ruder: An overview of gradient descent optimization algorithms — Surveys the batch, stochastic and mini-batch variants of gradient descent and the common optimizers built on it; checked October 24, 2026.
- PyTorch: torch.optim.SGD — Reference for the stochastic gradient descent optimizer, with the learning rate as its main setting; checked October 24, 2026.