Think of tuning a shower. Turn the knob a huge amount each time and you swing between freezing and scalding; turn it a hair each time and you stand there cold for minutes. Training a model works the same way. After every batch of examples, the training procedure (gradient descent) works out which direction would reduce the model’s error, measured by its loss function, and the learning rate decides how far to move the model’s parameters in that direction.
The learning rate is a hyperparameter: people pick it, the model doesn’t learn it. Google’s crash course describes the two failure modes. Set it too low and the model takes a very long time to settle; set it too high and it never settles, overshooting the best values over and over. In deep networks a rate that is far too high can make the loss jump to huge values or “NaN” (not a number), which is one reason teams watch loss curves closely in the first minutes of a run.
In practice the rate is rarely a single fixed number. Many runs use a schedule: a short warm-up at a low rate, then a gradual decay as training approaches convergence. Optimizers such as Adam also adapt the effective step for each parameter. The rate interacts with batch size, so changing one usually means re-checking the other. When a fine-tuning or training run goes wrong, the learning rate is one of the first settings to inspect, alongside the data.
Sources
- Google Machine Learning Crash Course: Hyperparameters — Defines the learning rate as a value you set that influences how quickly the model converges; too low takes a long time, too high never converges and bounces around the best weights; checked October 24, 2026.
- Google Machine Learning Glossary: learning rate — Describes the learning rate as the number gradient descent multiplies by the gradient to set the size of each weight update, and calls it a key hyperparameter; checked October 24, 2026.
- Smith: Cyclical Learning Rates for Training Neural Networks — Proposes letting the learning rate vary between bounds during training instead of holding it fixed, an early example of a learning-rate schedule; checked October 24, 2026.