Training a model is a loop: make predictions, measure how wrong they are with a loss function, work out which direction would reduce that error with backpropagation, then nudge the model’s parameters in that direction. The optimization algorithm is the part that decides how big each nudge is and how to combine it with earlier nudges.
A rough analogy is walking downhill in fog. Plain stochastic gradient descent takes a step in the steepest downhill direction you can feel right now. Adam, introduced by Kingma and Ba, also keeps a running memory of recent slopes and adapts the step size for each parameter, which often makes training more stable. PyTorch’s torch.optim package ships these and other optimizers, and a training script usually names one in a single line, such as choosing AdamW with a learning rate (a hyperparameter) for a fine-tuning run.
Why it matters: the choice of optimizer and its settings can decide whether a training run converges, how long it takes and how much it costs. The Deep Learning textbook by Goodfellow, Bengio and Courville devotes a chapter to why optimizing neural networks is harder than ordinary mathematical optimization. No Chain of Thought episode has discussed optimizers directly; this entry is background for the show’s training discussions.
Sources
- PyTorch: torch.optim — Official docs; supports the description of a package implementing optimizers such as SGD, Adam and AdamW, checked October 24, 2026
- Kingma and Ba, Adam: A Method for Stochastic Optimization (2014) — Primary paper; supports the description of Adam as adaptive estimates of first and second moments of the gradient, checked October 24, 2026
- Goodfellow, Bengio and Courville, Deep Learning, Chapter 8: Optimization for Training Deep Models — Authors' free textbook chapter; supports how training optimization differs from pure optimization, checked October 24, 2026