AI Glossary

Backpropagation

Backpropagation computes how a neural network's loss changes with each learned parameter. It applies the chain rule backward through the computation; an optimizer then uses those gradients to update the parameters.

· Updated · Chain of Thought

Training first runs a forward calculation, producing a prediction and a loss. Backpropagation works backward through those operations to compute gradients: how sensitive that loss is to each weight and bias. The optimizer chooses how to use those gradients.

For a small example, let the prediction be y = w × x and loss be L = (y − target)². With input 2, weight 3 and target 4, the prediction is 6 and loss is 4. The chain rule gives dL/dw = 2(y − target) × x = 8. A gradient-descent step with learning rate 0.1 changes the weight to 2.2. This example is illustrative, not a general choice of learning rate.

The distinction matters when debugging training. Computing the gradient does not guarantee a useful update, a global optimum or good performance on new examples. Ordinary serving inference uses learned parameters without training them; fine-tuning adds a training loop that can use backpropagation.

Compute gradients, then update weightsA forward pass predicts y equals 6 and computes squared loss 4. Backward chain-rule derivatives give dL/dy equals 4, dy/dw equals 2 and dL/dw equals 8. A separate gradient-descent step with rate 0.1 updates the weight from 3 to 2.2. Compute gradients, then update weightsAn illustrative one-weight example: input x = 2, weight w = 3, target = 4 1. Forward: prediction and lossy = w × xy = 3 × 2 = 6L = (y − target)²L = (6 − 4)² = 42. Backward: chain ruledL/dw = 4 × 2 = 8dL/dy = 2(6 − 4) = 4dy/dw = x = 23. Optimizer step: w ← w − 0.1 × 8 = 2.2Backpropagation supplies gradients; the optimizer chooses the update.
An illustrative chain-rule calculation following PyTorch’s autograd tutorial. The learning rate is an example, not a recommended setting. Download the image

Sources

Go deeper

  • 3Blue1Brown: Backpropagation, intuitively video

    See how an error signal changes weights through several layers.

Related terms

Agent MemoryAgent SwarmAgentic AIAgentic RAG