Think of editing a document with tracked changes. Each reviewer marks only what they would change, and the original text passes through untouched. A residual connection works the same way. The layer’s output is “input plus a correction”, so if a layer has nothing useful to add, it can learn to add close to zero and the signal passes straight through.
The problem it solved. Before 2015, adding more hidden layers to a network often made it worse, even on its own training data. He and colleagues called this the degradation problem and showed it wasn’t simply vanishing gradients: the deeper plain networks were harder to optimize, so even their training error rose. He et al.’s ResNet paper added shortcut connections that skip over blocks of layers. The paper trained image networks up to 152 layers deep, and ResNet won the 2015 ImageNet classification challenge.
Where it is now. Residual connections are standard. In the original transformer, each attention block and each feed-forward block has a residual connection around it. A modern large language model stacks dozens of those blocks. Interpretability researchers often describe the running sum that passes through all the blocks as the model’s “residual stream”.
Why it matters. Depth is one of the main levers behind deep learning progress, and residual connections are a large part of why depth became usable. Editorial synthesis: the change is a single addition per block, which shows that some important advances in AI come from making training stable rather than from a new kind of model.
Sources
- He et al.: Deep Residual Learning for Image Recognition — 2015 ResNet paper, which reformulated layers to learn residual functions and trained networks up to 152 layers deep; checked October 24, 2026
- Vaswani et al.: Attention Is All You Need — 2017 transformer paper, which wraps each attention and feed-forward sub-layer in a residual connection followed by layer normalization; checked October 24, 2026