Each neuron in a neural network adds up its weighted inputs. The activation function then transforms that sum. The most common one, ReLU (rectified linear unit), is simple: keep the number if it’s positive, otherwise output zero. Others, such as sigmoid and tanh, squash values into a fixed range, and GELU and SwiGLU are smoother variants used in many modern language models.
Without activation functions, stacking many layers would be no more powerful than a single layer, because a chain of purely linear steps collapses into one linear step. The bends that activation functions add are what let networks recognize faces, understand language and so on.
Why it matters
The choice of activation function affects how well and how fast a network trains. Early functions like sigmoid made deep networks hard to train because learning signals faded across layers; ReLU helped make deep learning practical.
No published Chain of Thought episode discusses activation functions directly yet, so this entry relies on the sources below.
Related terms
Multilayer perceptron is the simplest network where activations matter. Backpropagation is how networks learn through them.
Sources
- Goodfellow, Bengio and Courville: Deep Learning, Chapter 6 (Deep Feedforward Networks) — Textbook chapter explaining why networks need non-linear activation functions and recommending the rectified linear unit as the default for hidden units; checked October 24, 2026.
- PyTorch documentation: torch.nn.ReLU — Official documentation defining ReLU as max(0, x) applied element-wise; checked October 24, 2026.