Softmax

Picture a support-ticket classifier choosing between “billing”, “bug” and “feature request”. Its last layer produces three raw numbers, say 2.0, 0.5 and -1.0. Those numbers, called logits, are not probabilities. Softmax exponentiates each one and divides by the total, which gives roughly 79%, 18% and 4%. The ticket goes to billing, and the system also knows how confident the model was.

How it works. Softmax keeps the ordering of the scores but exaggerates the gaps between them, because it uses exponentials. A score a few points higher than the rest ends up with most of the probability. Google’s crash course presents it as the standard way to get class probabilities that sum to 1.

In language models. A large language model runs a softmax over its whole vocabulary at every step to get a probability for each possible next token. Temperature divides the logits before the softmax: low temperature sharpens the distribution toward the top token, high temperature flattens it. Training usually pairs softmax with cross-entropy loss, which punishes the model when the correct answer gets a low probability. The attention mechanism inside a transformer also uses softmax to decide how much weight each earlier token gets.

Why it matters. Softmax probabilities look like calibrated confidence, but they are not guaranteed to be. A model can put 95% on a wrong answer. Editorial note: treat a softmax score as the model’s relative preference among the options it was given, and check calibration with real evaluation data before using the number to automate decisions.