Cross-entropy loss

Imagine a spam filter that sees an email which really is spam. If the model says “99% spam”, the loss is close to zero. If it says “50% spam”, the loss is moderate. If it says “1% spam”, the loss is large. Cross-entropy is the negative logarithm of the probability the model gave the right answer, so it punishes confident mistakes far more than hesitant ones.

Where it shows up. Cross-entropy is the default loss function for classification. The model’s raw logits go through softmax to become probabilities, and cross-entropy scores the probability on the correct class. PyTorch’s implementation takes the logits directly and does both steps together for numerical stability. Gradient descent then nudges the model parameters to lower the average loss across the training data.

In language models. Pretraining a large language model is, at its core, cross-entropy on the next token: at each position, how much probability did the model give the token that actually came next? Perplexity is the exponential of that average loss, which is why the two numbers move together. The loss curves that labs report in papers on scaling laws are cross-entropy curves.

Why it matters. A lower loss means the model predicts its training distribution better. It does not mean the model is more truthful, safer or more useful for a given job. Editorial synthesis: loss is a training signal, and product decisions still need task-level evals.