Skip to content

AI Glossary · Concepts

What is Temperature?

Temperature

Temperature controls randomness when sampling a model's output. Lower temperature concentrates probability on more likely tokens; higher temperature spreads it more broadly. Lower temperature can reduce variation, but it does not guarantee repeatable output.

· Updated · Chain of Thought

AI Engineering

  • 3 sources

At each step a language model produces a score (a logit) for every token in its vocabulary, and a softmax turns those scores into probabilities. Temperature divides the logits by a number t before the softmax. Below 1, the distribution skews toward the most likely tokens and the tail shrinks; above 1, it flattens and unlikely tokens get picked more often (Holtzman et al., The Curious Case of Neural Text Degeneration). Near 0, sampling approaches always taking the top token.

The same paper documents the tradeoff. Lowering temperature improves quality at the cost of diversity, and always picking the most likely continuation tends to produce bland, repetitive text, while sampling from the full distribution drifts into incoherence. The authors proposed nucleus (top-p) sampling, which samples only from the smallest set of tokens covering a set share of the probability, as a way to cut the unreliable tail without fixing the candidate count.

Practical guidance follows the task. Anthropic’s API documentation recommends a temperature closer to 0 for analytical and multiple-choice work and closer to 1 for creative and generative tasks, and adds that even at 0 results will not be fully deterministic (Anthropic, Messages API reference). So low temperature reduces variation; it does not guarantee repeatable output, and it does not make a model more accurate.

The control is also changing. The same reference marks temperature as deprecated and states that models released after Claude Opus 4.6 accept only the default of 1.0. If your evaluation or extraction pipeline relies on low temperature for consistency, check what each model version supports, and get consistency from structured outputs, validation and evaluation instead. How to test an AI system covers testing outputs that vary from run to run.

Follow two tokens through three settings

Dividing the logits by t before the softmax is the same as raising each probability to the power 1/t and renormalizing. Take an invented case with only two candidate tokens, A and B, at probabilities 0.8 and 0.2 when t = 1:

tProbability of AProbability of B
0.50.8² / (0.8² + 0.2²) = 16/17 ≈ 94.1%1/17 ≈ 5.9%
180%20%
2√0.8 / (√0.8 + √0.2) = 2/3 ≈ 66.7%1/3 ≈ 33.3%

Lower t concentrates the distribution and higher t flattens it, but the ranking never changes: A stays the more likely token at every setting. If A is the wrong answer, a low temperature makes the model wrong more consistently. These are hand calculations, not probabilities from a real model, and other filters such as top-p can change the candidate set further.

Two practical details. In Hugging Face Transformers, temperature must be strictly positive and does nothing unless sampling is turned on with do_sample=True; greedy decoding is a separate setting (Transformers docs). And on Anthropic’s API, version 2026-10-21 removed the parameter outright: a Messages or Message Batch request that sends temperature, top_p or top_k returns a 400 error, even at the default value. Record the model, API version and decoding settings with every evaluation run.

Sources

Go deeper

  • Hugging Face: Generation configuration docs

    Use do_sample to compare sampling with greedy decoding; inspect temperature and top-p, and check the model's generation configuration before changing defaults.

  • Anthropic: Messages API sampling controls docs

    Check whether your model version accepts temperature changes before using them in a repeatability test. The reference also explains why zero temperature does not guarantee identical responses.