Top-p and Top-k Sampling

At each step, a model scores every token it could produce next. After “The coffee was”, words like “hot”, “strong” and “cold” get high probability and thousands of unlikely words get tiny probabilities. Sampling picks one at random according to those probabilities. Without a cutoff, a rare odd word occasionally gets picked, and one odd word can derail the rest of the text.

Top-k sampling keeps a fixed number of candidates: with k set to 40, only the 40 most likely tokens are eligible. Top-p, also called nucleus sampling, keeps a variable number: with p at 0.9, it keeps the most likely tokens until their probabilities add up to 90%. When the model is confident that might be two tokens; when many continuations are plausible it might be hundreds. Holtzman and colleagues proposed nucleus sampling because fixed cutoffs and pure sampling both produced degenerate text in their experiments (Holtzman et al.). Libraries and APIs expose both as settings, often named top_k and top_p (Hugging Face; Gemini API).

These settings work together with temperature, which reshapes the probabilities before the cutoff is applied. Lower values give more predictable output, which suits extraction or classification; higher values give more variety, which suits brainstorming. Not every model or provider supports every setting, and defaults differ, so check the API reference for the model you use, change one setting at a time and judge the result on a fixed test set. Even with tight settings, output is not guaranteed to repeat exactly.

“Top-k” also appears in retrieval-augmented generation, where it means how many retrieved chunks are passed to the model. That is a retrieval setting, unrelated to token sampling.