BroadRouter
Get started

Top-p (nucleus sampling)

Updated

Top-p, also called nucleus sampling, is a decoding method in which a language model samples the next token only from the smallest set of most likely tokens whose combined probability reaches a threshold p, such as 0.9, and discards the unlikely tail.

Also called: nucleus sampling, top_p, top-p sampling

Top-p decides which tokens are allowed into the draw before the model picks one.

How it works

Holtzman and colleagues introduced nucleus sampling in 2019. They define the top-p vocabulary as the smallest set of tokens whose cumulative probability is at least p. Every token outside that set gets probability zero. The rest are rescaled to sum to one, and the next token is sampled from them. Their main results use p = 0.95.

The set changes size at every step:

Next-token distribution Tokens kept at p = 0.9
One token at 0.95 1
Four tokens at 0.3, 0.3, 0.2, 0.15 4
A long flat spread many

That is the point of the method. The paper argues the “less reliable tail” of the distribution causes degenerate text, and that cutting it gives varied output without losing coherence. A fixed top-k cannot adapt this way: it keeps the same count whether the model is certain or unsure.

In OpenAI’s API, top_p defaults to 1, meaning no cut. The reference gives the example that “0.1 means only the tokens comprising the top 10% probability mass are considered.” Hugging Face Transformers also defaults to 1.0 unless the model’s generation_config.json sets another value.

Why it matters

  • Fewer strange tokens. Cutting the tail removes the rare picks that make text drift, which matters for long answers and for hallucination risk.
  • Interaction with temperature. Temperature is applied to the scores; top-p then trims the result. Tuning both at once makes behaviour hard to predict, which is why OpenAI suggests changing only one.
  • Debugging. If logprobs show the model chose a low-probability token, a lower top-p would have blocked it.

Top-p does not change price. Every generated token is billed the same.

On BroadNet Inference

BroadNet’s API reference does not document top_p, top_k or their defaults. Its request examples set neither. An OpenAI SDK will pass top_p through, but whether a given model applies it is not stated, so test on your own prompts before relying on it. See the Python SDK guide and the post on evaluating a model before switching, which explains why sampling settings must be fixed during a comparison.

Common mistakes

  • Tuning top_p and temperature together, then not knowing which one changed the result.
  • Copying sampling settings from one model to another. Recommended values differ by model.

Questions

What is the difference between top-p and temperature?

Temperature reshapes all token probabilities. Top-p cuts off the unlikely tail and samples from what is left. OpenAI recommends changing one or the other, not both.

What is the difference between top-p and top-k?

Top-k keeps a fixed number of tokens, such as 50. Top-p keeps however many tokens it takes to reach the probability threshold, so the set is small when the model is sure and large when it is not.

Does BroadNet document top_p?

No. BroadNet's API reference does not list top_p or its default for any model. Set it in the request and test the effect yourself.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity