Top-p decides which tokens are allowed into the draw before the model picks one.
How it works
Holtzman and colleagues introduced nucleus sampling in 2019. They define the top-p vocabulary as the smallest set of tokens whose cumulative probability is at least p. Every token outside that set gets probability zero. The rest are rescaled to sum to one, and the next token is sampled from them. Their main results use p = 0.95.
The set changes size at every step:
| Next-token distribution | Tokens kept at p = 0.9 |
|---|---|
| One token at 0.95 | 1 |
| Four tokens at 0.3, 0.3, 0.2, 0.15 | 4 |
| A long flat spread | many |
That is the point of the method. The paper argues the “less reliable tail” of the distribution causes degenerate text, and that cutting it gives varied output without losing coherence. A fixed top-k cannot adapt this way: it keeps the same count whether the model is certain or unsure.
In OpenAI’s API, top_p defaults to 1, meaning no cut. The reference gives the example that “0.1 means only the tokens comprising the top 10% probability mass are considered.” Hugging Face Transformers also defaults to 1.0 unless the model’s generation_config.json sets another value.
Why it matters
- Fewer strange tokens. Cutting the tail removes the rare picks that make text drift, which matters for long answers and for hallucination risk.
- Interaction with temperature. Temperature is applied to the scores; top-p then trims the result. Tuning both at once makes behaviour hard to predict, which is why OpenAI suggests changing only one.
- Debugging. If logprobs show the model chose a low-probability token, a lower top-p would have blocked it.
Top-p does not change price. Every generated token is billed the same.
On BroadNet Inference
BroadNet’s API reference does not document top_p, top_k or their defaults. Its request examples set neither. An OpenAI SDK will pass top_p through, but whether a given model applies it is not stated, so test on your own prompts before relying on it. See the Python SDK guide and the post on evaluating a model before switching, which explains why sampling settings must be fixed during a comparison.
Common mistakes
- Tuning
top_pand temperature together, then not knowing which one changed the result. - Copying sampling settings from one model to another. Recommended values differ by model.