Sampling Parameters Demystified: Temperature, Top-p, and Top-k

Part 7 of the LLM Inference from Scratch series.

When an LLM generates text, it does not "choose" the next word. It calculates a probability distribution over its entire vocabulary and samples from it. This process is governed by a sequence of mathematical transformations applied to raw model outputs called logits. Developers often treat parameters like temperature, top_p, and top_k as vague controls for "creativity," but they are precise mechanisms for reshaping probability distributions. Misunderstanding their interaction leads to unpredictable outputs, especially in production environments where consistency matters.

This guide breaks down the mechanism of each parameter, how they chain together, and why min_p is an emerging alternative to traditional truncation methods in specific inference engines.

The Foundation: From Logits to Probabilities

Before any sampling occurs, the model produces a vector of raw scores called logits for every token in its vocabulary. These logits are unnormalized log-odds. To convert them into probabilities, the inference engine applies the Softmax function:

$P(x_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}}$

Where $z_i$ is the logit for token $i$, and the denominator sums the exponentials of all logits in the vocabulary. This results in a probability distribution where all values sum to 1.0. Without sampling parameters, most systems default to Greedy Decoding, which simply selects the token with the highest probability ($\arg\max P(x)$). This is deterministic but often produces repetitive, low-quality text because it ignores the uncertainty inherent in language.

Sampling introduces randomness by selecting tokens based on their probabilities rather than just the maximum. The parameters below modify the distribution before this random selection occurs.

Temperature: Scaling Uncertainty

Temperature ($T$) acts as a scaler on the logits before Softmax is applied. The modified formula becomes:

$P(x_i) = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$

The mechanism is simple but profound:

• $T < 1$ (Sharp Distribution): Dividing logits by a small number amplifies differences between high and low scores. The highest-probability token becomes overwhelmingly likely, reducing randomness. This is ideal for factual QA or code generation where consistency is required.
• $T > 1$ (Flat Distribution): Dividing by a large number compresses the differences between logits. The distribution becomes flatter, giving lower-probability tokens a fair chance of being selected. This increases diversity but risks incoherence.
• $T = 1$: No modification. The raw Softmax probabilities are used.

Critical Note: Temperature does not change the rank of the tokens, only the confidence of the model's prediction. If the model is confident (high peak), lowering $T$ makes it more confident. If the model is uncertain (flat distribution), lowering $T$ may still result in a weak peak, but it reduces the impact of noise.

Top-k and Top-p: Truncation Strategies

After applying temperature (or if skipping it), the inference engine must decide which tokens are eligible for sampling. This is where truncation happens. The two most common strategies are Top-k and Top-p (nucleus sampling).

Top-k: Absolute Count

Top-k restricts sampling to the $k$ most likely tokens. All other tokens are assigned a probability of 0.

• Mechanism: Sort tokens by probability. Keep the top $k$. Renormalize the remaining probabilities so they sum to 1.
• Flaw: In a large vocabulary (e.g., 50k+ tokens), a fixed $k$ might include highly improbable tokens if the probability mass is spread out, or exclude valid tokens if the distribution is sharply peaked.

Top-p (Nucleus Sampling): Cumulative Probability

Top-p selects the smallest set of tokens whose cumulative probability exceeds a threshold $p$ (usually 0.9).

• Mechanism: Sort tokens by probability. Accumulate probabilities from highest to lowest until the sum $\ge p$. Discard all remaining tokens. Renormalize.
• Advantage: Top-p adapts to the confidence of the model. If the model is very certain (one token has 90% probability), Top-p will select only that token. If the model is uncertain, it will include more tokens. This makes it more robust than Top-k for variable contexts.

Min-p: The Modern Normalization

While Top-p is widely used, it has edge cases. For example, if the highest probability token is only 10%, Top-p=0.9 might still include hundreds of tokens with near-zero probability, leading to hallucination. Min-p addresses this by setting a relative threshold.

Mechanism of Min-p

Min-p defines a minimum probability threshold relative to the highest-probability token. If $P_{max}$ is the probability of the top token, and $min\_p$ is the threshold (e.g., 0.05), then any token with probability $P_i$ is kept only if:

$P_i \ge min\_p \times P_{max}$

• Example: If the top token has $P=0.5$ and $min\_p=0.05$, the cutoff is $0.025$. Any token with $P < 0.025$ is discarded.
• Comparison: If the top token has $P=0.01$, the cutoff becomes $0.0005$. This automatically prunes extremely unlikely tokens even when the overall distribution is flat.

Min-p is particularly effective in long-context scenarios where probability mass dilutes across many tokens. It prevents the "tail" of the distribution from influencing generation while preserving the shape of the high-probability region.

Synthesis: How They Chain Together

The order of operations in most inference engines (like vLLM or Hugging Face generate) is critical:

1. Logits: Raw model output.
2. Temperature: Scales logits. (Optional)
3. Truncation (Top-k / Top-p / Min-p): Filters the vocabulary. Tokens below the threshold are masked (probability set to $-\infty$ before Softmax, or 0 after).
4. Renormalization: The remaining probabilities are scaled to sum to 1.0.
5. Sampling: A token is drawn from the final distribution (e.g., using multinomial sampling).

Worked Scenario: Code Generation vs. Creative Writing

Scenario A: Python Function Generation

• Goal: High accuracy, no hallucination.
• Parameters: temperature=0.1, top_p=0.95, top_k=50.
• Mechanism: Low temperature sharpens the peak. Top-k limits the search space to 50 likely tokens. Top-p acts as a secondary safety net. The result is deterministic, coherent code.

Scenario B: Creative Storytelling

• Goal: Diversity, unexpected plot twists.
• Parameters: temperature=0.8, top_p=0.9, min_p=0.01.
• Mechanism: Higher temperature flattens the distribution, allowing less likely words to be considered. Top-p ensures we don't sample from the extreme tail. Min-p prevents the inclusion of tokens with negligible probability, maintaining basic coherence. The result is varied but readable text.

Common Pitfalls

When configuring sampling parameters, developers often overlook the interaction between truncation strategies and vocabulary size.

1. Top-k Fixed Count Flaw: As noted, Top-k is rigid. In contexts where the model is uncertain (flat distribution), a small k might exclude valid but low-probability tokens, forcing the model to choose from an artificially constrained set. Conversely, a large k in a sharp distribution might include "noise" tokens that slightly degrade output quality.
2. Top-p Edge Cases: Top-p can fail when the highest probability token is weak. If $P_{max} = 0.1$ and $p=0.9$, the algorithm must include tokens down to very low probabilities to reach the cumulative threshold. This can introduce significant noise and hallucination risk, effectively ignoring the "nucleus" concept if the distribution is uniformly flat.
3. Over-constraining: Using both strict Top-k and strict Top-p simultaneously can sometimes conflict, leading to unexpected renormalization effects where the remaining probability mass is concentrated on tokens that might not be semantically relevant.

Practical Takeaways

To optimize LLM outputs, align your parameters with the task requirements:

• Structured Outputs (Code, JSON, QA): Use low temperature ($T \le 0.2$) combined with strict truncation. A low top_k (e.g., 10-50) or low top_p (e.g., 0.9) ensures the model sticks to the most probable, accurate tokens.
• Creative Tasks (Storytelling, Brainstorming): Use higher temperature ($T \ge 0.7$) with adaptive truncation. top_p or min_p works better here because it adapts to the model's confidence. min_p is particularly useful for preventing extreme outliers in creative generation.
• Verification: Always log the effective vocabulary size after truncation. If the size is too small, you may be stifling creativity; if too large, you may be introducing noise.

FAQ

Does temperature change token rank? No. Temperature scales the logits but does not change their relative order. The token with the highest probability before temperature scaling will still have the highest probability after scaling. It only changes the magnitude of the differences between probabilities.

What is the difference between top-k and top-p? Top-k selects a fixed number of tokens ($k$) with the highest probabilities, regardless of their cumulative mass. Top-p (nucleus sampling) selects the smallest set of tokens whose cumulative probability exceeds a threshold $p$. Top-p adapts to the model's confidence, while Top-k does not.

When should I use min-p over top-p? Use min_p when you want to filter out tokens based on their probability relative to the most likely token, rather than a cumulative sum. This is especially effective in long-context scenarios or when the model's confidence is generally low, as it prevents including tokens with negligible probability that might still be included by top_p.

Conclusion

Sampling parameters are not interchangeable. Temperature controls the sharpness of the distribution. Top-k and Top-p control the breadth of the search space. Min-p provides a relative filter that adapts to model confidence.

For developers, the key takeaway is to avoid static configurations. Use low temperature and strict truncation (low top-k, low top-p) for structured outputs. Use higher temperature and adaptive truncation (top-p, min-p) for creative tasks. Always verify the effective vocabulary size by logging the distribution after truncation, as the interplay between these parameters can produce non-intuitive results.

Understanding these mechanisms transforms LLM generation from a black box into a tunable engineering process.