Temperature Sampling: Controlling Randomness in Autoregressive Language Model Generation

Temperature sampling controls the randomness of language model generation by scaling the logit vector before applying softmax. The technique originates from the Boltzmann distribution in statistical mechanics and was adapted to neural language generation as a direct, differentiable control over output entropy.

The Formula

Standard softmax computes: p_i = exp(z_i) / Σ exp(z_j)

With temperature T: p_i = exp(z_i / T) / Σ exp(z_j / T)

The effect is equivalent to dividing all logits by T before the softmax operation. This does not change the ranking of tokens — only the sharpness of the resulting distribution.

Temperature Effects on Distribution

Temperature	Behavior	Typical Use Case
T → 0	Greedy: always argmax token	Factual Q&A, structured output
T = 0.3–0.7	Concentrated; low diversity	Code generation, factual writing
T = 1.0	Standard softmax (unchanged)	Default generation
T = 1.2–1.5	Higher diversity, more creative risk	Story generation, brainstorming
T > 2.0	Near-uniform distribution	Mostly noise; rarely useful

Repetition and Greedy Decoding

Greedy decoding (T→0) consistently selects the maximum-probability token at each step. While this produces locally coherent text, it creates global repetition: once a high-probability phrase begins, the model assigns high probability to its continuation indefinitely. Holtzman et al. (2020) showed that humans judge greedy-decoded text as significantly worse than human text, even when per-token probabilities are high — the “most probable” sequence is not the most human-like.

Relationship to Other Sampling Methods

Temperature is typically the first step in a multi-stage pipeline:

Method	Mechanism	Interaction with Temperature
Top-k sampling	Sample from k highest-probability tokens only	Applied after temperature scaling
Nucleus (top-p) sampling	Sample from smallest set with cumulative prob ≥ p	Applied after temperature scaling
Beam search	Maintain k hypotheses	Does not use sampling; temperature irrelevant

Standard practice: apply temperature scaling first, then top-k or top-p truncation, then sample from the resulting renormalized distribution.

Entropy and Temperature: Formal Relationship

For a distribution with entropy H at T=1, the entropy at temperature T is approximately:

H(T) ≈ H(1) + (T − 1) · Var(log p) / T

At T=1 the distribution is unchanged. At T→∞ the entropy approaches log|V| (uniform over vocabulary). At T→0 entropy approaches 0 (deterministic).

See top-p-sampling for the nucleus sampling approach that pairs with temperature, beam-search for deterministic sequence search, and autoregressive-decoding for the overall token-by-token generation loop.