Temperature and Sampling

The Idea

The model hands back a score for every next token; sampling decides which one is used. Temperature rescales those scores first — below one sharpens toward the favourite, above one flattens the field. Top-k and top-p trim the tail before sampling.

Real-World Example

A tour guide answering the same question twenty times. Perfectly flat, they recite one sentence word for word. Loosened a little, every group gets fresh phrasing. Loosened too far, they start inventing history.

The Code

import math

def softmax(scores, t):
    exp = [math.exp(s / t) for s in scores]
    total = sum(exp)
    return [round(e / total, 3) for e in exp]

logits = [2.0, 1.0, 0.0]
print(softmax(logits, 0.5))   # sharp: the favourite dominates
print(softmax(logits, 1.0))   # the model's own distribution
print(softmax(logits, 2.0))   # flat: rare tokens get a real chance

import math

def softmax(scores, t):
  exp = [math.exp(s / t) for s in scores]
  return [round(e / sum(exp), 2) for e in exp]

# which temperature sharpens [1.0, 0.0] into [0.88, 0.12]?
print(softmax([1.0, 0.0], ___))