Skip to content
ignitae
Academy
Decoding: logits to text
How a model's output scores become the next token: greedy, softmax, top-k, sampling and beam search.
Syllabus
Greedy decoding
Greedy decoding takes the most likely token
Greedy decoding (exercise)
Softmax
Softmax turns logits into a probability distribution
Softmax (exercise)
Top-k filtering
Top-k keeps only the most likely candidates
Top-k filtering (exercise)
Temperature sampling
Sampling with temperature picks a token at random, weighted by probability
Temperature sampling (exercise)
Beam search
Beam search keeps several candidates and expands the best
Beam search step (exercise)