Decoding Lab
How does an LLM choose its next word?
An interactive companion to the lecture. One toy language model, six ideas — scores, greedy, beam search, sampling, logit controls, and how to choose between them — each with something to drag, step through, or predict. Explore at your own pace by scrolling, or start the Guided tour for a full-screen, one-step-per-page class presentation.
- 6 ideas
- 6 predict & check questions
- 25 guided steps
- press F for fullscreen
The model gives scores. Who picks the word?
Every candidate below is a perfectly good continuation. Click one to fill in the blank — then ask who should decide which one wins.
The rest of this page is about exactly that choice: decoding.
- Scores → probabilitiesWhat the model outputs
- GreedyTake the favorite
- Beam searchKeep options open
- SamplingT, top-k, top-p
- Modify logitsPenalties and masks
- CompareQuality vs. variety
The model speaks in scores, not probabilities.
Each word gets a raw score, a logit. Softmax turns any scores into probabilities that add up to 100%. Drag a handle.
Add +1 to every score. What happens to the probabilities?
Only score differences matter. Softmax = exponentiate, then normalise.
Always pick the favorite.
The simplest decoder: read the context, get probabilities, take the tallest bar, append it, repeat — until <eos>.
Greedy took the best word at every step. Is the finished sentence the most likely sentence?
Greedy is fast and deterministic, but it can’t undo an early choice.
Keep a few options open.
Keep the B best partial sentences. Each round: extend every one, rank by total probability, prune the rest.
Greedy starts with “cat”. Does the best complete sentence start with “cat” too?
Advanced: length penalty α & the score formula
Ranked by Σ log P / lengthα. Try B = 4 and slide α from 0 to 1.
A wider beam searches more — at B× the cost, biased toward short outputs.
Add randomness, then shape it.
Instead of always taking the top word, draw from the distribution. Build the pipeline one stage at a time — each stage has exactly one knob.
No knob here — press › for the next stage.
Same recipe, whole sentences.
Apply the recipe at every step and you get full sentences — different every roll, but the junk tail stays out.
Top-k keeps a fixed number of words; top-p keeps a fixed probability. When does top-p keep more words?
Five sentences from this recipe
Uses the current T, top-k and top-p (set them on the previous slide), starting from “the”.
Temperature reshapes; top-k and top-p cut — then renormalise and draw.
Edit the scores before the choice.
Greedy can get stuck: same context, same favorite, forever. A logit processor sits between the model and softmax and rewrites the scores. First rule: every word loses λ points for each time it has already been used. Raise λ.
A penalty breaks the loop. Is a bigger penalty always better?
This is an additive frequency penalty. Libraries ship variants: Hugging Face’s repetition_penalty divides or multiplies the score instead, and no_repeat_ngram_size bans a repeat outright. Click any word to see how it was chosen.
A logit processor rewrites the scores before softmax — here z′ = z − λ × count.
A template decides what is allowed.
Task: return {"name": ‹string›, "age": ‹integer›}. The same box can do more than nudge: give every fragment the template forbids a score of −∞ and it can no longer be chosen. You are at the age slot — switch the mask on.
With the mask on, the output is always valid JSON with an integer age. Is the age also correct?
Allowed fragments depend on the prefix: mask the rest to −∞, renormalise, then decode as usual.
Quality, variety, and your choice.
Same model, same prompt. Move the knobs and watch each strategy slide between safe and varied.
Facts, code, translation — the most likely output, every time.
Stories, chat, brainstorming — randomness avoids dull repetition.
Pure sampling can wander into nonsense; cutting the tail drops the junk.
30 samples per sampling strategy. Greedy and beam are deterministic, so they sit at the far left.