DSAI4207 · Lecture 6 · Generation, Decoding & Inference

Speculative Lab

Can a small model make a big one faster — same output?

An interactive companion for first-time learners, in three parts. Part I · Why — a big model writes one word per slow call, and two everyday observations hint at a way out. Part II · The idea — a small model guesses ahead and the big model checks: the whole method in one picture. Part III · The details — the accept rule, the correction that keeps the output exact, a round in code, and when it really pays off. Explore by scrolling, or start the Guided tour for a full-screen, one-step-per-page class presentation.

  • 3 parts · 11 short chapters
  • 10 predict & check questions
  • 36 guided steps
  • press F for fullscreen
Part I of III · Intuition & motivation
Why do we need a trick at all?

No formulas yet. First the problem — what makes a large model slow — and then two everyday observations that point to a way out.

PART IWhy
  • Good is slow, fast is sloppy
  • Most words are easy
  • Checking is faster than writing
PART IIThe idea
  • Guess, check, keep
  • Five questions it raises
PART IIIThe details
  • One pass · accept rule · correction
  • A full round, then in code
  • Speed-up and trade-off
1
01The dilemma

Good is slow. Fast is sloppy.

A large model is a careful writer, but it writes one word per call and every call is expensive. A model ten times smaller answers ten times quicker — yet it is not the writer we asked for. Press Write and watch both.

✎ Pause & predict

A GPU does thousands of things at once. Why can’t the big model produce all 20 words in a single call?

Press Write
Big modelthe writer we want — one slow call per word
0of 20 words
Small modelten times quicker per word
0of 20 words
time
small model donebig model done
0.0
one slow big-model callone quick small-model calla word the big model would pick less than 15% of the time here

Recall the KV Cache Lab: a cache makes each call cheaper, but the big model must still be called once for every new word.

Takeaway

We want the big model’s words at something like the small model’s speed.

2
02A first lucky fact

Most words are easy.

Here are the big model’s 20 words again. At every position we ask the small model for its first guess. Green means it would have guessed the very same word; red means it would have said something else. Click a word.

✎ Pause & predict

The small model is a much weaker writer. For how many of the big model’s 20 words is its first guess exactly that word?

Click a word
The big model’s 20 words
easy: the small model’s first guess is the same wordhard: it would have guessed a different word
small modelbig model
Takeaway

The big model’s care is only needed at a few hard words.

3
03A second lucky fact

Checking is faster than writing.

To write word 4, the big model must wait for word 3. But if the words are already on the page — guessed by someone else — nothing has to wait: the model can read them all in one call and say, for every position, what it would have written there.

✎ Pause & predict

Five guessed words are already on the page. How many big-model calls does it take to check all five?

Tick the clock
Writingeach word waits for the one before it
0big-model calls
Checkingthe guessed words are already on the page
0big-model calls

One tick is one call of the big model. A call that looks at a handful of positions takes about as long as a call that looks at one — the GPU handles them side by side.

Takeaway

Writing n words takes n calls. Checking n words takes one.

Part II of III · The approach
Let the small model guess. Let the big model check.

Put the two facts together — most words are easy, and checking is cheap — and the whole method fits in one picture: a junior writer drafts ahead, the senior writer approves in one look. (In the lecture’s Manga Studio: the Junior Artist sketches, the Artist approves.) Details wait until Part III.

PART IWhy
  • Good is slow, fast is sloppy
  • Most words are easy
  • Checking is faster than writing
PART IIThe idea
  • Guess, check, keep
  • Five questions it raises
PART IIIThe details
  • One pass · accept rule · correction
  • A full round, then in code
  • Speed-up and trade-off
4
04The idea in one picture

Guess, check, keep.

Two names first. The big model is the target: its text is what we want. The small one is the draft model. One round has three moves — and from here on we say token for a word or word piece. Press Next round and count how often each lane calls the target.

✎ Pause & predict

The draft model proposed most of the tokens. Whose writing is the final text?

Run a few rounds
1
GuessThe draft model writes a few tokens ahead — fast.
→
2
CheckThe target reads all of them in one call.
→
3
KeepKeep what passes, fix the first miss, drop the rest.
↻ repeat
Standard decodeone target call for every token
0target calls
Draft + verifythe draft guesses, the target checks once per round
0target calls
one target callone cheap draft calldraft token keptthe target’s fixbonus token from the target

Takeaway

The target is called once per round, not once per token.

  1. One passHow can one call check many tokens?
  2. Accept ruleWhen is a guess good enough to keep?
  3. CorrectionWhat do we write after a miss?
  4. A round, in codeHow do the pieces fit together?
  5. Speed-upWhen is it really faster?
Part III of III · Details & implementation
Now open the box.

Each chapter answers one of the five questions. The symbols p, q, γ and α appear only when they are needed, and every formula is filled in with the numbers on screen.

PART IWhy
  • Good is slow, fast is sloppy
  • Most words are easy
  • Checking is faster than writing
PART IIThe idea
  • Guess, check, keep
  • Five questions it raises
PART IIIThe details
  • One pass · accept rule · correction
  • A full round, then in code
  • Speed-up and trade-off
5
05Question 1 · one pass

One pass gives the target’s view at every position.

Call the number of drafted tokens γ. The draft samples each of them from its own distribution q. One target pass then returns the target’s distribution p at all γ positions — plus one more, for the token after them. Click a position to compare the two models.

✎ Pause & predict

The check at position 3 was computed assuming guesses 1 and 2 are right. Guess 2 turns out to be wrong. Is the check at position 3 still usable?

Click a position
draft qtarget p
Takeaway

γ cheap draft calls, then one target pass that returns γ + 1 distributions.

6
06Question 2 · the accept rule

Keep a guess as often as the target agrees.

In Part I, “agree” meant “same first guess”. But models roll dice, so agreement comes in degrees: the draft proposes a token with probability q, and the target would have chosen it with probability p. One position, two possible tokens — drag both and press Draw.

✎ Pause & predict

The draft proposes A with q = 0.8, but the target only gives A p = 0.6. What should we do with a proposed A?

Set p and q, then draw
One position · two tokens
both agreedraft too eagerdraft too shy
Takeaway

Accept a draft token with probability min(1, p / q).

7
07Question 3 · the correction

Corrections go where the draft was too shy.

After a rejection we must still output a token. Sample it from the residual — the part of the target’s probability the draft did not cover (purple). Switch the rule and compare what comes out with the target.

✎ Pause & predict

After a rejection, why not simply sample the replacement from the target p?

Pick the correction rule
Target p vs. draft q · “the cat sat on the ___”
both agreedraft too eagerdraft too shy = residual
What actually comes out
outputtarget psampled
Takeaway

Accept + residual correction reproduces the target distribution exactly.

8
08Question 4 · a full round

Keep the prefix, correct once, drop the rest.

Now the three moves from Part II with real numbers. Check the draft tokens left to right: everything before the first rejection is kept, the rejected position is corrected from the residual, and later drafts are thrown away because their context is now wrong.

✎ Pause & predict

All γ = 4 draft tokens are accepted. How many tokens does this round add to the text?

Step through a round
Text so far
Takeaway

Each target call commits 1 to γ + 1 tokens — more when the draft is good.

9
09Question 4, continued · implementation

The whole round fits in seventeen lines.

The same three moves as pseudocode. Step through the round you just watched: the executing lines light up, with their numbers on the right.

✎ Pause & predict

Set the temperature to 0, so the target always picks its single top token. What does line 10 turn into?

Step through the code
What these lines just did
Text so far
Takeaway

Draft, verify, commit — then repeat from the longer prefix.

Two cachesroll back after a miss

Each model keeps its own KV cache. After a rejection, the entries of the discarded draft tokens are dropped and the next round continues from the accepted prefix.

p and q are finalafter temperature, top-k, top-p

Apply the decoding settings first; the rule compares what is left. So the target’s sampling settings are preserved as well.

One argumentin a library

Hugging Face Transformers calls this assisted generation — one extra argument: generate(…, assistant_model=draft)

10
10Question 5 · when does drafting save time?

Tokens per round, divided by cost per round.

A round costs γ draft calls plus one target call, and returns a few tokens. Three numbers decide whether that beats standard decoding: the acceptance rate α, the draft’s relative cost c, and the draft length γ.

✎ Pause & predict

The draft is accepted 60% of the time and each draft call costs 30% of a target call. With γ = 5, is drafting faster?

Find the break-even

Analytic model of Leviathan, Kalman & Matias (2023): acceptances are treated as independent with rate α; one verification pass over γ + 1 positions takes about as long as one target call; c is the time of a draft call divided by a target call. It predicts shapes, not benchmarks.

Takeaway

Drafting pays off when α is high and c is small — and it can be slower.

11
11Putting it together

Same distribution, a different schedule.

Back to the dilemma of Part I. One request, drawn both ways on the same time axis as in the KV Cache Lab. Each tick is a token reaching the user: standard decoding ticks evenly, draft-and-verify delivers tokens in bursts.

Read the timeline

Time is counted in target-call units (one cached decode step = 1). Prefill is the same in both lanes, so the first token is not faster.

Takeaway

Speed comes from a second model — never from a different answer.

What stays the same?the output distribution

Tokens are still samples from the target model, with its temperature, top-k or top-p settings.

What gets faster?fewer target calls

Decode time, when the draft is cheap and usually right. Prefill and time-to-first-token do not improve.

What does it cost?a second model

Draft compute, its own KV cache in memory, wasted work after a rejection, and burstier streaming.