What Is an LLM
AI & ML: lesson 8 of 32
A next-token predictor trained on a lot of text.
Lesson 8 of 32 · 5 min
What Is an LLM
Step 1 of 12
vocabulary5 tokens (real: tens of thousands)
thecatsatonthematthe
cat
mat
sat
ran
on
A large language model has one job: given the tokens so far, score every possible next token.
The Idea
A large language model has one job: given the tokens so far, score every possible next token. Answering, translating and writing code all fall out of running that prediction in a loop, feeding each choice back in.
Real-World Example
A radio announcer who has read the shipping forecast for twenty years can finish a sentence before it is spoken. The phrasing is that patterned — no understanding of the weather required.
The Code
text = "the cat sat on the mat the cat ran"
words = text.split()
nxt = {}
for a, b in zip(words, words[1:]): # count what follows what
nxt.setdefault(a, []).append(b)
def most_likely(word): # a real model conditions on
options = nxt[word] # thousands of tokens, not one
return max(set(options), key=options.count)
print(nxt["the"]) # ['cat', 'mat', 'cat']
print(most_likely("the")) # catYour turn
What does this print?
text = "red fish blue fish red fish red"
words = text.split()
nxt = {}
for a, b in zip(words, words[1:]):
nxt.setdefault(a, []).append(b)
opts = nxt["fish"]
print(max(set(opts), key=opts.count))Mini quiz
1 / 3