BPE vs WordPiece
AI & ML: lesson 23 of 32
Two ways to decide which pair of pieces becomes one piece.
Lesson 23 of 32 · 5 min
BPE vs WordPiece
Step 1 of 9
vocabularycharacters
Both algorithms start in the same place: every character is its own piece, and nothing is merged yet.
The Idea
Both algorithms start from single characters and repeatedly glue one adjacent pair into a new piece. Byte-pair encoding picks the pair that occurs most often. WordPiece picks the pair whose joint count is highest relative to its parts' counts, which favours pairs that genuinely belong together.
Real-World Example
Shorthand invented by two clerks. One merges whatever they write most; the other merges only where two marks almost never appear apart. Both shrink the page, and they disagree about the odd word.
The Code
from collections import Counter
toks = [list(w) + ["_"] for w in ["low", "low", "low", "lower"]]
for _ in range(2):
pairs = Counter(a + b for t in toks for a, b in zip(t, t[1:]))
best, n = pairs.most_common(1)[0]
print(best, n) # lo 4 then low 4
merged = []
for t in toks:
out, i = [], 0
while i < len(t):
hit = i + 1 < len(t) and t[i] + t[i + 1] == best
out.append(best if hit else t[i]); i += 2 if hit else 1
merged.append(out)
toks = merged
print(toks[0]) # ['low', '_']Your turn
Fill in the blank.
from collections import Counter
t = list("cocoa")
pairs = Counter(a + b for a, b in zip(t, t[1:]))
print(pairs.most_common(1)[0]) # ('___', 2)Mini quiz
1 / 3