BPE vs WordPiece Tokenization: How Each Picks Its Merges

Byte-pair encoding and WordPiece are the two subword tokenizers interviewers ask about by name. Both start from characters and glue adjacent pieces together, one merge at a time, until the vocabulary is big enough. They differ in which pair wins each round, and, less famously, in how a finished vocabulary cuts new text. Below, both rules run side by side on the same corpus.

The problem it solves

A model needs a fixed vocabulary of pieces that can spell any word: common words as one piece, rare words as a few. Why subwords at all, and what token counts cost you, is covered in tokenization explained. The question here is narrower: given a corpus, which merges make the vocabulary?

Both algorithms answer with the same loop:

1. Split every word into characters, plus an end-of-word marker.
2. Score every adjacent pair.
3. Merge the best pair everywhere, add it to the vocabulary, recount.
4. Stop at the target vocabulary size.

Only step 2 differs.

The intuition

Byte-pair encoding scores a pair by its raw count. If e + s sits side by side 9,000 times, it merges early, even though e and s are everywhere anyway.

WordPiece scores a pair by its count relative to its parts: count(ab) / (count(a) × count(b)). A pair whose halves almost never appear apart scores high even if it is rare; two very common pieces that merely happen to sit together score low. The original WordPiece paper (Schuster and Nakajima, 2012) phrases the choice as the merge that most increases the training data's likelihood under a language model; the ratio above is the usual way to compute that, and the original training code was never published (from memory). In the code below the ratio is a toy model of the scoring rule, not any library's trainer.

Encoding new text differs too:

• BPE replays its merges in the order it learned them, so a word is cut exactly as training would have cut it.
• WordPiece encoders match greedily: from the start of a word, take the longest vocabulary entry that fits, then continue. Pieces after the first carry a ## prefix, and if any stretch of the word cannot be matched, the whole word becomes [UNK] (from memory).

Byte-level BPE starts from the 256 byte values, so it never needs an unknown token. As of October 2026, large generative models mostly ship byte-level BPE variants, while WordPiece is mainly found in older encoder-only models (from memory).

Watch it run

The animation tokenizes "lower", the lesson's corpus being "low" three times and "lower" once. Both start with every character as its own piece. Byte-pair encoding counts every adjacent pair across the corpus and merges the commonest one: lo and ow both occur 4 times, and the first one seen wins, so merge 1 is l + o. The new piece is as wide as the two it swallowed, and the counts are redone. Next round, lo + w is the commonest pair, so "low" is born. Repeat a few thousand times and that pile of merges is the vocabulary.

WordPiece scores differently: not raw count, but the pair's count relative to its parts. er scores high because e and r are rare apart. Two very common pieces that merely sit together often score poorly under that rule. So the two tokenizers cut the same word in different places, and both are fine: four pieces against three. Vocabulary size is the real dial: too small and every rare word shatters, and you are billed per shard.

The code

One trainer, two scoring functions. Ties go to the pair seen first, as in the lesson. Fraction keeps WordPiece's ratio exact:

from collections import Counter
from fractions import Fraction

def counts(words):
    """words maps a tuple of symbols to its frequency in the corpus."""
    pairs, singles = Counter(), Counter()
    for symbols, freq in words.items():
        for s in symbols:
            singles[s] += freq
        for a, b in zip(symbols, symbols[1:]):
            pairs[a, b] += freq
    return pairs, singles

def bpe_score(pair, pairs, singles):
    return pairs[pair]                                     # raw count

def wordpiece_score(pair, pairs, singles):
    a, b = pair
    return Fraction(pairs[pair], singles[a] * singles[b])  # count relative to its parts

def merge(words, pair):
    out = Counter()
    for symbols, freq in words.items():
        new, i = [], 0
        while i < len(symbols):
            if symbols[i:i + 2] == pair:
                new.append(pair[0] + pair[1])
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] += freq
    return out

def train(corpus, rounds, score):
    words = Counter(tuple(w) + ("_",) for w in corpus.split())
    merges = []
    for _ in range(rounds):
        pairs, singles = counts(words)
        if not pairs:
            break
        best = max(pairs, key=lambda p: score(p, pairs, singles))  # ties: first seen
        merges.append(best[0] + best[1])
        words = merge(words, best)
    return merges, words

corpus = "low low low lower"
print(train(corpus, 3, bpe_score)[0])         # ['lo', 'low', 'low_']
print(train(corpus, 3, wordpiece_score)[0])   # ['er', 'lo', 'low']

corpus = "this is his list this is it his this list is quiz"
print(train(corpus, 4, bpe_score)[0])         # ['is', 'is_', 'his_', 'this_']
print(train(corpus, 4, wordpiece_score)[0])   # ['qu', 'th', 'thi', 'hi']

On the second corpus the two rules disagree from round one. BPE chases is, the most frequent pair. WordPiece's first merge is qu, a pair seen once, because q and u never appear apart: its score is 1, the maximum possible.

WordPiece's encoder, greedy longest match first:

def wordpiece_encode(word, vocab):
    """Greedy longest match first; non-initial pieces carry a ## prefix."""
    pieces, start = [], 0
    while start < len(word):
        for end in range(len(word), start, -1):
            piece = word[start:end] if start == 0 else "##" + word[start:end]
            if piece in vocab:
                pieces.append(piece)
                start = end
                break
        else:
            return ["[UNK]"]                          # one gap loses the whole word
    return pieces

vocab = {"un", "##aff", "##able", "a", "ab", "##bc"}
print(wordpiece_encode("unaffable", vocab))   # ['un', '##aff', '##able']
print(wordpiece_encode("unafable", vocab))    # ['[UNK]']
print(wordpiece_encode("abc", vocab))         # ['[UNK]'], yet 'a' + '##bc' would do

The last line is the surprise: greedy took ab, found nothing for ##c, and gave up, although a + ##bc spells the word. The seeded check: on 1,500 random corpora, WordPiece's pick must beat every pair by integer cross-multiplication, BPE's must be the most frequent pair, and merging must never change the text. Every non-[UNK] encoding must use only vocabulary pieces, and a brute force that tries every cut confirms the greedy miss happens:

import random

def segmentable(word, vocab):
    """Brute force: can the word be cut into vocabulary pieces at all?"""
    ok = [True] + [False] * len(word)
    for end in range(1, len(word) + 1):
        for start in range(end):
            piece = word[start:end] if start == 0 else "##" + word[start:end]
            ok[end] |= ok[start] and piece in vocab
    return ok[-1]

def best_by_cross_multiplying(pairs, singles):
    """Brute force for WordPiece's choice: integers only, every pair against every pair."""
    order = list(pairs)
    for p in order:
        if all(pairs[p] * singles[q[0]] * singles[q[1]] >= pairs[q] * singles[p[0]] * singles[p[1]]
               for q in order):
            return p

rng = random.Random(39)
ok, greedy_missed = True, 0
for _ in range(1500):
    corpus = " ".join("".join(rng.choice("abcde") for _ in range(rng.randint(1, 6)))
                      for _ in range(rng.randint(1, 12)))
    words = Counter(tuple(w) + ("_",) for w in corpus.split())
    for _ in range(rng.randint(1, 6)):
        pairs, singles = counts(words)
        if not pairs:
            break
        best = max(pairs, key=lambda p: wordpiece_score(p, pairs, singles))
        ok &= best == best_by_cross_multiplying(pairs, singles)
        ok &= max(pairs, key=lambda p: bpe_score(p, pairs, singles)) == max(pairs, key=pairs.get)
        before = sorted("".join(s) * f for s, f in words.items())
        words = merge(words, best)
        ok &= sorted("".join(s) * f for s, f in words.items()) == before
    vocab = {rng.choice(["", "##"]) + "".join(rng.choice("abc") for _ in range(rng.randint(1, 3)))
             for _ in range(8)}
    word = "".join(rng.choice("abc") for _ in range(rng.randint(1, 7)))
    got = wordpiece_encode(word, vocab)
    if got == ["[UNK]"]:
        greedy_missed += segmentable(word, vocab)
    else:
        ok &= all(p in vocab for p in got)
        ok &= "".join(p[2:] if p.startswith("##") else p for p in got) == word
print(ok, greedy_missed > 0)                     # True True

The complexity

• Training, as written: every round recounts every pair, roughly corpus size × merges. Real trainers update only the counts a merge touched.
• WordPiece encoding: up to L end points tried from each start of a length-L word, so O(L²) lookups; a cap on piece length bounds it.
• BPE encoding: with a rank lookup, the work is per merge applied, not per merge learned.

Where it goes wrong

• Calling WordPiece "frequency-based". It is frequency relative to the parts; a rare pair can win.
• Assuming greedy finds a cut whenever one exists. It does not, and the miss is a whole-word [UNK].
• Forgetting the markers. ## in WordPiece and the end-of-word or leading-space marker in BPE are part of the piece; low and ##low are different entries.
• Tuning the wrong dial. Which rule you pick matters less than vocabulary size: too small, and rare words shatter into many billed tokens.

When it shows up in interviews

In ML and LLM-engineering interviews: "how does BPE work?", then "how is WordPiece different?", and as a follow-up to token cost, such as why a rare name costs more tokens than plain English. Expect to explain both the training rule and the encoding rule.

How to say it in an interview

"Both learn a vocabulary bottom-up by merging adjacent pieces. BPE merges the most frequent pair. WordPiece divides the pair's count by the product of its parts' counts, so it prefers pairs that rarely occur apart, even rare ones. At encoding time BPE replays its merges in order, while WordPiece takes the longest matching piece from the left, marks continuations with ##, and maps the whole word to [UNK] if it gets stuck. Byte-level BPE avoids unknowns entirely by starting from bytes."