Skip to content
BytePatterns

Tokenization Explained: How LLMs Split Text Into Subword Tokens

8 min readBytePatterns

Why language models read subword tokens, not words or letters: how byte-pair encoding learns merges, greedy matching, and why token counts set cost and context.

A language model never sees your text. It sees a list of integers, each the id of a token from a fixed vocabulary, and the step that produces that list is tokenization. It decides how many tokens a prompt costs, how much fits into a context window, and why a random identifier is expensive while a common word is cheap. This article covers the idea, a working byte-pair encoding trainer, and the edge cases that matter in practice. Descriptions of specific algorithms follow the papers and documentation listed at the end, as of September 2026.

The problem it solves

A model needs a finite vocabulary: each token id indexes a row in an embedding table. The two obvious choices both fail.

  • Whole words. The vocabulary becomes enormous, since "love", "loved", "loving" and "lovingly" each need an entry, and any word not seen in training maps to a single unknown token. The model cannot read a new name or a typo.
  • Single characters. The vocabulary is tiny and nothing is unknown, but sequences become several times longer, and a single letter carries little meaning on its own.

Subword tokenization sits between them. Frequent words get their own token; rare words are spelled with a few frequent pieces; anything at all can fall back to single characters or bytes.

The intuition

A subword tokenizer has two separate phases, and mixing them up is the most common source of confusion.

Training builds the vocabulary once, from a large corpus. Byte-pair encoding, introduced for machine translation by Sennrich, Haddow and Birch, starts from single characters and repeatedly merges the most frequent adjacent pair into a new symbol, recording each merge, until the vocabulary reaches a target size. WordPiece merges bottom-up too but scores a pair by its frequency relative to the frequencies of its two parts, favouring pieces that almost always occur together.

Encoding applies that fixed vocabulary to new text. BPE replays the learned merges in the order they were learned. The lesson's simpler matcher scans left to right and takes the longest vocabulary entry that fits, falling back to one character.

Byte-level BPE starts from the 256 possible byte values instead of characters, so every string is representable and there is no unknown token at all. Its vocabulary size is simply 256 byte tokens, plus the number of learned merges, plus any special tokens such as an end-of-text marker.

Watch it run

The animation runs the lesson's greedy matcher over three 12-character strings. "tokenization" becomes two tokens, "token" and "ization", because the vocabulary knows both. "unbelievable" shatters into three: "un", "believ", "able". The random identifier "x7fk9q2v8mz1" matches nothing, so the fallback emits one character at a time: twelve tokens. Same length, 2, 3 or 12 tokens, depending on what the vocabulary has seen.

Tokenization

Step 1 of 13

Before a model reads text, the text is split into tokens from a fixed, learned vocabulary.

The same interactive animation as the lesson — step through it with the controls.

The code

A byte-pair encoding trainer on a toy corpus. Words are split into characters plus an end-of-word marker _, and each round merges the most frequent adjacent pair everywhere:

from collections import Counter

def merge_word(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
            out.append(symbols[i] + symbols[i + 1])
            i += 2
        else:
            out.append(symbols[i])
            i += 1
    return out

def train_bpe(corpus, num_merges):
    words = Counter(corpus.split())
    split = {w: list(w) + ["_"] for w in words}
    merges = []
    for _ in range(num_merges):
        pairs = Counter()
        for w, count in words.items():
            s = split[w]
            for a, b in zip(s, s[1:]):
                pairs[(a, b)] += count
        if not pairs:
            break
        best = max(pairs, key=lambda p: (pairs[p], p))   # tie: larger pair wins
        merges.append(best)
        split = {w: merge_word(s, best) for w, s in split.items()}
    return merges

corpus = "low low low low lower lower newest newest newest widest"
merges = train_bpe(corpus, 6)
print(merges)
# [('o', 'w'), ('l', 'ow'), ('t', '_'), ('s', 't_'), ('low', '_'), ('e', 'st_')]

"ow" and "lo" both occur six times; the tie-break picks the alphabetically larger pair, so "ow" merges first; "low" follows, then the "est" ending is built from the back. Encoding a new word replays the merges in training order. Words the corpus never contained still come out as known pieces, and a word no merge touches falls back to single characters:

def encode(word, merges):
    symbols = list(word) + ["_"]
    for pair in merges:
        symbols = merge_word(symbols, pair)
    return symbols

print(encode("lowest", merges))   # ['low', 'est_']
print(encode("slow", merges))     # ['s', 'low_']
print(encode("newer", merges))    # ['n', 'e', 'w', 'e', 'r', '_']

The lesson's greedy longest-match encoder is simpler but not always the shortest encoding. With the vocabulary below, greedy grabs abc and is left with two single characters, while ab + cde needs only two tokens. A short dynamic program finds the true minimum:

def greedy(text, vocab):
    out, i = [], 0
    while i < len(text):
        fits = [v for v in vocab if text.startswith(v, i)]
        piece = max(fits, key=len) if fits else text[i]
        out.append(piece)
        i += len(piece)
    return out

def fewest(text, vocab):
    best = [0] + [None] * len(text)          # best[i]: fewest tokens for text[:i]
    for i in range(1, len(text) + 1):
        options = [best[i - 1] + 1]          # single-character fallback
        options += [best[i - len(v)] + 1 for v in vocab if text[:i].endswith(v)]
        best[i] = min(options)
    return best[-1]

vocab = ["ab", "abc", "cde"]
print(greedy("abcde", vocab), fewest("abcde", vocab))   # ['abc', 'd', 'e'] 2

Both checked on 2,000 random strings and vocabularies. Every encoding must join back to the original text, and the dynamic program must agree with a brute force that tries every way of cutting the string:

import random
from functools import lru_cache

def brute_fewest(text, vocab):
    allowed = set(vocab) | set(text)
    @lru_cache(maxsize=None)
    def go(i):
        if i == len(text):
            return 0
        return min(1 + go(j) for j in range(i + 1, len(text) + 1) if text[i:j] in allowed)
    return go(0)

random.seed(5)
ok, greedy_longer = True, 0
for _ in range(2000):
    text = "".join(random.choice("abc") for _ in range(random.randint(0, 10)))
    vocab = list({"".join(random.choice("abc") for _ in range(random.randint(2, 4))) for _ in range(4)})
    g = greedy(text, vocab)
    ok &= "".join(g) == text
    ok &= fewest(text, vocab) == brute_fewest(text, vocab) <= len(g)
    greedy_longer += len(g) > fewest(text, vocab)
    word = "".join(random.choice("lowerstin") for _ in range(random.randint(1, 8)))
    ok &= "".join(encode(word, merges)) == word + "_"
print(ok, greedy_longer > 0)   # True True

The complexity

Training BPE as written recounts every pair after every merge, so its cost grows with corpus size times the number of merges; production trainers update counts incrementally instead. Encoding with a list of merges is linear in the number of merges per word, which is why practical encoders look merges up by rank rather than looping over all of them.

The complexity that matters to users is the token count. Context limits, latency and API billing are all counted in tokens, so the same text can cost very different amounts depending on how well it matches the vocabulary. Common English words are often one token each; code, rare names, long numbers and text in scripts under-represented in training tend to split into many more pieces.

Where it goes wrong

  • Estimating tokens from characters. A rule like "four characters per token" is only a rough average for one kind of text. Count with the model's own tokenizer.
  • Mixing tokenizers. Token ids are meaningful only for the vocabulary that produced them. Counting with one model's tokenizer and sending text to another gives wrong budgets.
  • Expecting character-level skills. A model that receives a word as a few multi-letter tokens never directly sees its individual letters, which is one reason letter counting and spelling tasks can go wrong.
  • Forgetting whitespace and case. A leading space, a capital letter or a trailing newline can change how text is split, and therefore its cost.

How to say it in an interview

"Models read token ids from a fixed subword vocabulary. Whole words would need a huge vocabulary with unknown-word problems; characters make sequences too long. Byte-pair encoding learns the vocabulary by repeatedly merging the most frequent adjacent pair; at encoding time it replays those merges, so common words become one token and rare ones split into known pieces. Byte-level BPE starts from 256 bytes, so nothing is ever unknown. Token count drives context usage, latency and cost, so I always measure with the model's own tokenizer."

How the two main merge rules differ is the subject of BPE vs WordPiece; what happens to token ids next is covered in embeddings.

Sources