Tokenizers and the Vocabulary Budget

The tokenizer is chosen early and cannot be changed afterwards without retraining, which makes it one of the highest-leverage decisions available at the start of a project. It sets the sequence length for every document the model will ever see, the size of two of its largest matrices, and how much users writing in any given language pay per request.

It also offers unusually tractable wins. Splitting numbers consistently improves arithmetic, dedicated indentation tokens cut Python sequence length, and upsampling low-resource languages when fitting the vocabulary lowers their fertility directly. Each is a decision, not a research problem.

Fertility

The quantity that matters is fertility: tokens produced per unit of text. Lower fertility means shorter sequences, which reduces cost at every layer.

For English, a well-fitted 32k BPE vocabulary lands around 1.3. For languages written in scripts underrepresented in the training corpus, the same tokenizer can produce 3 to 5. This is not a small inefficiency: the same semantic content costs several times more, consumes several times more of the context window, and is billed several times more.

Fertility is also compute. Attention is quadratic in sequence length during prefill and linear per token during decode. A fertility ratio of 3 between two languages means a factor of 3 in decode cost and up to 9 in prefill attention for the same content.

The vocabulary tradeoff

Larger vocabularies reduce fertility. They also enlarge two matrices:

At $V = 128\text{k}$ and $d_{\text{model}} = 8192$ with untied embeddings, that is 2.1B parameters, a meaningful fraction of a mid-sized model, spent entirely on the interface between text and hidden states.

The softmax cost is the other side. Every generated token requires a projection to $V$ logits, which at large $V$ becomes a nontrivial share of per-token work, and the logit tensor itself becomes a real memory allocation in a large batch.

The tradeoff:

• Sequence length savings apply at every layer, every token and compound with context length.
• Embedding cost is paid once in parameters and once per token at the output projection.

Doubling the vocabulary yields diminishing fertility improvements, because Zipf’s law means each doubling covers progressively rarer strings, while parameter cost grows linearly. The optimum has moved upward over time as models got wider and multilingual coverage became a priority: 32k was standard, 128k is now common, and 256k appears in the most multilingual designs.

Empirical work on vocabulary scaling finds that optimal vocabulary size grows with model size, and that many models are under-vocabularized relative to their compute budget.¹ The reasoning is the same as any scaling-law allocation: the embedding parameters compete with depth and width for a fixed budget, and the balance shifts as the budget grows.

Algorithms

BPE starts from bytes and iteratively merges the most frequent adjacent pair, recording the merge sequence. Encoding replays the merges in order. Deterministic, fast, and the dominant choice.²

WordPiece is similar but selects merges by likelihood improvement rather than raw frequency.

Unigram starts from a large candidate set and prunes to maximize corpus likelihood under a unigram model, which yields a probabilistic segmentation and enables subword regularization, meaning different segmentations are sampled during training as an augmentation.

Byte-level fallback guarantees no out-of-vocabulary input by allowing any byte as a token. Every modern tokenizer includes this. The safety is real; the cost is that rare scripts degrade to near-byte-level fertility.

The pre-tokenization regex matters as much as the algorithm. It decides where merges may not cross, typically at whitespace and punctuation boundaries, with special handling for digits and contractions. Two tokenizers with the same algorithm and vocabulary size but different pre-tokenization patterns produce meaningfully different vocabularies.

Tokenizer fixes for apparent model limitations

Arithmetic. If numbers tokenize inconsistently, with 1234 as one token and 1235 as two, the model must learn arithmetic over a representation where digit position is not consistently exposed. Splitting numbers into individual digits, or into fixed groups of three, measurably improves arithmetic. This is a pure tokenizer fix for what is invariably reported as a reasoning limitation.

Character-level tasks. Counting letters in a word is hard when the word is one token. The model has no direct access to the characters; it must have learned the spelling as a property of the token embedding. The well-known counting failures are tokenization artifacts, not reasoning failures.

Code indentation. Python tokenizers that do not have dedicated multi-space tokens spend many tokens on leading whitespace. Adding explicit indentation tokens is a common and effective optimization.

Glitch tokens. Strings present in the tokenizer’s training corpus but absent or near-absent from the model’s, leaving their embeddings essentially untrained. Prompting with them produces erratic behavior. These arise when the tokenizer is trained on a different corpus than the model, which is easy to do accidentally.

Trailing whitespace. A prompt ending in a space often tokenizes differently than the model expects, because the merge rules attach leading spaces to following words. The result is a token sequence off the training distribution, and noticeably worse completions. Most APIs now strip trailing whitespace for this reason.

Multilingual fairness

Vocabulary allocation follows the tokenizer’s training corpus. A corpus that is 90% English produces a vocabulary where most entries are English subwords, and every other language falls back toward bytes.

The consequences are concrete and asymmetric:

• Cost. Per-token billing means users writing in high-fertility languages pay several times more for equivalent content.
• Context. A fixed context window holds several times less text.
• Quality. Longer sequences for the same content mean more positions to model and, empirically, worse performance.

Mitigation is a corpus-sampling decision at tokenizer training time. Upsampling low-resource languages when fitting the tokenizer, independently of the model’s training mix, raises their vocabulary allocation and lowers their fertility, at some cost to English fertility. Given how much larger the effect is for the disadvantaged languages, the trade is usually clearly worth making.

Measure it directly:

def fertility_report(tokenizer, corpora):
    rows = []
    for lang, text in corpora.items():
        n_tok = len(tokenizer.encode(text))
        n_word = len(text.split())
        rows.append((lang, n_tok / n_word, n_tok / len(text)))
    base = dict((l, f) for l, f, _ in rows)["en"]
    for lang, f, tpc in sorted(rows, key=lambda r: -r[1]):
        print(f"{lang:<8} {f:5.2f} tok/word  {tpc:5.3f} tok/char  {f / base:4.1f}x en")
    return rows

Run this against a parallel corpus, so the comparison is over identical content. The ratio in the last column is the multiplier those users pay.

Extending a vocabulary

Adapting a model to a new domain or language by adding tokens is possible but has a specific pitfall.

New embedding rows are randomly initialized while every existing row is trained. The mismatch in magnitude and direction destabilizes training and can degrade the base model’s existing capability. The standard fix is to initialize each new token’s embedding as the mean of its constituent old tokens’ embeddings, which places it in an already-sensible region of the space:

def init_new_token(new_str, old_tok, old_emb):
    pieces = old_tok.encode(new_str, add_special_tokens=False)
    return old_emb[pieces].mean(dim=0)

Then train, typically with a lower learning rate on the embedding layer than on the rest. Even so, expect some regression on the original distribution. Extension is not free, and for a large change a fresh tokenizer plus full training is often cleaner.

Choosing

The decision sequence:

1. Fit the vocabulary to the actual deployment mix, not to a general web corpus, and upsample underrepresented languages when fitting.
2. Size it against model width. Larger $d_{\text{model}}$ supports a larger vocabulary before embedding cost dominates. Frontier-scale models sit at 128k–256k.
3. Handle numbers explicitly. Digit-level or fixed-group splitting.
4. Reserve special tokens generously. Adding them later means resizing the embedding, and running out mid-project forces a resize that invalidates every existing checkpoint.
5. Decide tying deliberately. Tied input and output embeddings halve the vocabulary parameter cost with modest quality effect at large scale; untied is standard at frontier scale where the parameters are affordable.
6. Train tokenizer and model on the same corpus to avoid glitch tokens.
7. Measure fertility across the deployment mix before committing. This is the last cheap moment to change it.

The tokenizer is the only architectural component that cannot be changed after training without starting over, which is precisely why an afternoon spent on it pays back across the model’s entire life. Give it the same scrutiny as depth and width and the wins are available before a single training step runs.

References