What Is Byte-Pair Encoding?
- 7 hours ago
- 37 min read
Every time you type a prompt into an AI chatbot, the model never actually reads your words. It reads numbers. Byte-Pair Encoding (BPE) is the quiet, unglamorous piece of engineering that turns your sentence into those numbers — and the specific way it does that job shapes how much a prompt costs, how fast a model responds, and even how fairly the model treats different languages.
TL;DR
Byte-Pair Encoding (BPE) began in 1994 as a general-purpose data-compression algorithm invented by Philip Gage, long before anyone used it for language modeling.[1]
In 2016, Sennrich, Haddow, and Birch adapted BPE into a subword tokenization method for neural machine translation, and it became the basis for how models like GPT-2 and RoBERTa split text into tokens.[2][3][8]
BPE training repeatedly merges the most frequent adjacent pair of symbols in a corpus until it reaches a target vocabulary size; encoding new text later applies those learned merge rules in order.
Byte-level BPE (used by GPT-2 and its successors) operates on raw byte values instead of Unicode characters, so it can represent any text without an unknown-token fallback.[3][6]
BPE is not the only subword method — WordPiece and Unigram language-model tokenization solve similar problems differently, and SentencePiece is a framework that can implement several of these algorithms, not a single competing algorithm.[5][7]
Tokenization is not neutral: because vocabularies are learned from training corpora, some languages get split into far more tokens than others, which affects cost and context usage.[12]
What Is Byte-Pair Encoding?
Byte-Pair Encoding (BPE) is a subword tokenization method that builds a vocabulary by repeatedly merging the most frequent pair of adjacent symbols in a training corpus. Originally a 1994 data-compression algorithm, BPE was adapted in 2016 for neural machine translation and now underlies tokenizers used by GPT-2, RoBERTa, and many other language models.
Table of Contents
Why Computers and Language Models Need Tokenization
Computers do not store or process letters the way people read them. Underneath every character is a number, and a machine learning model needs a fixed, finite list of possible inputs to work with — it cannot have a separate parameter for every conceivable word, because new words, misspellings, and made-up terms appear constantly. Tokenization is the preprocessing step that solves this: it breaks raw text into a manageable set of reusable pieces called tokens, each of which maps to a numeric ID the model can actually consume.
If a model used whole words as its vocabulary, it would need an entry for every word it might ever see, including brand names, typos, and words in other languages. That is an open-vocabulary problem, and it is unworkable at a fixed size. If a model used single characters, the vocabulary would stay small, but ordinary sentences would explode into very long sequences, which is expensive for a Transformer to process. Byte-Pair Encoding exists as a practical middle ground between these two extremes, and understanding it is the first step to understanding how any modern artificial intelligence language system actually reads your text.
What Is Byte-Pair Encoding? A Plain-English Definition
Byte-Pair Encoding is a method for breaking text into subword pieces by learning, from a body of text, which pairs of adjacent symbols occur together most often, and then treating those frequent pairs as single new units. It repeats this merging process again and again, each time creating a slightly larger reusable chunk, until it reaches a target vocabulary size or number of merges.
The result is a vocabulary that sits between full words and single characters: common words like "the" or "and" typically remain a single token, while rarer or longer words get split into a handful of familiar pieces, such as prefixes, suffixes, and common syllables. Because the vocabulary is built from data rather than from a fixed dictionary, BPE can represent virtually any input text, even words it has never explicitly seen, by falling back to smaller, more common fragments.
It is worth being precise from the outset: BPE the compression algorithm, BPE the subword tokenization method, and byte-level BPE are three related but distinct things, and conflating them is one of the most common sources of confusion in this space. The next three sections untangle them in order.
The Origin of BPE: A 1990s Data-Compression Algorithm
Byte-Pair Encoding did not begin in artificial intelligence research. It was introduced by Philip Gage in a February 1994 article, "A New Algorithm for Data Compression," published in The C Users Journal.[1] Gage's goal had nothing to do with language — he was solving a general-purpose data-compression problem, competing with algorithms like Lempel-Ziv-Welch (LZW).
The original algorithm works directly on bytes of arbitrary binary data. It scans a file for the most frequently occurring pair of adjacent bytes, replaces every occurrence of that pair with a single byte value that does not otherwise appear in the file, and writes a small substitution table recording what that replacement byte stands for. It repeats this process — find the most frequent pair, replace it, record the substitution — until no pair occurs often enough to be worth replacing, or until it runs out of unused byte values to use as replacements.[1]
Gage's own paper is explicit about the mechanics: the algorithm is fundamentally multi-pass, requires the data to be buffered in memory, and adapts "exponentially" because a replacement byte can itself become part of a later pair, allowing nested substitutions to collapse long repeated runs very efficiently. As his own worked example shows, the string ABABCABCD becomes XXCXCD after replacing AB with X, and then XYYD after replacing XC with Y — two merge steps collapsing a 9-byte string into 4 bytes plus a small table.[1] Gage reported that his method delivered compression roughly comparable to LZW on typical binary files, with the specific advantage of a very small, very fast decompression routine, at the cost of slower compression and a lower ratio on some file types.[1] This is the literal, original meaning of "byte pair" in Byte-Pair Encoding: two adjacent bytes, replaced by one.
From Compression to Language: How BPE Became a Subword Tokenizer
Gage's algorithm sat mostly in the compression literature until 2016, when Rico Sennrich, Barry Haddow, and Alexandra Birch, researchers at the University of Edinburgh, repurposed it for a very different problem: open-vocabulary neural machine translation.[2] Their paper, "Neural Machine Translation of Rare Words with Subword Units," observed that translation systems with fixed word vocabularies could not handle rare words, compound words, or names they had never seen, and that many such words are translatable if you break them into smaller, more common pieces — a name via transliteration, a compound via its parts, a loanword via shared roots.[2]
Their key adaptation kept Gage's core idea — repeatedly merge the most frequent adjacent pair — but changed what a "pair" was made of. Instead of merging raw bytes to shrink a binary file, they merged pairs of characters (or, after the first few merges, pairs of previously-created subword units) inside words from a text corpus, and they used the number of merge operations as a way to control the final vocabulary size rather than to minimize file size.[2] Sennrich and colleagues showed empirically that this subword segmentation improved translation quality over a dictionary-based back-off approach on WMT 2015 English–German and English–Russian tasks.[2] This is the paper that reframed BPE from a compression trick into a general-purpose subword tokenization algorithm, and it is the version of BPE that shows up throughout modern natural language processing, including in tokenizers documented by Hugging Face and used to pretrain models such as GPT-2.[3][6]
Essential Terminology Before You Go Further
Before working through how BPE training and encoding actually operate, it helps to fix the vocabulary this article uses, because these concepts build directly on one another.
Characters, bytes, and code points. A character is a unit of writing, like the letter "a" or the emoji "🙂". Computers store text as Unicode code points, and each code point is in turn encoded as one or more bytes — for example, common UTF-8 encodes ASCII characters as a single byte but many accented letters, non-Latin scripts, and emoji as two to four bytes. Whether a tokenizer works on code points or on raw bytes is one of the central distinctions in this article.
Symbols. During BPE training, a "symbol" is whatever the current unit of merging is — initially individual characters or bytes, and later, merged subword units.
Tokens, subwords, and vocabulary. A token is a single unit the tokenizer outputs; a subword is a token that represents part of a word rather than a whole word or a single character. The vocabulary is the complete, fixed list of tokens the tokenizer can produce, and the vocabulary size is how many entries that list has.
Token strings and token IDs. A token string is the human-readable text a token represents (for example, "ing"). A token ID is the integer the model actually receives, since neural networks operate on numbers, not text.
Pair frequency, merge operations, merge rules, and merge rank. Pair frequency is how often two adjacent symbols co-occur in the training corpus. A merge operation replaces every occurrence of the chosen pair with a new combined symbol. The ordered sequence of these operations is the merge rules or merge table, and each rule's position in that order is its merge rank — first-learned merges generally have priority over later ones during encoding, a point covered in depth further down.
Encoding, decoding, and detokenization. Encoding converts raw text into a sequence of tokens (and then token IDs). Decoding is the reverse: converting token IDs back into token strings. Detokenization additionally reassembles those token strings into properly formatted, human-readable text (getting spacing and punctuation right).
Reversibility and losslessness. A tokenizer is reversible, or lossless, if decoding always reconstructs the exact original input, byte for byte. This is a property of the tokenizer's implementation, not automatically true of every BPE variant.
Vocabulary size, reserved and special tokens, and unknown tokens. Vocabulary size is a design choice that trades off sequence length against embedding-table size. Special tokens (like sentence-boundary or padding markers) and reserved tokens are vocabulary slots set aside for structural purposes rather than for representing ordinary text. An unknown token (often <unk>) stands in for input the vocabulary cannot represent at all — a well-designed byte-level BPE tokenizer can avoid needing this token for ordinary text, since it always has raw bytes to fall back on.
Token fertility, compression ratio, context windows, and embedding tables. Token fertility is roughly how many tokens it takes, on average, to represent a fixed unit of text (such as a word) — higher fertility means more tokens for the same content. The compression ratio compares the size of the tokenized representation to the size of the original text. The context window is the maximum number of tokens a model can process at once. The embedding table (or embedding space) is the lookup matrix that converts each token ID into a vector the neural network can process; its size scales directly with vocabulary size.
Training-time versus inference-time tokenization. Training-time tokenization is the one-time process of learning a vocabulary and merge rules from a corpus. Inference-time (or encoding-time) tokenization is the repeated, deterministic process of applying those already-learned rules to new text every time the model is used. These are different processes, and conflating them is a common source of misunderstanding, addressed directly below.
How BPE Training Works, Step by Step
Training a BPE tokenizer is a one-time, offline process that produces two artifacts: a vocabulary of token strings and an ordered list of merge rules. The general procedure, as formalized by Sennrich and colleagues and implemented across current toolkits, looks like this:[2][6]
Gather a representative training corpus. The text used to learn the vocabulary should resemble the text the tokenizer will later encode — a tokenizer trained mostly on English news text will behave differently on source code or on a low-resource language.
Decide on text normalization. This might include Unicode normalization, case handling, or leaving text untouched; whatever is chosen must also be applied consistently at encoding time.
Decide on pre-tokenization. Most BPE implementations first split the corpus into rough word-like chunks (often using whitespace and punctuation boundaries) before learning merges within those chunks, rather than merging freely across an entire unsegmented text stream.[6]
Build the initial symbol set. Each pre-tokenized word is broken into its smallest units — individual characters for classic subword BPE, or individual bytes for byte-level BPE.
Count all adjacent symbol-pair frequencies across the entire corpus, respecting word boundaries.
Select the most frequent eligible pair. In the simplest formulation, this is just $\arg\max_{(x,y)} \text{count}(x, y)$ — the pair of adjacent symbols $x$ and $y$ that occurs most often anywhere in the corpus.
Merge that pair everywhere it occurs, creating one new symbol out of the two.
Record the merge, in order, in the merge table.
Update the pair-frequency counts to reflect the merge (removing the pairs that were consumed and adding any new pairs the merge created).
Repeat steps 6–9 until a stopping criterion is reached — most commonly, either a target number of merge operations or a target total vocabulary size.
Save the final vocabulary and the ordered merge table. These two files together are the trained tokenizer.
Two practical notes matter here. First, tie-breaking: real training corpora are large enough that exact ties in pair frequency are uncommon, but they do happen, especially in small or synthetic corpora, and different implementations resolve ties differently (for example, by insertion order or by a secondary sort key on the symbol pair itself) — there is no single universal rule, and this is one place where implementations genuinely diverge. Second, the final vocabulary size is not simply the number of merges: it equals the size of the initial symbol alphabet plus the number of merge operations performed, plus any special tokens reserved by the model.
A Complete Worked BPE Example
To make this concrete, here is a small, original training corpus, deliberately simplified for teaching purposes. It is not meant to represent realistic language statistics — production tokenizers train on many gigabytes of text — but every frequency and merge below has been worked out by hand and is internally consistent.
The corpus contains four words, each ending in a special end-of-word marker _ so the tokenizer can tell where one word stops and the next begins:
Word | Frequency in corpus |
sun_ | 5 |
sung_ | 2 |
sunny_ | 4 |
runny_ | 3 |
Initial symbols. Splitting into characters gives the starting alphabet: s, u, n, g, y, r, _.
Iteration 1. Counting every adjacent pair, weighted by word frequency:
Pair | Count |
(u, n) | 5 + 2 + 4 + 3 = 14 |
(s, u) | 5 + 2 + 4 = 11 |
(n, y) | 4 + 3 = 7 |
(y, _) | 4 + 3 = 7 |
(n, n) | 4 + 3 = 7 |
(n, _) | 5 |
(n, g) | 2 |
(g, _) | 2 |
(r, u) | 3 |
The most frequent pair is (u, n), with 14 occurrences. Merge it into a new symbol un. The corpus becomes: s un (5), s un g (2), s un n y (4), r un n y (3).
Iteration 2. Recount pairs on the updated corpus. The top pair is now (s, un) with 5 + 2 + 4 = 11 occurrences (runny_ does not start with s, so it is unaffected). Merge to sun. Result: sun (5), sun g (2), sun n y (4), r un n y (3).
Iteration 3. Recounting again, the top pair is (n, y) with 4 + 3 = 7 occurrences. Merge to ny. Result: sun (5), sun g (2), sun ny (4), r un ny (3).
Iteration 4. Recounting once more, the top pair is (ny, ) with 4 + 3 = 7 occurrences. Merge to ny. Result: sun (5), sun g (2), sun ny_ (4), r un ny_ (3).
Training could continue further, but four merges are enough to illustrate the process. The learned merge table, with rank in order of learning, is:
Rank | Merge | New token |
1 | u + n | un |
2 | s + un | sun |
3 | n + y | ny |
4 | ny + _ | ny_ |
Encoding new text with these rules. Encoding applies the merges in rank order, not by recounting frequencies in the new sentence.
A word seen during training, sun_: starting from s u n , apply rank 1 → s un ; apply rank 2 → sun . No further ranks apply. Output: sun, (2 tokens).
A related but unseen word, gunny_: starting from g u n n y , apply rank 1 (merge the first u,n) → g un n y ; rank 2 does not apply (no s); apply rank 3 (merge n,y) → g un ny ; apply rank 4 → g un ny. Output: g, un, ny_ (3 tokens).
An unusual, rare sequence, sunnnny_ (extra ns): starting from s u n n n n y , apply rank 1 (only one u,n pair exists) → s un n n n y ; apply rank 2 → sun n n n y ; apply rank 3 (merge the last n,y) → sun n n ny ; apply rank 4 → sun n n ny_. Output: sun, n, n, ny_ (4 tokens) — the extra, unusual repetition of n was not covered by any merge rule, so it falls back to individual characters, which is exactly the graceful degradation subword tokenization is designed to provide.
This is a pedagogical simplification of a process that, in production tokenizers, runs on billions of words, uses efficient priority-queue data structures to avoid recomputing every pair count from scratch, and typically stops after tens of thousands of merges rather than four.
How a Trained BPE Tokenizer Encodes New Text
It's worth restating a point implicit in the worked example above because it is widely misunderstood: encoding new text does not retrain the tokenizer, and it does not recompute "the most frequent pair in this sentence." The vocabulary and merge rules are frozen once training finishes. At inference time, the tokenizer:
Applies the same text normalization used during training.
Applies the same pre-tokenization scheme used during training.
Splits the input into its initial symbols (characters or bytes).
Applies the learned merge rules, generally trying merges in the order they were learned (lowest rank first), repeating until no further learned merge applies.
Produces a final sequence of token strings.
Maps each token string to its integer token ID using the fixed vocabulary lookup.
Adds any special tokens the specific model architecture requires (such as beginning- or end-of-sequence markers).
This is why the same input text, run through the same tokenizer twice, always produces the same tokens: the process is deterministic given a fixed vocabulary and merge table, in contrast to probabilistic subword segmentation methods discussed later in this article.
Merge Rules, Merge Order, and Merge Rank
The order in which merges are learned is not incidental bookkeeping — it is what makes encoding deterministic and consistent with training. Because later merges are typically built out of earlier ones (for example, sun in the worked example only exists because un was already merged), applying merges out of order, or in the wrong priority, can produce a different — and wrong — tokenization for the same input. Most implementations therefore apply eligible merges in ascending rank order: whichever learned merge has the lowest (earliest) rank among those currently possible in the string gets applied first, and the process repeats until no more learned merges apply to what remains. This is precisely why merge rank matters: it encodes not just what to merge, but the sequence in which those merges must be attempted so that longer, later-learned tokens are correctly built from shorter, earlier-learned ones.
Decoding and Detokenization
Decoding reverses encoding: given a sequence of token IDs, the tokenizer looks up each ID's corresponding token string in the vocabulary and concatenates them. For a subword vocabulary that used an end-of-word marker like the _ in the worked example above, decoding also needs to convert that marker back into an actual space or word boundary, and detokenization has to handle any additional formatting conventions of the specific tokenizer (for instance, GPT-2-style tokenizers mark the start of a new word with a special leading-space character rather than an end-of-word marker, so decoding reconstructs spacing from that marker instead).[6] Whether this whole pipeline is truly lossless — meaning decoding always reproduces the original text exactly, including unusual whitespace and punctuation — depends on the specific implementation's design choices, not on the fact that "it uses BPE."
Byte-Level BPE vs Character-Level BPE
This is one of the most important and most frequently blurred distinctions in the whole topic. Classic subword BPE, as introduced by Sennrich and colleagues, builds its initial alphabet out of Unicode characters.[2] The trouble is that a corpus containing many languages and scripts can have an enormous number of distinct base characters, and any character not seen during training still has to be handled somehow at encoding time — usually by falling back to an unknown-token symbol.
Byte-level BPE, the variant used by GPT-2 and adopted afterward by RoBERTa, sidesteps this by building the initial alphabet out of raw bytes instead of characters.[3][8] Since a byte can only take one of 256 values, the base vocabulary before any merges is fixed at exactly 256 symbols, and because every possible piece of UTF-8 encoded text is, by definition, some sequence of these 256 byte values, the tokenizer can represent any input text without ever needing an unknown token for ordinary text.[3][6] GPT-2's paper describes this directly as a way to combine "the empirical benefits of word-level [language models] with the generality of byte-level approaches," letting the model assign a probability to any Unicode string regardless of preprocessing choices.[3]
There is a subtlety worth being precise about, since it is a common source of confusion: the Hugging Face implementation of GPT-2's byte-level scheme does not feed raw byte values (0–255) directly into the BPE merge algorithm as opaque numbers. Instead, it first remaps each of the 256 possible byte values to a printable, visible Unicode character (using a fixed lookup table), and then runs the ordinary character-level BPE merge process over those remapped characters.[6] This means a single visible character you type — especially outside the basic Latin alphabet, such as an accented letter, an emoji, or a character in a non-Latin script — can correspond to more than one byte, and therefore to more than one initial unit, before any merges are applied, because UTF-8 encodes such characters using multiple bytes. This is precisely why it is inaccurate to claim that "every tokenizer called BPE works directly on raw bytes" or that byte-level BPE guarantees one token per visible character — the reality is a byte-to-visible-character remapping layer followed by ordinary BPE merges on top of it.[6]
Unicode, Bytes, Spaces, and Pre-Tokenization
Text on a computer is stored as a sequence of Unicode code points, and those code points are, in turn, encoded as one or more bytes depending on the encoding scheme (UTF-8 is by far the most common on the web and in modern NLP pipelines). A plain ASCII letter takes one byte in UTF-8; many accented Latin letters, Cyrillic, Greek, and Hebrew characters take two bytes; most Chinese, Japanese, and Korean characters take three bytes; and many emoji take four bytes. This matters directly for tokenization because a byte-level tokenizer's very first step — before any merges happen — is determined by this byte layout, not by how many visible characters a human reader would count.
Spaces deserve special mention because different tokenizer families treat them differently. GPT-2 and RoBERTa's byte-level BPE tokenizers fold the preceding space into the following token, so the same word is represented by a different token depending on whether it starts a sentence or follows a space — this is why the GPT-2 tokenizer's pre-tokenizer replaces spaces with a special marker character before running the merge algorithm.[6][8] Other tokenizer families, including several used with SentencePiece, instead treat the space itself as an ordinary symbol to be merged like any other character.[5]
Pre-tokenization — the step of first splitting raw text into rough word-like chunks before learning or applying merges — also varies substantially by implementation. GPT-2's tokenizer uses a specific regular expression to separate letters, numbers, and punctuation into different chunks before applying byte-level BPE within each chunk, while other systems use whitespace splitting, or language-specific segmenters such as the Moses tokenizer for XLM and FlauBERT, or spaCy and ftfy for the original GPT model.[6] Because pre-tokenization rules differ across tokenizers, the same input sentence can be split at a different starting point even before any merge rules are applied, which is one reason two different BPE-based tokenizers rarely produce identical token boundaries for the same sentence.
BPE's Role in the Large Language Model Pipeline
Tokenization is one stage in a longer pipeline, and it is useful to see where it sits: raw input text is normalized and pre-tokenized, then tokenized into subword pieces, then those pieces are mapped to token IDs, then each token ID is looked up in an embedding table to produce a vector, then the sequence of vectors is processed by a transformer network (which relies on positional information, commonly supplied through positional encoding), and the model's output is a probability distribution over predicted next token IDs, which are then decoded back into text.
It is important to be precise about what BPE contributes to this pipeline and what it does not. Byte-Pair Encoding does not understand grammar, meaning, or context — it does not "think" or "know" anything about language. It is a fixed, deterministic (in its standard, non-regularized form) preprocessing step that decides how to chop text into reusable pieces; all of the actual language understanding happens later, in the transformer's learned parameters, not in the tokenizer.
How Tokenization Affects Context Length, Cost, Latency, and Memory
Because a model's context window is measured in tokens, not characters or words, how a tokenizer segments text has direct, practical consequences.
Context-window consumption and input/output token counts. A piece of text that a byte-level BPE tokenizer splits into fewer tokens leaves more of the context window available for the rest of a conversation or document, and a text that gets split into more tokens uses up context faster, which can force earlier truncation.
API billing. Providers that charge per token are, in effect, charging based on how the tokenizer segments your specific input, and identical content in two different languages, or in prose versus code, can produce noticeably different token counts and therefore different costs.
Latency and memory. Transformer-based architectures generally scale in compute with the square of sequence length for their attention mechanism, so a longer token sequence for the same content directly increases both processing latency and memory use.
Embedding and output-layer size. Vocabulary size is a direct multiplier on the size of the embedding lookup table and the final output projection layer, so choosing a larger vocabulary is not free — it is a genuine hyperparameter trade-off between shorter sequences and a larger model.
Any specific "characters per token" ratio you might hear — such as tiktoken's documented observation that, on average, each token corresponds to about four bytes of typical English text — is empirically true only for the specific tokenizer, language, and text domain measured, and should never be treated as a universal constant across languages or content types.[4]
BPE Behavior on Rare Words, Code, Numbers, Emojis, and Multilingual Text
Because subword vocabularies are learned from a training corpus, how gracefully BPE handles unusual input depends heavily on how well that input's patterns were represented during training.
Rare words and misspellings get split into smaller, more common fragments rather than mapped to an unknown token, since even an unfamiliar word is usually built from familiar character sequences — this is precisely the property that motivated Sennrich and colleagues' original adaptation of BPE for machine translation.[2]
Code often tokenizes less efficiently than prose from a comparable training domain, because identifiers, indentation, and symbol-heavy syntax do not match the frequency patterns learned from natural-language corpora, unless the tokenizer was specifically trained on a corpus that included a meaningful share of source code.
Numbers are a known trouble spot: depending on merge rules learned, the same numeral sequence can be split inconsistently (for example, some three-digit sequences forming a single token while others split into two), which can make arithmetic and numeric reasoning harder for a downstream model, since the tokenization does not respect place value.
Emoji and non-Latin scripts are handled very differently depending on whether the tokenizer is byte-level or character-level. A byte-level tokenizer can represent any emoji or script without a fallback token, though a script rarely seen in training may still be split into many small byte-level tokens rather than efficient, larger merged units — a point developed further in the multilingual section below.
Morphologically rich languages, where a single word can carry many grammatical suffixes (as in Turkish or Finnish), tend to produce longer token sequences for equivalent meaning compared to English, because a frequency-driven algorithm trained mostly on English text will not have learned merges for those languages' morphological patterns.[6]
BPE Compared With WordPiece, Unigram, SentencePiece, Word, Character, and Byte Approaches
Approach | Basic unit | Vocabulary construction | Encoding procedure | Unknown-text handling | Typical / historic use |
Word-level | Whole words | Frequency list from corpus | Direct lookup | Falls back to <unk> for unseen words | Early neural language models |
Character-level | Individual characters | Fixed, small alphabet | Direct character lookup | Naturally covers any text in that alphabet | Some early sequence models; robust to noise |
Raw-byte / byte-level (no merges) | Bytes (0–255) | Fixed 256-symbol alphabet | Direct byte lookup | Fully covers any text | ByT5, CANINE-style token-free models[11] |
BPE (classic, character-based) | Characters, then merged subwords | Iterative greedy merging of the most frequent adjacent pair | Apply learned merges in rank order | Can require <unk> for unseen base characters | Original NMT subword models[2] |
Byte-level BPE | Bytes, remapped to visible characters, then merged | Same greedy merge process, starting from a 256-symbol byte alphabet | Apply learned merges in rank order | No <unk> needed for ordinary text | GPT-2, GPT-3-era models, RoBERTa[3][8] |
WordPiece | Characters, then merged subwords | Iterative merging chosen by maximizing training-corpus likelihood, not raw frequency | Longest-match-style segmentation against the learned vocabulary | Falls back to <unk> for unrepresentable spans | BERT, DistilBERT[7] |
Unigram language model | Subword units scored probabilistically | Starts from a large candidate set and prunes to a target size using a probabilistic unigram model | Finds the most probable segmentation (e.g., via Viterbi) | Rare, since candidate set is broad | T5, ALBERT, several SentencePiece-based models[9] |
Two clarifications the table cannot fully capture on its own. First, SentencePiece is a tokenizer framework and toolkit, not a single algorithm — it can be configured to run either the BPE algorithm or the Unigram language-model algorithm underneath, and its key design contribution is that it trains directly from raw sentences rather than requiring pre-tokenized, whitespace-separated input, which makes it language-independent in a way that whitespace-based pre-tokenization is not.[5] Saying "SentencePiece" and "BPE" are two separate, mutually exclusive things is therefore a category error; SentencePiece is more accurately described as one popular implementation vehicle that can house BPE.[5]
Second, the difference between BPE and WordPiece is not merely cosmetic. Both start from a character-level vocabulary and grow it through iterative merges, but where BPE selects the pair with the single greatest raw frequency, WordPiece instead selects the pair that most increases the likelihood of the training corpus once merged — informally, it favors merges that make the training data more probable under a simple language model, not merely merges that are common.[7] At encoding time, this can also produce different segmentation behavior for out-of-vocabulary spans, since WordPiece implementations typically look for the longest matching vocabulary piece rather than strictly replaying merge history in learned order.
Advantages of Byte-Pair Encoding
BPE has become the default choice across a large share of modern language models for several concrete reasons. It produces a fixed-size vocabulary that the model architecture can be built around in advance, unlike an open-ended word list. It naturally reuses common substrings — a prefix like "un-" or a suffix like "-ing" is learned once and reused across many words, giving the model repeated exposure to the same meaningful unit in different contexts.[6] It handles rare and unseen words far better than a rigid word-level vocabulary, since almost any string can be decomposed into smaller learned pieces rather than mapped to a single unknown-token symbol.[2] It offers a genuine compromise between word-level and character-level modeling, keeping sequences shorter than character-level tokenization while staying far more flexible than word-level tokenization. In its standard, non-regularized configuration, encoding is deterministic — the same input always produces the same output, which is valuable for reproducibility and for API billing predictability. It is also efficient to implement, since merge counting and application can be done with well-understood data structures at scale. Byte-level BPE specifically achieves near-open-vocabulary coverage, representing effectively any text without an unknown-token fallback, and when implemented carefully, byte-level BPE tokenizers are also fully reversible, reconstructing the exact original byte sequence on decoding.[3][6]
Limitations and Failure Modes
BPE's advantages come with real trade-offs, and a fair account of the method has to include them. Frequency-driven merges do not equal linguistic meaning: the pieces BPE learns are statistically common substrings, not necessarily true morphemes, and research such as Bostrom and Durrett's evaluation of BPE for language-model pretraining has directly examined whether alternative segmentations that better respect morphology can outperform BPE's frequency-only criterion.[3] Because vocabularies are learned from a specific training corpus, the resulting tokenizer is corpus-dependent and domain-dependent — a tokenizer trained on general web text will not segment legal, medical, or code text as efficiently as one trained on those domains specifically. Tokenization is also sensitive to normalization and pre-tokenization choices made before training, so two teams following slightly different preprocessing conventions can end up with meaningfully different tokenizers even from similar training data.
There is an unavoidable trade-off in vocabulary-size allocation: every slot given to one kind of pattern (say, English-specific merges) is a slot not available for another (say, another language's morphology), and this allocation is fixed once training completes. This produces documented unequal efficiency across languages and scripts, discussed in depth in the next section. Rare word forms, unusual spellings, and typos can fragment into many small tokens, since the merges that would compress them were never learned, and long token sequences for underrepresented text categories raise both compute cost and effective context-window consumption for the same content. Once a model has been trained against a specific tokenizer, that tokenizer becomes difficult to change without retraining or otherwise adapting the embedding table, since the model's learned parameters are tied to the specific vocabulary indices it was trained with. Finally, it is worth stressing that compression efficiency and downstream task quality are correlated in general discussion, but not perfectly determined by one another — a tokenizer that produces shorter sequences is not automatically the one that yields the best model performance on a given task, and claims of a simple causal relationship between the two should be treated with caution.
Multilingual Efficiency and Fairness
Because a BPE vocabulary is learned from a training corpus, and because most large corpora used to train widely deployed tokenizers are disproportionately composed of English and other high-resource languages, the resulting vocabulary tends to contain many efficient, larger merged tokens for those languages and comparatively fewer for underrepresented languages. Petrov, La Malfa, Torr, and Bibi's 2023 NeurIPS study measured this directly: the same content translated into different languages can require drastically different numbers of tokens to encode, with the researchers reporting differences of up to roughly fifteen times between languages for some tokenizers, and noting that these disparities persist even in tokenizers that were intentionally trained to support many languages.[12]
This matters in concrete, practical ways. Higher token fertility for a given language — needing more tokens to express the same amount of content — means that language consumes more of a fixed context window and, where billing is per token, costs more for equivalent content.[12] The underlying causes are a mix of factors: corpus composition (how much text from each language went into training), script differences (some writing systems require more bytes per character in UTF-8, which affects byte-level tokenizers specifically), and morphological complexity (languages with rich inflectional systems produce more distinct word forms for a segmentation algorithm to try to compress).
It is important to state this carefully rather than overclaim it: byte-level coverage guarantees that any script can be represented, but it does not guarantee equally efficient representation — a script that is rare in the training corpus may still end up broken into many small, individually meaningful byte-level tokens rather than being covered by larger, efficient merges, precisely because merges are learned by frequency and the language was underrepresented.[6][12] Normalization choices interact with this too, since scripts with combining characters or multiple valid Unicode representations for visually identical text can end up tokenized inconsistently unless normalization is applied uniformly. None of this means that a single fertility number fully determines a model's downstream quality in a given language — quality depends on far more than tokenization alone — but the tokenization-stage disparity itself is well documented and worth factoring into any decision about which languages a deployed system needs to serve well.
Building a BPE Tokenizer From Scratch in Python
The following is a small, self-contained implementation using only Python's standard library. It trains merge rules on the same toy corpus used in the worked example above, and then encodes new text with those learned rules. It is written for clarity, not speed.
from collections import defaultdict
# --- Training data: the same toy corpus used in the worked example ---
# Each word is represented as a tuple of single-character symbols,
# with "_" used as an explicit end-of-word marker.
corpus = {
("s", "u", "n", "_"): 5,
("s", "u", "n", "g", "_"): 2,
("s", "u", "n", "n", "y", "_"): 4,
("r", "u", "n", "n", "y", "_"): 3,
}
def get_pair_counts(word_freqs):
"""Count frequency of every adjacent symbol pair across the corpus."""
pair_counts = defaultdict(int)
for symbols, freq in word_freqs.items():
for i in range(len(symbols) - 1):
pair = (symbols[i], symbols[i + 1])
pair_counts[pair] += freq
return pair_counts
def merge_pair(pair_to_merge, word_freqs):
"""Replace every occurrence of pair_to_merge with a single merged symbol."""
new_word_freqs = {}
merged_symbol = pair_to_merge[0] + pair_to_merge[1]
for symbols, freq in word_freqs.items():
new_symbols = []
i = 0
while i < len(symbols):
# If the current and next symbol match the pair, merge them.
if (
i < len(symbols) - 1
and symbols[i] == pair_to_merge[0]
and symbols[i + 1] == pair_to_merge[1]
):
new_symbols.append(merged_symbol)
i += 2
else:
new_symbols.append(symbols[i])
i += 1
new_word_freqs[tuple(new_symbols)] = freq
return new_word_freqs
def train_bpe(word_freqs, num_merges):
"""Learn an ordered list of merge rules from a training corpus."""
word_freqs = dict(word_freqs)
merge_rules = [] # ordered list of (symbol_a, symbol_b) tuples
for _ in range(num_merges):
pair_counts = get_pair_counts(word_freqs)
if not pair_counts:
break
# Select the most frequent pair (ties broken by Python's stable max,
# which returns the first maximal pair encountered).
best_pair = max(pair_counts, key=pair_counts.get)
if pair_counts[best_pair] < 2:
break # stop if no pair repeats meaningfully
word_freqs = merge_pair(best_pair, word_freqs)
merge_rules.append(best_pair)
return merge_rules
def encode(word, merge_rules):
"""Apply learned merge rules, in rank order, to tokenize a new word."""
symbols = list(word) + ["_"]
for pair_to_merge in merge_rules:
merged_symbol = pair_to_merge[0] + pair_to_merge[1]
new_symbols = []
i = 0
while i < len(symbols):
if (
i < len(symbols) - 1
and symbols[i] == pair_to_merge[0]
and symbols[i + 1] == pair_to_merge[1]
):
new_symbols.append(merged_symbol)
i += 2
else:
new_symbols.append(symbols[i])
i += 1
symbols = new_symbols
return symbols
if __name__ == "__main__":
learned_merges = train_bpe(corpus, num_merges=4)
print("Learned merge rules, in rank order:")
for rank, rule in enumerate(learned_merges, start=1):
print(f" Rank {rank}: {rule[0]} + {rule[1]} -> {rule[0]}{rule[1]}")
for test_word in ["sun", "gunny", "sunnnny"]:
tokens = encode(test_word, learned_merges)
print(f"'{test_word}' -> {tokens}")Running this script reproduces exactly the merge table and encoding results from the worked example section above: the four learned rules u+n, s+un, n+y, ny+_, and token sequences ['sun', '_'] for sun, ['g', 'un', 'ny_'] for gunny, and ['sun', 'n', 'n', 'ny_'] for sunnnny.
What this implementation deliberately omits, and why production tokenizers are more complex:
Byte-level encoding. This example merges characters, not bytes, so it does not have the full-coverage guarantee byte-level BPE provides; a production implementation would first map raw UTF-8 bytes to a working symbol alphabet.[3][6]
Efficient pair-count updates. This code recomputes all pair counts from scratch after every merge, which is simple but slow; production implementations use a priority queue and update only the counts affected by each merge, which is essential for training on real-world corpora.
Robust pre-tokenization and normalization. Real tokenizers apply careful, often regex-based, pre-tokenization and Unicode normalization before training or encoding begins, none of which is handled here.
Special tokens. No reserved tokens (padding, sequence boundaries, or unknown-token fallback) are implemented.
Serialization. A production tokenizer saves its vocabulary and merge table to disk in a defined format so it can be loaded quickly and consistently across training and deployment, which this teaching example does not do.
For a very short look at how a production-grade library exposes the same underlying idea, OpenAI's tiktoken library lets you load an existing trained tokenizer and encode text in a couple of lines, without any of the training logic above, since training already happened once, offline, when the vocabulary was built.[4]
How Practitioners Choose or Train a Tokenizer
Selecting or training a tokenizer is a decision that becomes very difficult to reverse once a model has been trained around it, so it deserves deliberate evaluation rather than a default choice. Key considerations include:
Corpus representativeness and quality. The training corpus should reflect the languages, domains, and text types (prose, code, formatted data) the deployed model actually needs to handle well, and should be deduplicated to avoid over-weighting repeated content.
Normalization and case handling. Decide up front whether to lowercase text, how to handle accented characters, and how to normalize Unicode — and apply the same choices consistently at both model training time and inference time.
Whitespace handling and domain coverage. If the deployed system needs to process source code, ensure the training corpus includes enough of it; the same applies to any specialized or technical vocabulary the system must handle efficiently.
Vocabulary size. This is a genuine hyperparameter trade-off between average sequence length and embedding-table size, not a setting with one universally correct value.
Special-token and reserved-capacity planning. Decide in advance how many vocabulary slots to reserve for structural tokens and for any future additions, since expanding a vocabulary after training is disruptive.
Evaluation methodology. Before committing to a tokenizer, measure it on held-out data drawn from a representative test set or validation set: check token fertility (average tokens per word or per fixed unit of text) across the languages and domains that matter, run round-trip decoding tests to confirm losslessness, and measure the unknown-token rate on realistic inputs where applicable.
Speed, memory, and architectural compatibility. Confirm that the tokenizer's runtime performance and vocabulary size fit the constraints of the model architecture and the deployment environment.
Stability and versioning. Because the tokenizer becomes part of the trained model's interface — its output indices are baked into what the embedding table learned — changing the tokenizer after training generally requires retraining or carefully re-aligning the embedding table, so tokenizer choices should be versioned and treated as a durable architectural decision rather than a casual configuration setting.
Common Misconceptions About BPE
"A token is the same thing as a word."
Not necessarily — a token can be a whole word, part of a word, a single character, punctuation, or in byte-level tokenizers, even part of a single visible character.
"Byte-pair means every token contains exactly two bytes."
No — after the first merge, a "pair" being merged is a pair of symbols, and each symbol can already be a multi-byte or multi-character unit from an earlier merge; the name describes the merging mechanism, not a fixed output size.
"BPE always operates directly on bytes."
No — classic subword BPE, as introduced by Sennrich and colleagues, operates on characters; only byte-level BPE variants, such as those used in GPT-2 and RoBERTa, operate on byte-derived symbols.[2][3]
"BPE discovers true morphemes."
Not reliably — BPE merges are chosen by statistical frequency, not by linguistic analysis, so learned pieces frequently do not align with genuine prefixes, roots, and suffixes as a linguist would define them.
"The tokenizer chooses merges by frequency every time a user enters a prompt."
No — merge selection by frequency happens only once, during training; encoding new text at inference time simply applies the already-fixed, already-ranked merge rules.
"A larger vocabulary is always better."
No — vocabulary size is a genuine trade-off between shorter sequences and a larger embedding table and output layer, and the optimal size depends on the model's scale and target domains.
"Byte-level BPE gives every language equal tokenization efficiency."
No — byte-level coverage guarantees representability, not equal efficiency; documented research shows large fertility differences across languages even with byte-level and multilingual tokenizers.[12]
"Any text that looks identical must have the same underlying byte sequence."
Not always — Unicode allows some visually identical text to be represented by different code point sequences (for example, through combining characters), which is why normalization choices matter.
"SentencePiece and BPE are mutually exclusive."
No — SentencePiece is a tokenizer framework that can implement the BPE algorithm (or the Unigram algorithm) internally; it is not a rival algorithm to BPE.[5]
"Every GPT-style model uses exactly the same tokenizer."
No — tokenizer vocabularies and merge tables are trained separately for different model families and versions, and even within one company's model lineup, tokenizers can differ across generations.
"Tokenization is merely cosmetic preprocessing."
No — tokenization choices directly affect context-window usage, per-request cost, latency, and measurable disparities in efficiency across languages, as documented above.[12][14]
"Lossless decoding means the tokenizer preserves linguistic meaning."
No — losslessness only means the original byte sequence can be exactly reconstructed; it says nothing about whether the intermediate token boundaries align with anything meaningful linguistically.
Alternatives and the Future of Tokenization
BPE is not the only path to subword segmentation, and ongoing research continues to explore whether tokenization is even necessary at all. Taku Kudo's Unigram language-model approach, and the subword regularization technique built on it, starts from a large candidate vocabulary and iteratively removes the least useful pieces according to a probabilistic model, and additionally allows sampling multiple valid segmentations of the same text during training rather than always using one fixed, deterministic split — a technique shown to improve translation robustness by exposing a model to varied segmentations of the same input.[9] Because standard BPE's deterministic merge order doesn't naturally support this kind of sampling, Provilkov, Emelianenko, and Voita proposed BPE-Dropout, which achieves a similar regularization effect by randomly dropping some merge operations during training-time encoding, producing varied segmentations while keeping the simplicity of ordinary BPE merge rules.[10]
At the far end of the spectrum, researchers have explored removing subword tokenization altogether. Xue and colleagues' ByT5 model processes raw UTF-8 bytes directly, with no separate tokenizer, no fixed byte-pair vocabulary beyond individual bytes, and no external vocabulary file to keep in sync with the model; the trade-off is longer input sequences (bytes rather than subword tokens) in exchange for eliminating tokenizer-related preprocessing complexity and improving robustness to spelling variation and noisy text.[11] Whether token-free, byte-level approaches like ByT5 gradually displace subword methods such as BPE, WordPiece, and Unigram, or whether they remain a specialized option for particular robustness-sensitive tasks, is an open and active area of research rather than a settled question — and it is one worth watching if you build systems that depend on how text becomes numbers.
FAQ
What is Byte-Pair Encoding in simple terms?
Byte-Pair Encoding is a way of breaking text into smaller, reusable pieces called tokens by repeatedly combining whichever pair of adjacent symbols shows up most often in a body of training text. It started as a 1994 data-compression algorithm and was later adapted into a subword tokenization method widely used to prepare text for language models.[1][2]
Why was BPE originally created?
Philip Gage created BPE in 1994 to solve a general data-compression problem: shrinking arbitrary binary files by replacing frequently occurring byte pairs with single unused bytes, competing with algorithms like LZW on speed and memory use rather than on language modeling.[1]
Why is BPE used for tokenization in language models?
BPE gives language models a fixed-size, manageable vocabulary while still being able to represent virtually any input, including rare or unseen words, by decomposing them into smaller, more common subword pieces instead of falling back to a single unknown-token symbol.[2][3]
How is a BPE vocabulary trained?
Training starts from an initial alphabet of characters or bytes, counts every adjacent pair in a training corpus, repeatedly merges the single most frequent pair into a new symbol, and records each merge in order until a target vocabulary size or number of merges is reached.
How does a trained BPE tokenizer encode new text?
It applies the already-learned merge rules, in the order they were learned, to the new text's initial symbols — it does not recompute frequencies or retrain on the new sentence, which is why encoding is fast and deterministic.
What is the difference between ordinary BPE and byte-level BPE?
Ordinary subword BPE builds its starting alphabet from Unicode characters, which can require an unknown-token fallback for unseen characters. Byte-level BPE instead starts from a fixed 256-value byte alphabet, so it can represent any UTF-8 text without needing an unknown token.[3][6]
How do merge rules and merge rank work?
Each merge rule records which pair of symbols to combine and what new symbol to create; its merge rank is its position in that ordered list. Encoding applies eligible learned merges in ascending rank order so that later, larger tokens are correctly built from earlier, smaller ones.
How are tokens converted into token IDs?
After text is split into token strings using the learned merge rules, each token string is looked up in the tokenizer's fixed vocabulary table, which maps it to a unique integer ID — the actual number the neural network receives as input.
Why do different tokenizers split the same text differently?
Different tokenizers are trained on different corpora, use different pre-tokenization and normalization rules, and sometimes use different underlying algorithms (BPE, WordPiece, or Unigram), so their learned vocabularies and merge behavior diverge even on identical input text.
How does tokenization affect API cost and context length?
Since providers typically bill per token and context windows are measured in tokens, text that a given tokenizer splits into more tokens consumes more of the context window and costs more, even if the underlying content is identical to a more efficiently tokenized version.
What are BPE's main strengths and weaknesses?
Strengths include a fixed, manageable vocabulary size, reuse of common substrings, and graceful handling of rare words. Weaknesses include corpus- and domain-dependence, frequency-based merges that don't necessarily reflect true linguistic structure, and documented disparities in tokenization efficiency across languages.[12]
How does BPE compare with WordPiece and Unigram?
BPE selects merges purely by raw pair frequency; WordPiece selects merges by how much they increase the training corpus's likelihood; and Unigram starts from a large candidate vocabulary and prunes it down using a probabilistic scoring model rather than growing it through merges.[7][9]
Is SentencePiece a different algorithm from BPE?
No — SentencePiece is a tokenizer framework and toolkit that can implement either the BPE algorithm or the Unigram algorithm internally, and it is notable for training directly from raw sentences rather than requiring pre-tokenized, whitespace-separated text.[5]
Do all large language models use the same tokenizer?
No — different model families, and often different versions within the same family, train separate tokenizer vocabularies and merge tables suited to their own training data and design goals, so token counts for identical text can differ noticeably between models.
Can I build a basic BPE tokenizer myself?
Yes — the core training loop (count pairs, merge the most frequent one, repeat) and the core encoding loop (apply learned merges in rank order) can both be implemented in a small amount of standard-library Python, as shown in this article's from-scratch example, though production tokenizers add byte-level encoding, efficient pair-count updates, and robust special-token handling on top of that core idea.
Key Takeaways
Byte-Pair Encoding began in 1994 as Philip Gage's general-purpose data-compression algorithm, unrelated to language modeling.[1]
Sennrich, Haddow, and Birch adapted BPE into a subword tokenization method for neural machine translation in 2016, and it has since become foundational to how many language models process text.[2]
BPE training repeatedly merges the most frequent adjacent symbol pair in a corpus until a target vocabulary size is reached; encoding new text applies those learned merge rules in order rather than retraining.
Byte-level BPE, used by GPT-2 and RoBERTa, starts from a fixed 256-byte alphabet, letting it represent virtually any text without an unknown-token fallback.[3][6][8]
SentencePiece is a framework capable of implementing BPE or Unigram, not a separate rival algorithm to either.[5]
WordPiece differs from BPE in how it selects merges — by corpus likelihood rather than raw frequency — and Unigram differs by pruning a large candidate vocabulary rather than growing a small one.[7][9]
Tokenization has measurable, documented disparities across languages, with some languages requiring many more tokens than others for equivalent content.[12]
Because a tokenizer becomes part of a trained model's interface, it is a durable architectural decision, not a casual setting to revisit later.
Actionable Next Steps
Inspect how your own text tokenizes using an official tokenizer tool (such as OpenAI's tiktoken) before assuming a "characters per token" rule applies to your content.[4]
If you are evaluating or selecting a tokenizer for a project, measure token fertility and round-trip decoding on a representative sample of your actual target languages and content types, not just English prose.
If your application serves multiple languages, explicitly test for and account for tokenization-efficiency disparities rather than assuming byte-level coverage implies equal treatment.[12]
Try implementing the small from-scratch Python BPE example in this article yourself, then compare its output against a production tokenizer library on the same input text to see where the simplified version diverges.
Before changing a deployed model's tokenizer, plan for the fact that this generally requires retraining or carefully re-aligning the embedding table, since the two are tightly coupled.
Glossary
Byte: A unit of digital storage representing one of 256 possible values, used to encode Unicode text via schemes like UTF-8.
Byte-level BPE: A BPE variant whose initial alphabet is built from byte values rather than characters, enabling full text coverage without an unknown token.
Code point: A numeric value assigned to a specific character in the Unicode standard.
Compression ratio: A measure comparing the size of tokenized or compressed output to the size of the original input.
Context window: The maximum number of tokens a model can process in a single input at once.
Decoding: Converting token IDs back into token strings.
Detokenization: Reassembling decoded token strings into properly formatted, human-readable text.
Embedding table: The lookup matrix that converts each token ID into a numeric vector for a neural network to process; see also embedding space.
Encoding: Converting raw text into a sequence of tokens and then token IDs.
Hyperparameter: A configuration choice, such as vocabulary size, set before training rather than learned from data; see hyperparameter tuning.
Merge operation: A single step in BPE training that replaces every occurrence of a chosen symbol pair with one new combined symbol.
Merge rank: The position of a specific merge rule within the ordered list of all learned merges.
Merge rules (merge table): The full ordered list of merge operations learned during BPE training.
Normalization: Standardizing text (case, Unicode form, etc.) before tokenization, applied consistently at both training and encoding time.
Out-of-vocabulary (OOV): Text that a tokenizer's vocabulary cannot directly represent as a whole unit.
Pair frequency: How often a specific pair of adjacent symbols occurs across a training corpus.
Pre-tokenization: The step of splitting raw text into rough word-like chunks before applying tokenization merges.
Reserved / special tokens: Vocabulary slots set aside for structural purposes, such as sequence boundaries, rather than for ordinary text content.
Reversibility / losslessness: A tokenizer's ability to reconstruct the exact original text from its token sequence.
SentencePiece: A language-independent tokenizer framework that can implement either the BPE or Unigram algorithm, notable for training directly from raw sentences.[5]
Subword: A token representing part of a word rather than a whole word or a single character; see subword tokenization.
Symbol: The current unit being merged during BPE training — initially a character or byte, later a merged unit.
Token: A single unit of output from a tokenizer.
Token fertility: A measure of how many tokens, on average, are needed to represent a fixed unit of text.
Token ID: The integer a token string is mapped to for input into a neural network.
Tokenization: The overall process of converting raw text into tokens.
Training-time tokenization: The one-time process of learning a tokenizer's vocabulary and merge rules from a corpus.
Inference-time tokenization: The repeated process of applying an already-trained tokenizer's rules to new text.
Unicode: The international standard assigning a unique code point to virtually every character used in written language.
Unigram language-model tokenization: A subword method that prunes a large candidate vocabulary down to a target size using probabilistic scoring, rather than growing it through merges.[9]
Unknown token (<unk>): A placeholder token representing input text the vocabulary cannot otherwise represent.
Vocabulary: The complete, fixed set of tokens a trained tokenizer can produce.
Vocabulary size: The total number of distinct entries in a tokenizer's vocabulary.
WordPiece: A subword tokenization algorithm, used by BERT, that selects merges based on corpus likelihood rather than raw pair frequency.[7]
Sources & References
Gage, P. (1994). A New Algorithm for Data Compression. The C Users Journal, 12(2), 23–38. http://www.pennelynn.com/Documents/CUJ/HTML/94HTML/19940045.HTM
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). https://aclanthology.org/P16-1162/
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
OpenAI (2024–2026). tiktoken (official GitHub repository, README and _educational.py). https://github.com/openai/tiktoken
Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP). https://aclanthology.org/D18-2012/
Hugging Face (2024–2026). Byte-Pair Encoding tokenization, LLM Course, Chapter 6. https://huggingface.co/learn/llm-course/en/chapter6/5
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805. https://arxiv.org/abs/1810.04805
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692. https://arxiv.org/abs/1907.11692
Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. arXiv:1804.10959. https://arxiv.org/abs/1804.10959
Provilkov, I., Emelianenko, D., & Voita, E. (2020). BPE-Dropout: Simple and Effective Subword Regularization. arXiv:1910.13267. https://arxiv.org/abs/1910.13267
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10, 291–306. arXiv:2105.13626. https://arxiv.org/abs/2105.13626
Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://arxiv.org/abs/2305.15425
Hugging Face (2024–2026). Pre-tokenizers: ByteLevel, Tokenizers documentation. https://huggingface.co/docs/tokenizers/en/api/pre-tokenizers
Hugging Face (2024–2026). Tokenizer summary / Tokenization algorithms. https://huggingface.co/docs/transformers/tokenizer_summary