Byte-pair encoding (BPE)

BPE is the heart of tokenization. It is the gold-standard algorithm trusted by some of the most powerful language models ever built: GPT-2, GPT-3, and most modern LLMs rely on it to convert raw text into tokens. But what makes it special?

To understand that, we need to look at the core problems any tokenization algorithm must solve.

The first is efficiency: how do we convert text into tokens in a way that is fast and practical at scale? The second, and more interesting, challenge is handling unknown words: what happens when the model encounters a word it has never seen during training?

This is where BPE excels. It was designed with both constraints in mind, and the elegance of its solution is what made it the industry standard.

Byte-pair encoding (BPE) has two phases: first it builds a vocabulary from a training corpus, then it uses that vocabulary to encode new text. Let us walk through each.

Building the vocabulary (training)

BPE builds its vocabulary by starting from the smallest possible units, individual characters, and learning to merge them upward.

Here is how it works, step by step:

1. Split every word in the training corpus into individual characters.
2. Scan the entire corpus and ask one question: which two adjacent units appear together most often?
3. Merge that most frequent pair into a single new token and add it to the vocabulary.
4. Repeat. Every iteration merges the next most frequent pair.
5. Stop once the vocabulary reaches a predefined size.

Example: watching token and tokenization merge

Suppose our training corpus contains just two words:

• token, appearing 6 times
• tokenization, appearing 4 times

BPE starts by splitting everything into individual characters:

t o k e n                (×6)
t o k e n i z a t i o n  (×4)

Then it asks one question on repeat: which two adjacent units appear together most often?

• (t, o) appears 10 times across both words, so it merges into to.
• (to, k) is now the most frequent, so it merges into tok.
• This continues until token has become a single unit.

token                (×6)
token i z a t i o n  (×4)

BPE then keeps merging inside tokenization, surfacing ization as a reusable subword along the way.

Nobody told it that token is a meaningful word or that ization is a common suffix. It discovered both purely from frequency.

Encoding new text

Once the vocabulary is built, BPE uses it to encode new text, including words it has never seen before.

• It tries to match the longest possible vocabulary entry against the input word.
• If the full word exists in the vocabulary, it uses it directly.
• If not, it breaks the word into the largest matching subword chunks it can find.
• As a last resort, it falls back to individual characters, which are always in the vocabulary.

Example: encoding an unseen word

Say that during training, BPE encountered words like serial and deserialize. After the vocabulary-building phase, subwords like de, serial, and ization are all part of its vocabulary.

Now, at encoding time, it sees deserialization, a word it has never encountered before.

1. The full word is not in the vocabulary, so start breaking it down.
2. Match the longest known subword from the left: de.
3. Continue on the remainder: serial.
4. Continue on the remainder: ization.

deserialization  →  de + serial + ization

Three pieces, each surfaced during vocabulary building, are now assembled to handle something entirely new.

Nothing became unknown, and no frequency counting was needed at this stage. The vocabulary was simply rich enough that the pieces were already there.

Code

We will not build a tokenizer from scratch. BPE is well understood, and production-grade implementations already exist, so instead we use gpt-tokenizer: a popular JavaScript/TypeScript library that implements the exact BPE tokenizers used by GPT-2, GPT-3, and GPT-4. It ships with their real vocabularies and merge rules, runs in pure JS (no Python or native bindings), and gives us encode (text to ids) and decode (ids back to text).

This gives us the same tokenization the real models use, out of the box, so we can focus our effort on the rest of the LLM.

import { encode as gptEncode, decode as gptDecode } from "gpt-tokenizer";

const text = "This is a cold day";

const ids = gptEncode(text); // text -> token ids
console.log("tokens:", ids);

// Encoding then decoding should return the original text exactly.
console.log("round-trip ok:", gptDecode(ids) === text);

You can run this in the chapter’s practice code at packages/chapters/src/ch02-tokenizer/02-byte-pair-encoding.ts.

PyTorch → JS mapping

Reference (Python)	Here (TypeScript)
tiktoken (OpenAI BPE)	gpt-tokenizer (OpenAI BPE)