Chapter 2 · Tokenizer & Data

Chapter 2 covers the full input pipeline: turning raw text into token ids, feeding them to the model as batched, sliding-window (input, target) pairs, and finally into the embeddings the Transformer works with.

Lessons

1. A simple word-level tokenizer
2. Byte-pair encoding (BPE)
3. Data sampling
4. Dataset
5. DataLoader
6. Embeddings

Code

• @node-llm/core → src/tokenizer/bpe.ts, src/data/dataset.ts, src/data/dataloader.ts
• Runnable practice code, one script per lesson:
  • pnpm ch02:1 → word-level tokenizer
  • pnpm ch02:2 → byte-pair encoding
  • pnpm ch02:3 → data sampling
  • pnpm ch02:4 → Dataset & DataLoader
  • pnpm ch02:5 → embeddings
  • pnpm ch02:6 → positional encoding