Skip to content

Tokenizers, by hand

How text becomes tokens: word splitting, byte-pair encoding, a vocabulary, and encode/decode that round-trips.

Syllabus

Word tokens

Vocabulary and encode

Decode and round-trip

Byte-pair encoding