Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Beginner · Text as numbers

Tokens and byte-pair encoding

Build an online shop's tokenizer from its product titles: learn byte-pair merges, encode and decode titles as tiktoken does, measure what titles cost in tokens in three languages, and ship a shortener that fits any title into a token limit.

You build
bpe.py, then prompts.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3.5 hours

Loading the workspace…