Beginner · Text as numbers
Build an online shop's tokenizer from its product titles: learn byte-pair merges, encode and decode titles as tiktoken does, measure what titles cost in tokens in three languages, and ship a shortener that fits any title into a token limit.
Loading the workspace…