Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Transformers from scratch

Training a small GPT

Train the reading app's story model on your own machine: a tokenizer learnt from the stories, batches of tokens packed as GPT-3 packs its documents, a training loop with AdamW, warm-up, cosine decay and clipping, a review of an assistant's trainer, and a model that beats a target loss within a budget of compute.

You build
tokenizer.py, then data.py, then train.py, then trainer.py, then story.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 4 hours
Runs on
Your machine

Loading the workspace…