Intermediate · Transformers from scratch
Training a small GPT
Train the reading app's story model on your own machine: a tokenizer learnt from the stories, batches of tokens packed as GPT-3 packs its documents, a training loop with AdamW, warm-up, cosine decay and clipping, a review of an assistant's trainer, and a model that beats a target loss within a budget of compute.
- You build
- tokenizer.py, then data.py, then train.py, then trainer.py, then story.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
Loading the workspace…
Loading the workspace…