Intermediate · Transformers from scratch
The transformer block
Build the app's GPT as GPT-2 builds it: layer normalization by hand, the MLP and the pre-norm residual block, a GPT with learned positions and a tied output layer, and what a GPT costs, worked out from its configuration.
- You build
- block.py, then gpt.py, then costs.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
Loading the workspace…
Loading the workspace…