Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Transformers from scratch

The transformer block

Build the app's GPT as GPT-2 builds it: layer normalization by hand, the MLP and the pre-norm residual block, a GPT with learned positions and a tied output layer, and what a GPT costs, worked out from its configuration.

You build
block.py, then gpt.py, then costs.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3.5 hours
Runs on
Your machine

Loading the workspace…