Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Transformers from scratch

Attention

Build the attention a reading app's story model runs in every layer: scaled dot-product attention with its causal mask, heads split and merged, masks for batches of stories, and attention in tiles that never holds a whole row of scores.

You build
attention.py, then self_attention.py, then tiles.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 4 hours
Runs on
Your machine

Loading the workspace…