Intermediate · Transformers from scratch
Attention
Build the attention a reading app's story model runs in every layer: scaled dot-product attention with its causal mask, heads split and merged, masks for batches of stories, and attention in tiles that never holds a whole row of scores.
- You build
- attention.py, then self_attention.py, then tiles.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
Loading the workspace…
Loading the workspace…