Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Transformers from scratch

Rotary positions and the KV cache

Run Hugging Face's SmolLM2-135M in your own code: rotary positions, RMSNorm, SwiGLU and grouped-query attention matched to transformers' Llama, a KV cache that writes each token without reading the story again, a review of an assistant's cached generator, and a generator for batches of prompts.

You build
rope.py, then smollm.py, then cache.py, then generator.py, then batch.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3.5 hours
Runs on
Your machine

Loading the workspace…