Lesson 1: Scaled dot-product attention, derived and built by hand

Module 2 · Lesson 1 · About 50 minutes · Needs: Module 1, Lessons 1 and 2 (tensors, dot products, softmax, the copy race)

The situation

Sol 404 at Aurora Station. In Module 1 your transformer beat the GRU, but its attention came from PyTorch's ready-made nn.TransformerEncoderLayer. Kofi Mensah will not sign off a part nobody in the lab has opened. "Write attention yourself," he says, "and show me it gives the same numbers."

This lesson starts tinyformer/attention.py. If you met attention in the ai-engineering course (Module 6, Lesson 1), skim sections 1 and 3; the checks, shapes and head survey are new.