Module 1: The Road to Attention

Why recurrent networks gave way to attention, measured rather than asserted. Everything runs on a Colab CPU.

What you need first

• Python and NumPy; matrix products, transposes and the chain rule
• Having trained one neural network (AI Engineering, Module 4, if not)
• If you have done AI Engineering Module 5 (why recurrent models struggle) and Module 6 (attention in overview), skim Lesson 2's section 4 and Lesson 3's section 6
• No GPU, no API key

What you will build

The tinyformer-lab repository, with the shared data as a tested package and a three-seed race of a GRU, a GRU with attention and a one-layer transformer encoder, tagged v0.1.0.

Lessons

Extras

• 04-module-lab.md: Lab: Start tinyformer-lab with one honest race (50 minutes). Nine seeded runs read against a noise floor.
• 05-incident-file.md: Incident: The causal flag that masked nothing (30 minutes). PyTorch issue #96941 (2023), where is_causal=True without a mask did not mask, replayed on the lab's encoder and caught with a causal check.