CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 8: Pretraining · part 58 of 111
Part 58 · Module 8: Pretraining

Lesson 2: Eight ways to learn from raw text

32 min read·13 Oct 2026
Module 8 · Lesson 2 · About 60 minutes · Needs: Lesson 1 (the harness: compare_run, late_valid, verdict); Module 7, Lesson 1 (masks and the three shapes)

The situation

Sol 421 at Aurora Station. The harness from Lesson 1 is ready, and Maya Lindqvist asks the question it was built for: "Which way of learning from the station record gives us the best model?" Kofi Mensah repeats his terms: one budget, paired seeds, a metric every contender can be scored on.

This lesson builds eight pretraining objectives on the package and races them. The running example is Lesson 1's: document 0's first row, "Bruno Ferreira works in the water plant on the coolant loop." Every objective turns that row into a different exercise.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.