CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 8: Pretraining · part 61 of 111
Part 61 · Module 8: Pretraining

Lab: Corpus, tokeniser, packing and objectives, tested

35 min read·13 Oct 2026
Module 8 · Lab · About 60 minutes · Needs: lessons 1 to 4 of this module, and Module 7's lab

What you are building

TINYFORMER-LAB AFTER MODULE 8 corpus.py report, dedup tokenizer.py BPE, 512 packing.py shards, mask objectives.py 8 + harness v0.7.0 package: data, model, trainer, unchanged test_m08_pretrain 7 tests corpus report control counted objective race 4 x 3 seeds git tag v0.8.0
The lab adds four pretraining files (blue) to the unchanged v0.7.0 package (grey), pins them with seven tests (green), saves two measured results (amber) and tags v0.8.0.

Sol 421 at Aurora Station. The four lessons left their pretraining tools in notebook cells: Lesson 1's pipeline and corpus report, Lesson 2's eight objectives, Lesson 3's deduplication, document mask and contamination check, Lesson 4's discipline of writing the plan and the manifest. Kofi Mensah asks the module's question once more: does the package say what the lessons said?

Today you save them as four files in tinyformer-lab: corpus.py, tokenizer.py, packing.py and objectives.py. You write the corpus report as a file, with the lookup contamination counted against a control. You pin the tools with seven tests, rerun part of Lesson 2's race from the package, and tag v0.8.0.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside