Lesson 3: Stretching a trained model to longer inputs

Module 4 · Lesson 3 · About 50 minutes · Needs: Module 4, Lessons 1 and 2 (train short, read long; rope and the frequency bands)

The situation

Sol 408 at Aurora Station. Ana Dubois's log for sol 286 is 293 characters, and base reads 128. Maya Lindqvist wants whole logs read at once. Hana Aoki asks the money question: "Do we have to retrain from scratch at the longer length, or can we stretch the model we have?" Kofi Mensah sets the test before anyone answers: "Two measurements. Loss at every position, and a retrieval task at every distance. Nothing ships on one of them."

This lesson adds context extension to tinyformer/positions.py: one rope_scaling switch with four published recipes, and both of Kofi's measurements.