Lesson 4: Preference optimisation: teaching which card is better
27 min read·13 Oct 2026
Module 12 · Lesson 4 · About 75 minutes · Needs: Lesson 1 of this module (the window,sft_example,score_cards); Lesson 2 helps
The situation
Sol 444 at Aurora Station. A fine-tuned model writes most of Maya Lindqvist's log cards, not all of them right. Maya will not retype the wrong ones, but she will look at two cards for one log and say which is better.
That judgement is a preference. This lesson turns preferences into training: a reward model, DPO and GRPO. It ends with Kofi Mensah's warning: a reward optimised hard enough gets gamed. The running example is log 5190, Omar Winters at the comms mast.