Lesson 5: Quantisation: fewer bits per weight, measured

Module 11 · Lesson 5 · About 60 minutes · Needs: Module 10, Lesson 3 (fake_quant and KV cache quantisation); Module 6's base recipe; Module 8's package as pasted in Module 10, Lesson 3

The situation

Sol 434 at Aurora Station, the second day of the Earth link outage. Hana Aoki, the Compute Officer, wants a copy of base on each robot's controller, and those have very little memory. "Can it live in a sixth of its size? And what does it lose?"

You follow one log header, Log 4821, sol 212, the night shift., and one row of weights, at 8, 4, 3 and 2 bits. Module 10, Lesson 3 shrank the KV cache; this lesson shrinks weights and activations.