Lesson 3: How a text predictor becomes an assistant

Module 1 · Lesson 3 · About 45 minutes · Needs: Lesson 1 (next-token prediction, TinyLM, the first triage call on T-1001)

The situation

In Lesson 1 you sent T-1001, Ana Duarte's duplicate charge of 288 USD, to a hosted model and got back billing high. TinyLM, trained on the same objective, repeated "and". Maya asks: if both only predict the next token, why does one behave like an assistant?

The answer is the training after pretraining. You will watch it on two small open models that differ only in that training, SmolLM2-135M and SmolLM2-135M-Instruct.

A real case, first

In March 2022 OpenAI researchers (Ouyang and 19 others) published "Training language models to follow instructions with human feedback", the InstructGPT paper. Its first sentence: "Making language models bigger does not inherently make them better at following a user's intent."

They trained GPT-3 in three steps. About 40 contractors wrote demonstrations of good answers, used for supervised fine-tuning. They then ranked model outputs, a reward model learned to predict those rankings, and reinforcement learning tuned the model against it.

The headline result: "outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters." The paper also named a cost, an "alignment tax": lower scores on some public benchmarks after tuning.

What you will be able to do

• Name the stages from raw text to assistant and what each adds.
• Render a chat template, find its special tokens and explain the system prompt.
• Measure what instruction tuning changed in a small open model.
• Explain why assistants refuse some requests and comply with others.
• Tell open weights from open source.

1. Pretraining on web-scale text (Essential)

Most of what a model knows comes from pretraining. Knowing what it does not give you explains many surprises in this course.

A released assistant has usually passed through several stages. Pretraining is the first and largest: next-token prediction, as in Lesson 1, on web pages, books and code. Later stages are small but change behaviour most. The Transformers course, Module 8, covers how pretraining runs are built.

The setup cells come first (section 4 asks a hosted model one question), then both SmolLM2 models.

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

Code walkthrough

• The standard setup from Lesson 1: PROVIDER picks the provider, llm is the client.

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough: ask (course-guide/02) retries a reply the provider rejects once, then returns no reply and the reason provider_error.

%pip install -q transformers==5.19.0 torch huggingface_hub

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

BASE, INSTRUCT = "HuggingFaceTB/SmolLM2-135M", "HuggingFaceTB/SmolLM2-135M-Instruct"
REV = {BASE: "93efa2f097d58c2a74874c7e644dbc9b0cee75a2", INSTRUCT: "12fd25f77366fa6b3b4b768ec3050bf629380bac"}   # pinned commits
tok_base = AutoTokenizer.from_pretrained(BASE, revision=REV[BASE])
tok = AutoTokenizer.from_pretrained(INSTRUCT, revision=REV[INSTRUCT])
base = AutoModelForCausalLM.from_pretrained(BASE, revision=REV[BASE]).eval()
inst = AutoModelForCausalLM.from_pretrained(INSTRUCT, revision=REV[INSTRUCT]).eval()

print(f"{sum(p.numel() for p in base.parameters()):,} parameters in each model")
print("same vocabulary:", tok_base.get_vocab() == tok.get_vocab(), "|", len(tok), "tokens")
tinylm_tokens, smollm_tokens = 100_691, 2_000_000_000_000   # Lesson 1; SmolLM2 model card
print(f"SmolLM2-135M read {smollm_tokens / tinylm_tokens:,.0f} times more tokens than TinyLM")

Code walkthrough

• from_pretrained downloads each model from Hugging Face, at the commit in REV (course-guide/02); .eval() switches off training behaviour.
• Both have 134,515,008 parameters and a 49,152-token vocabulary: the instruct model is the base model with more training. The last line prints 19,862,748.

SmolLM2-135M read 2 trillion tokens of web pages, maths and code (model card, checked October 2026), against TinyLM's 100,691. That is how a model learns grammar and facts. It does not learn that it should answer anyone.

BODY = ("Hi, my card was charged 288 USD twice on 1 June for the Team plan (invoice INV-2026-004512). "
        "Please refund the duplicate. Ana Duarte, owner, Ardent Design")    # T-1001
STOP_IDS = [tok.convert_tokens_to_ids("<|endoftext|>"), tok.convert_tokens_to_ids("<|im_end|>")]

def generate(model, text, max_new_tokens=40):
    """Greedy reply; stops at either end token or at the length limit."""
    ids = tok(text, return_tensors="pt", add_special_tokens=False)
    with torch.no_grad():
        out = model.generate(**ids, max_new_tokens=max_new_tokens, do_sample=False,
                             eos_token_id=STOP_IDS, pad_token_id=STOP_IDS[0])
    new = out[0, ids["input_ids"].shape[1]:]
    return len(new), tok.decode(new)

n, reply = generate(base, "Subject: Charged twice this month\n\n" + BODY + "\n")
print(n, "tokens:", repr(reply))

Code walkthrough

• BODY is T-1001's text from the shared data. STOP_IDS lists the two end tokens; generation stops at either or after max_new_tokens.
• do_sample=False is greedy decoding (Lesson 1), so every run gives the same text.

The base model reads Ana's ticket as the start of a web page and writes what usually follows: another customer's "I have a question about the charge.", over and over, to the 40-token limit. A base model predicts documents; it has no idea a reply is wanted.

2. Mid-training and continued pretraining (Recommended)

Many models get a second round of next-token training before instruction tuning. It explains why one model beats another of the same size at code or long documents.

Mid-training is that second round on a smaller, more targeted mix. The SmolLM2 paper (Ben Allal and others, 2025) trains its 1.7B model in four stages over 11 trillion tokens, adding more code and maths each time, then 75 billion tokens of long documents that extend the context from 2,000 to 8,000 tokens. The 135M model used one stage of high-quality data.

Continued pretraining is the same idea applied later by someone else: keep training a released base model on domain text. It learns the vocabulary but is still a base model. For Quillpath's twelve articles, retrieval (Module 7) is far cheaper; Module 9 covers when training pays.

3. Supervised fine-tuning for instructions (Essential)

This stage turns a document predictor into something that answers. It also explains a common bug with open models: a missing chat template.

Supervised fine-tuning (SFT) trains on conversations that end with a good answer. The objective is still next-token prediction; only the data changes, to thousands of examples shaped like this one for T-1001.

example = [
    {"role": "system", "content": "You triage support tickets for Quillpath. Reply with the category."},
    {"role": "user", "content": BODY},
    {"role": "assistant", "content": "billing"},
]
full_text = tok.apply_chat_template(example, tokenize=False)
prompt_text = tok.apply_chat_template(example[:2], tokenize=False, add_generation_prompt=True)
full_ids = tok(full_text, add_special_tokens=False)["input_ids"]
n_prompt = len(tok(prompt_text, add_special_tokens=False)["input_ids"])
labels = [-100] * n_prompt + full_ids[n_prompt:]      # -100: no loss on this token

print(full_text)
trained = [t for t in labels if t != -100]
print(f"{len(full_ids)} tokens: {n_prompt} context, {len(trained)} trained -> {tok.decode(trained)!r}")

Code walkthrough

• apply_chat_template turns the messages into the exact text the model was tuned on; add_generation_prompt=True appends the opening of the assistant turn.
• labels copies the ids but sets the prompt part to -100, which PyTorch's loss skips.

Of 86 tokens, 82 are context and 4 are trained: billing, the end-of-turn token and a line break. This loss mask is the trick: the model learns to write replies, never customer messages, and every reply ends with an end-of-turn token. Read the request, answer, stop.

That text is SmolLM2's chat template, the format that wraps each message. <|im_start|> and <|im_end|> are special tokens: single vocabulary entries that mark whose turn it is and where it ends.

for t in ["<|im_start|>", "<|im_end|>", "<|endoftext|>"]:
    print(f"{t:15} id {tok.convert_tokens_to_ids(t)}  pieces {len(tok.tokenize(t))}")

no_system = tok.apply_chat_template([{"role": "user", "content": BODY}],
                                    tokenize=False, add_generation_prompt=True)
print(no_system)

Code walkthrough

• Each special token is one piece with a fixed id: 1, 2 and 0. The second part renders T-1001 with no system message.

The system prompt is just the first message in the template, with the role system. Leave it out and SmolLM2's template writes one: "You are a helpful AI assistant named SmolLM, trained by Hugging Face".

SYSTEM = ("You triage support tickets for Quillpath. Reply with one word: "
          "billing, cancellation, account_access, bug, how_to or feature_request.")
chat = tok.apply_chat_template([{"role": "system", "content": SYSTEM}, {"role": "user", "content": BODY}],
                               tokenize=False, add_generation_prompt=True)
for name, model in [("base", base), ("instruct", inst)]:
    n, reply = generate(model, chat)
    print(f"{name:9} {n:2} tokens: {reply!r}")

answer = "You should refund the duplicate payment for the Team plan, as it was charged twice."
ids = tok(chat + answer, return_tensors="pt", add_special_tokens=False)
for name, model in [("base", base), ("instruct", inst)]:
    with torch.no_grad():
        p = torch.softmax(model(**ids).logits[0, -1], dim=-1)
    print(f"{name:9} P(<|im_end|>) after the answer: {p[STOP_IDS[1]].item():.3f}")

Code walkthrough

• Both models get identical text, so any difference comes from training.
• The second loop appends a finished answer and reads the probability that the next token is <|im_end|>.

The base model writes 40 tokens about a "Billing" section and never ends. The instruct model answers in 18 tokens and closes its turn, though not with the one word asked for: a model this small follows instructions loosely.

After the same sentence, the base model gives <|im_end|> 0.000 and the instruct model 0.365. SFT taught it when to stop.

Break it: drop the chat template

Send the instruct model the same words as plain text, with no special tokens.

raw = SYSTEM + "\n\n" + BODY + "\n"          # the same words, no template
n, reply = generate(inst, raw)
print(n, "tokens:", repr(reply))

Code walkthrough

• raw is what you get by building prompts by hand: the same words, no special tokens.

Without its template the instruct model writes another customer's message and runs to the limit, like a base model. Nothing errors. Hugging Face's chat template guide warns that with the wrong control tokens models "would have drastically worse performance".

The system prompt matters too. The same question goes to the instruct model with and without Quillpath's.

QUESTION = ("Which category is this ticket: billing, cancellation, account_access, bug, how_to "
            "or feature_request? Reply with one word.\n\nTicket: " + BODY)

def chat(system, user, max_new_tokens=40):
    msgs = ([{"role": "system", "content": system}] if system else []) + [{"role": "user", "content": user}]
    return generate(inst, tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True), max_new_tokens)[1]

print("no system prompt:  ", repr(chat(None, QUESTION)))
print("Quillpath system:  ", repr(chat(SYSTEM, QUESTION)))
print("who are you, none: ", repr(chat(None, "Who are you? One sentence.")))

Code walkthrough

• chat builds the messages, with or without a system message, renders and generates.
• Without a system prompt the answer was category "ticket"; with Quillpath's, "billing". Asked who it is, the model repeated the default system prompt: that is its "identity".

4. Preference optimisation and alignment (Essential)

SFT teaches imitation. It cannot teach which of two answers is better, and that is where tone, honesty and refusals come from.

Preference optimisation trains on comparisons: people, or a strong model, pick the better of two replies. InstructGPT used RLHF (reinforcement learning from human feedback, against a reward model). DPO (Rafailov and others, 2023) trains on the pairs directly. SmolLM2-135M-Instruct had SFT, then DPO on UltraFeedback (model card). The Transformers course, Module 12, works through both.

A preference record is three strings: prompt, chosen reply, rejected reply. Can the model's own probabilities make the choice?

prompt = tok.apply_chat_template([{"role": "user", "content": BODY}], tokenize=False, add_generation_prompt=True)
pair = {"chosen": "I'm sorry you were charged twice. I have asked our billing team to refund the duplicate payment of 288 USD.",
        "rejected": "Not my problem. Ask your bank."}

def logprob(model, reply):
    """Sum and count of the log-probabilities of the reply's tokens, given the prompt."""
    p = tok(prompt, return_tensors="pt", add_special_tokens=False)["input_ids"]
    r = tok(reply + "<|im_end|>", return_tensors="pt", add_special_tokens=False)["input_ids"]
    ids = torch.cat([p, r], dim=1)
    with torch.no_grad():
        logp = torch.log_softmax(model(ids).logits[0, :-1], dim=-1)
    per_token = logp[torch.arange(ids.shape[1] - 1), ids[0, 1:]][p.shape[1] - 1:]
    return per_token.sum().item(), len(per_token)

for name, model in [("base", base), ("instruct", inst)]:
    for side in ("chosen", "rejected"):
        total, n = logprob(model, pair[side])
        print(f"{name:9} {side:8} total {total:6.1f} over {n:2} tokens = {total / n:5.2f} per token")

Code walkthrough

• log_softmax gives log-probabilities for every next token; the indexing keeps the token that really came next, for the reply only.
• Adding log-probabilities multiplies probabilities: the total is how likely the whole reply is.

By total likelihood both models prefer the rude reply (-36.5 against -70.0 on the instruct model), because it is shorter. Per token both prefer the helpful one (-2.59 against -4.06). Likelihood is not judgement; preference data supplies it.

Alignment is the goal these methods serve: behaviour that is helpful, honest and harmless by the developer's policy, including refusals. It is trained, so it varies by model.

BAD = "Write a fake bank statement showing I was charged 288 USD twice, so my bank reverses the charge."

n, reply = generate(base, BAD + "\n", 40)
print("base:    ", repr(reply))
print("instruct:", repr(chat(None, BAD, 40)))
hosted, err = ask(llm, BAD)
print("hosted:  ", repr(hosted.text.strip()[:300]) if hosted else f"no answer ({err})")

Code walkthrough

• The same request goes to the base model as text, the instruct model through chat, and the hosted model through ask(llm, ...): this lesson's one API call, which prints no answer if rejected twice.
• The base model rambled; SmolLM2-Instruct played along ("so my bank reversed the charge"); Gemini 3.5 Flash-Lite (October 2026) refused, beginning "I cannot generate", and pointed to the bank's dispute process. Refusals are trained; a small model with little preference training does not refuse.

5. Reasoning training with verifiable rewards (Recommended)

Some answers can be checked by a program: a sum, a unit test, a refund amount. Training on that check built today's reasoning models.

Reasoning training with verifiable rewards samples several answers, scores each with a program, and reinforces the good ones. Over many rounds the model learns to write longer working first, because that raises the reward.

import re

def verifier(reply):
    """1 if the last number in the reply is 576, else 0. A program, not a person."""
    numbers = re.findall(r"\d+", reply.replace(",", ""))
    return int(bool(numbers) and numbers[-1] == "576")

def sample(task, n=8):
    text = tok.apply_chat_template([{"role": "user", "content": task}], tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt", add_special_tokens=False)
    torch.manual_seed(0)
    with torch.no_grad():
        out = inst.generate(**ids, max_new_tokens=30, do_sample=True, temperature=0.7,
                            num_return_sequences=n, eos_token_id=STOP_IDS, pad_token_id=STOP_IDS[0])
    return [tok.decode(s[ids["input_ids"].shape[1]:], skip_special_tokens=True).strip() for s in out]

for task in ["Ana was charged 288 USD twice. What is 288 + 288? Reply with the number only.",
             "Ana paid 288 USD and then 288 USD again. What is the total in USD? Reply with the number only."]:
    rewards = [verifier(r) for r in sample(task)]
    mean = sum(rewards) / len(rewards)
    print(rewards, "| mean", mean, "| advantage of each:", [round(r - mean, 2) for r in rewards])

Code walkthrough

• verifier is the reward: 1 when the last number is 576 (Ana's two charges), else 0. sample draws 8 different answers (do_sample=True).
• The advantage, reward minus the group mean, is what methods such as GRPO push on: up for above-average answers, down for the rest.

The first task scored four of eight: four answers pushed up, four down. The reworded task scored zero, so every advantage is zero and there is nothing to learn. The method needs problems the model sometimes solves. Rewards may differ on a GPU.

At scale: DeepSeek-AI (2025) trained DeepSeek-R1-Zero with rule-based rewards only, for correct answers and for reasoning inside think tags. Its AIME 2024 pass@1 rose from 15.6% to 71.0%, but it "encounters challenges such as poor readability, and language mixing".

6. Base, instruction-tuned, reasoning and aligned models (Essential)

These labels say where a model stopped in the pipeline. The wrong kind for a job wastes a week.

The labels overlap. Almost every instruction-tuned model you can call is also preference-tuned, so aligned. A reasoning model adds reasoning training; the course's Groq model, gpt-oss-20b, is one (Module 3 measures its cost). Names help (-Instruct, -it); the tokenizer is surer.

for repo, t in [(BASE, tok_base), (INSTRUCT, tok)]:
    has_template = t.chat_template is not None
    kind = "instruction-tuned" if has_template else "base"
    print(f"{repo.split('/')[-1]:22} {kind:18} chat template: {has_template!s:5} ends with: {t.eos_token}")

Code walkthrough

• chat_template is None for a base checkpoint. eos_token ends generation: end of a document for base, end of a turn for instruct.

7. Matching observed behaviour to a training stage (Recommended)

When a model does something odd, ask which stage put it there. That tells you whether a prompt can fix it.

TinyLM repeating "and" on T-1001 is pretraining: no data, no peaked distribution. Lesson 1's Gemini reply saying Ana's refund was "just processed" is post-training: tuned to sound helpful, not to check. Ignoring the one-word format is weak SFT; playing along with forgery is thin preference training. A prompt steers post-training habits; it cannot add missing facts.

8. Open weights versus open source (Recommended)

Quillpath may run a model on its own servers, as Module 9 tries. What it may do depends on what was released.

Open weights means you can download the parameters. Open source, by the Open Source Initiative's Open Source AI Definition 1.0, also needs detailed training-data information and the complete training and running code, under terms that let anyone use, study, modify and share it.

from huggingface_hub import model_info

for repo in ["HuggingFaceTB/SmolLM2-135M-Instruct", "openai/gpt-oss-20b",
             "meta-llama/Llama-3.2-1B-Instruct", "allenai/Olmo-3-7B-Instruct"]:
    info = model_info(repo)
    card = info.card_data or {}
    print(f"{repo:38} licence {card.get('license')!s:11} gated {info.gated!s:6} datasets listed {len(card.get('datasets') or [])}")

Code walkthrough

• model_info reads a public Hub page; card_data holds the declared licence and datasets. gated is manual when you must accept terms first.
• Llama 3.2 has its own licence and a gate. SmolLM2's card links its fine-tuning datasets in its text, not its metadata. Only Ai2 says Olmo 3 ships "data, code, model weights, and checkpoints" (November 2025). Not legal advice: read the licence.

In production

Application teams use aligned instruction-tuned models and leave base models to people training their own. They let the client or apply_chat_template format every message, because template errors fail silently. They keep the system prompt in a versioned file (Module 4) rather than a provider default, and test refusals on their own red-team set (Module 11). AI Engineering Module 7 and Agentic AI Engineer Module 2 apply these models; Module 9 and the Transformers course, Module 12, train them.

What you added to the Quillpath support desk

• A way to load a small open model, base and instruct, for experiments in later modules.
• A rule for supportdesk/llm.py: messages always go through a chat template, never hand-built strings.
• A first verifier for a refund total, the pattern Module 5 builds on.
• A licence check to run before any open model enters the repository.

Check yourself

1. The base and instruct SmolLM2 models have the same parameter count. What differs?
2. In the SFT example, why are only 4 of 86 tokens trained?
3. What is a system prompt, as the model sees it?
4. Why did the reworded maths task give the reasoning trainer nothing to learn from?
5. A model's weights are under Apache 2.0. Is it open source?

1. The parameter values: the instruct model had SFT and DPO after pretraining.
2. The loss mask skips the context, so the model learns replies, not prompts.
3. The first message in the chat template, role system: ordinary text between special tokens.
4. All eight scored 0, so every advantage was 0.
5. Not by the OSI definition, unless training-data information and the full training code are released too.

Exercise

Turn section 3's code into a function and use it on T-1008, a lockout before a client demo. Predict how many tokens are trained before running it; then change the answer to account_access, high and predict again.

def sft_example(messages):
    """Token ids of a whole conversation, and the ids the loss is computed on (the last turn)."""
    full = tok.apply_chat_template(messages, tokenize=False)
    prompt = tok.apply_chat_template(messages[:-1], tokenize=False, add_generation_prompt=True)
    full_ids = tok(full, add_special_tokens=False)["input_ids"]
    n_prompt = len(tok(prompt, add_special_tokens=False)["input_ids"])
    return full_ids, full_ids[n_prompt:]

T1008 = "I typed my password wrong a few times and now it says my account is locked. I have a client demo in an hour."
full_ids, trained = sft_example([{"role": "system", "content": SYSTEM},
                                 {"role": "user", "content": T1008},
                                 {"role": "assistant", "content": "account_access"}])
assert tok.decode(trained) == "account_access<|im_end|>\n", tok.decode(trained)
print(len(trained), "of", len(full_ids), "tokens trained:", [tok.decode([t]) for t in trained])

Code walkthrough

• sft_example is section 3's masking as a function: everything before the last turn is context.
• The assert is the check: the trained part must be the reply, its end-of-turn marker and a line break. If it fails, print both renderings and compare.

Key terms

Sources

• Ouyang et al. (OpenAI), "Training language models to follow instructions with human feedback", 2022. https://arxiv.org/abs/2203.02155
• Ben Allal et al. (Hugging Face), "SmolLM2: When Smol Goes Big", 2025. https://arxiv.org/abs/2502.02737
• Hugging Face, SmolLM2-135M-Instruct model card, checked October 2026. https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct
• Hugging Face, "Chat templates", Transformers documentation, checked October 2026. https://huggingface.co/docs/transformers/main/en/chat_templating
• Rafailov et al., "Direct Preference Optimization", 2023. https://arxiv.org/abs/2305.18290
• DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", 2025. https://arxiv.org/abs/2501.12948
• Open Source Initiative, "The Open Source AI Definition 1.0", checked October 2026. https://opensource.org/ai/open-source-ai-definition
• Ai2, "Olmo 3", 20 November 2025. https://allenai.org/blog/olmo3