CourseRetrieval-Augmented Generation: The Complete Course · Module 1: RAG Foundations · part 7 of 95
Part 7 · Module 1: RAG Foundations

Lesson 1: Why a model needs your documents to answer

19 min read·13 Oct 2026
Module 1 · Lesson 1 · About 40 minutes · Needs: basic Python (lists, dictionaries, functions) and a free Groq or Gemini key (course-guide/02)

The situation

Hollin Home Appliances makes blenders, kettles and coffee machines; its support agents search nine systems by hand. On Friday 9 October 2026 Ellie Shaw writes in. Her X200 blender keeps stopping with error E7, as it did last month. Is it covered?

Grace Okoye, Head of Customer Support, wants an assistant that answers from Hollin's own documents, with a source for every claim. You are building it: Hollin Answers. This lesson shows why a model cannot answer Ellie alone, and builds the smallest system that can. Hollin is made up; the research cited is real.

A real case, first

In 2023 Alex Mallen, Akari Asai and colleagues at the University of Washington published "When Not to Trust Language Models" at ACL (arXiv 2212.10511, revised July 2023). They built PopQA, 14,000 questions about facts of every popularity, and asked ten language models to answer from memory alone.

4,000 LEAST POPULAR POPQA QUESTIONS GPT-Neo 6B 15% GPT-Neo 20B 16% GPT-3 davinci-003 19% answered from memory alone (Mallen et al., 2023)
On obscure facts, a far larger model barely helps: 15% to 19% correct from memory alone (amber marks the model).

Models knew famous facts and failed on obscure ones. On the 4,000 least popular questions, GPT-Neo 6B scored 15%, GPT-Neo 20B 16%, and GPT-3 davinci-003, far larger, 19%. In the authors' words, scaling "fails to appreciably improve memorization of factual knowledge in the long tail". A much smaller GPT-Neo 2.7B given retrieved passages beat davinci-003 on those questions.

GPT-3 DAVINCI-003 WITH RETRIEVED PASSAGES fixed 17% of questions answer in top passage: 88% broke 10% of questions answer in top passage: 14% what is retrieved decides which way it goes
Retrieval fixed answers when the top passage held the answer (green) and broke them when it did not (red).

Retrieval was not magic. For davinci-003 it fixed 17% of questions, where the top passage held the answer 88% of the time, and broke 10% the model had got right, where it did so only 14% of the time. A company's documents are the ultimate long tail, and what is retrieved decides the answer.

What you will be able to do

  • Tell what a model remembers from what you hand it.
  • Name the three kinds of fact a model cannot know, and route each one.
  • Measure why pasting every document into the prompt stops working.
  • Build an indexing pipeline and a query pipeline over Hollin's documents.
  • Produce a grounded answer for Ellie and check its citations in code.

1. Parametric versus external knowledge (Essential)

Everything a model "knows" was fixed when it was trained. Know where a fact lives and you know whether to trust the model with it.

Ellie asks "is it covered?" model alone memory only a guess no source model + Hollin docs reads the terms 5 years [pol-warranty-2026]
Alone, the model (amber) can only guess; handed Hollin's document by your system (blue), it reads the answer and cites it (green).

A model answers either from memory, facts stored in its weights during training (parametric knowledge), or from text you hand it with the question (external knowledge). Ellie's question needs Hollin's warranty terms, which were never public, so no memory holds them. Handing over the right document turns a memory test into a reading task.

parametric external lives in model weights your documents fixed at training today, any time update by retraining editing a file source cannot say document ID
Parametric knowledge (amber) is frozen and unsourced; external knowledge (blue) can be changed today and pointed to.

Parametric knowledge is fixed at training time, changes only by retraining, and cannot say where a fact came from. External knowledge can be changed today, checked and pointed to. Retrieval-augmented generation (RAG) is the second route done automatically: find the right text, then generate from it.

Start with the course's standard setup cell (course-guide/02 explains it).

python
%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama
python
import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

Code walkthrough

  • PROVIDER is the one word you change to switch provider; MODELS names each provider's model.
  • The if block copies your key from Colab's Secrets panel. init_chat_model(...) builds llm; max_retries=6 retries when the provider says "too many requests".

Now ask the model with nothing but its memory.

python
def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

closed_book = [("human", "Under Hollin Home Appliances' current warranty terms, how many years "
                         "is the X200 blender motor covered? Answer in one sentence.")]
reply, reason = ask(llm, closed_book)
print((reply.text if reply else "") or f"no answer: {reason or 'empty reply'}")

Code walkthrough

  • ask guards every model call: it tries twice, then returns a reason such as provider_error: BadRequestError instead of crashing.
  • closed_book is one message with no documents. The last line prints the reply, or why there is none.

Look at the reply. A careful model says it lacks Hollin's terms; a careless one names a number. Either way nothing came from Hollin, and a right number would be luck.

Hollin's documents are the course's shared data: forty simulated records used in every lesson. This lesson needs nine, built by this helper:

python
from datetime import date

COURSE_DATE = date(2026, 10, 9)   # "today" in every lesson; never use date.today()
SNAPSHOT = date(2026, 10, 2)      # the day the corpus below was exported

SOURCE = {"policy": "help-centre", "faq": "help-centre", "notice": "help-centre",
          "manual": "manuals", "bulletin": "service-portal", "ticket": "ticket-system",
          "partner": "partner-portal", "internal": "finance-share", "forum": "community-forum"}
AUTHORITY = {"policy": 1.0, "notice": 1.0, "manual": 0.9, "bulletin": 0.9, "partner": 0.8,
             "internal": 0.8, "faq": 0.7, "ticket": 0.3, "forum": 0.1}

def doc(id, type, title, updated, text, acl=("public",), products=(), status="current", authority=None):
    """One document record. acl lists the groups allowed to see it."""
    return {"id": id, "type": type, "title": title, "updated": updated, "text": text,
            "acl": list(acl), "products": list(products), "status": status,
            "source": SOURCE[type], "authority": AUTHORITY[type] if authority is None else authority}

DOCS = []

Code walkthrough

  • COURSE_DATE is the fixed "today"; SNAPSHOT is when the documents were exported.
  • SOURCE maps each type to its Hollin system; AUTHORITY is how far Hollin trusts it (used for ranking later).
  • doc(...) builds one record. acl (access control list) names the groups allowed to see it; status is "current" or "archived".
ONE RECORD: POL-WARRANTY-2024 id pol-warranty-2024 type policy updated 2024-02-10 status archived acl public text motors: 2 years ...
Every Hollin document carries the same fields; this one is public (green) but archived (red), so it must not be trusted.
python
DOCS += [
    doc("pol-warranty-2026", "policy", "Warranty terms (2026)", "2026-03-01",
        "Every Hollin appliance has a 2-year warranty from the date of purchase. Blender motors on the X200 "
        "and X300 Pro are covered for 5 years. The warranty covers manufacturing faults. It does not cover "
        "blade wear, cracked jars, limescale damage or accidental damage. You need proof of purchase to claim."),
    doc("pol-warranty-2024", "policy", "Warranty terms (2024, archived)", "2024-02-10",
        "Every Hollin appliance has a 1-year warranty from the date of purchase. Blender motors are covered "
        "for 2 years. The warranty covers manufacturing faults and cracked jars. You need proof of purchase "
        "to claim.", status="archived"),
    doc("pol-repairs", "policy", "Repairs and turnaround", "2026-05-12",
        "We aim to repair and return an appliance within 10 working days of receiving it. Collection by "
        "courier is free for appliances under warranty. Out-of-warranty repairs carry a £25 inspection fee, "
        "which is deducted from the repair price if you go ahead."),
    doc("faq-warranty-claim", "faq", "How do I make a warranty claim?", "2026-03-01",
        "Book a repair at hollin.example/repair with your model number and proof of purchase. A courier "
        "collects the appliance from your door. Most repairs are back with you within 10 working days."),
]

Code walkthrough

  • Four public help-centre documents from the shared data. pol-warranty-2026 gives X200 motors 5 years.
  • pol-warranty-2024 is the old version, still online with status="archived": motors 2 years. A trap you will meet again.
python
DOCS += [
    doc("man-x200-errors", "manual", "X200 error codes", "2025-06-01",
        "E4 means the jar lid is not locked: twist the lid until it clicks. E7 means the motor has overheated: "
        "unplug the blender for 20 minutes. E9 means a sensor fault: contact Hollin support.", products=["X200"]),
    doc("man-x300-errors", "manual", "X300 Pro error codes", "2026-04-10",
        "E4 means the vacuum lid seal is loose: take the seal out and press it back into its groove. E12 means "
        "the vacuum pump has failed: contact Hollin support.", products=["X300"]),
    doc("ntc-x200-recall", "notice", "Safety notice: X200 jar recall", "2026-09-14",
        "X200 jars from batch B-2291 may crack when used with hot liquids. Stop using the jar and order a "
        "free replacement at hollin.example/recall. The batch number is printed on the base of the jar.",
        products=["X200"]),
    doc("tkt-30412", "ticket", "Case 30412: X200 stopped mid-blend", "2026-09-05",
        "Case 30412. Customer Ellie Shaw reports her X200 stopped mid-blend showing E7. Agent advised unplugging for 20 "
        "minutes; the blender worked afterwards.", acl=["staff"], products=["X200"]),
    doc("int-parts-cost", "internal", "Parts cost prices (internal)", "2026-08-01",
        "Internal cost prices: X200 jar £6.10, BLD-X200 blade assembly £3.40, PMP-X300 pump £11.80, "
        "FLT-P9-HEPA filter £7.25. Do not share outside Finance.", acl=["finance"]),
]
for d in DOCS:
    if "motor" in d["text"]:
        print(f"{d['id']:18} {d['status']:8}", next(s for s in d["text"].split(". ") if "motor" in s))

Code walkthrough

  • Two error-code pages, a safety notice, a ticket about Ellie's first E7 (staff only) and cost prices (Finance only).
  • The loop prints each sentence mentioning "motor". Two answer the warranty question and disagree: 5 years (current), 2 years (archived).

2. Three things a model cannot know (Essential)

Some facts a model can never supply. Each kind needs a different fix, and the wrong fix gives answers that look fine and are not.

WHAT ELLIE NEEDS, AND WHY MEMORY FAILS motor cover private only in Hollin docs jar recall after the cut-off published 2026-09-14 repair status fast-changing case 30540: in workshop
Each of Ellie's three facts is out of a model's reach for a different reason (red marks what memory cannot supply).

Ellie needs three facts. The motor cover is private: it lives only in Hollin's documents. The jar recall is after the cut-off: published on 14 September 2026, after the model's training data was collected. Her repair's status is fast-changing: case 30540 moves from booked to in workshop to dispatched within days. A model's memory covers none of them.

HOW OFTEN EACH HOLLIN SOURCE CHANGES manuals on change help centre weekly tickets hourly case database every minute retrieve from an index query live
Slow-changing documents are retrieved from an index (blue); the case database changes too fast and is queried live (green).

The first two are fixed by retrieval: put the document in the prompt. The third is not. An index is a copy taken at one moment, and a repair status copied on Friday is wrong by Monday. Facts that change faster than you can re-index belong in a database query. That is why Hollin's case database is never embedded.

python
import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE cases (case_id INTEGER PRIMARY KEY, customer TEXT, product TEXT,
              opened TEXT, status TEXT, partner TEXT, labour_minutes INTEGER)""")
db.executemany("INSERT INTO cases VALUES (?, ?, ?, ?, ?, ?, ?)", [
    (30412, "Ellie Shaw",   "X200", "2026-09-05", "closed",      None,           None),
    (30540, "Ellie Shaw",   "X200", "2026-09-30", "in workshop", "brook-street", None),
])
print(db.execute("SELECT case_id, status, partner FROM cases WHERE customer = ?", ("Ellie Shaw",)).fetchall())

recent = [d["id"] for d in DOCS if d["updated"] >= "2026-01-01"]
print(len(recent), "of", len(DOCS), "documents changed in 2026:", recent)

Code walkthrough

  • sqlite3.connect(":memory:") creates a temporary database holding Ellie's two cases from the shared data. The ? inserts the value safely.
  • The query prints 30412 closed and 30540 in the workshop at Brook Street Repairs. Tomorrow it gives tomorrow's status.
  • recent lists the documents updated in 2026: 7 of the 9, including the recall and the current warranty.

3. The ceiling of putting everything in the prompt (Essential)

Why not paste every document into every prompt? For nine documents it works. It stops working for reasons you can measure.

WHAT EACH QUESTION SENDS stuffing retrieval 9 documents every time, or the 3 that match
Stuffing sends every document (grey) with every question; retrieval (blue) sends only the few that match.

Prompt stuffing sends every document with every question. It needs no index and makes a fine first prototype. Retrieval sends only the few documents that match. The gap grows with every document Hollin adds.

python
%pip install -q sentence-transformers==6.1.0
python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
def tokens(text):
    return len(model.tokenizer(text, add_special_tokens=False)["input_ids"])

doc_tokens = sum(tokens(f"[{d['id']}] {d['text']}") for d in DOCS)
words = sum(len(d["text"].split()) for d in DOCS)
PRICE_IN = 0.075 / 1_000_000       # $ per input token, groq:openai/gpt-oss-20b (course-guide/02)
print(doc_tokens, "tokens for", words, "words, sent with every question")
print(f"${doc_tokens * PRICE_IN * 100_000:.2f} of input per 100,000 questions")
library = 2_000 * 400 * doc_tokens / words  # simulated: 2,000 manual pages of 400 words
print(f"a 2,000-page library: about {library:,.0f} tokens; context window 131,072")

Code walkthrough

  • SentenceTransformer(...) downloads a small open embedding model (about 130 MB, once). Section 4 searches with it; here its tokenizer counts tokens, the pieces a model reads and is billed by (the chat model's count differs a little).
  • doc_tokens is the size of a stuffed prompt; PRICE_IN comes from the course's price table.
  • library applies the same tokens-per-word ratio to a simulated 2,000-page manual library.
PROMPT TOKENS, TO SCALE 9 docs 506 window 131,072 library about 1.36 million (simulated)
Nine documents fit easily (blue), but a simulated 2,000-page library (red) is ten times the model's window (amber).

The nine documents come to 506 tokens, about $3.79 of input per 100,000 questions. Trivial. But a real manual library of 2,000 pages would be about 1.36 million tokens, several times the 131,072-token window of the course's model. Long before that, every question pays for every page, and replies slow down as prompts grow.

ACCURACY BY WHERE THE ANSWER SITS no documents: 56.1% first middle last
Schematic of Liu et al. (2023): with the answer mid-prompt, GPT-3.5-Turbo could do worse (amber curve) than with no documents (red line).

Even when everything fits, models do not read it evenly. Nelson Liu and colleagues ("Lost in the Middle", TACL 2023) moved the answer through a prompt of 20 or 30 documents. Accuracy was highest at the start and end and dropped in the middle. In the worst case, GPT-3.5-Turbo did worse than with no documents at all (56.1%).

Break it: stuff Ellie's prompt

Stuffing has a worse problem than cost. Run this, which adds three of the shared data's users.

python
USERS = {
    "ellie": {"name": "Ellie Shaw", "role": "customer", "groups": {"public"}},
    "grace": {"name": "Grace Okoye", "role": "Head of Customer Support", "groups": {"public", "staff", "partners"}},
    "owen":  {"name": "Owen Fletcher", "role": "Finance Manager", "groups": {"public", "staff", "partners", "finance"}},
}

def can_see(user, d):
    return bool(USERS[user]["groups"] & set(d["acl"]))

print("in Ellie's prompt, not hers to see:", [d["id"] for d in DOCS if not can_see("ellie", d)])
print("in Ellie's prompt, archived:", [d["id"] for d in DOCS if d["status"] != "current"])

Code walkthrough

  • USERS gives each person a set of groups. Ellie, a customer, is only in public.
  • can_see is the permission rule: one of the user's groups must be on the document's access list (& is set intersection).
  • The prints show what stuffing hands the model for Ellie.
ELLIE'S STUFFED PROMPT pol-warranty-2026 pol-warranty-2024 pol-repairs faq-warranty-claim man-x200-errors man-x300-errors ntc-x200-recall tkt-30412 int-parts-cost red: not Ellie's to see; amber: archived
Stuffing hands Ellie's model two documents she may not see (red) and the archived terms (amber) beside the safe ones (green).

Ellie's stuffed prompt holds the staff ticket with her case notes, cost prices marked "Do not share outside Finance", and the archived terms beside the current ones. Nothing stops the model quoting any of them. Retrieval lets you choose, per user and per question, what the model reads.

4. Indexing pipeline versus query pipeline (Essential)

Retrieval needs two pieces of software that run at different times. Putting work in the right one keeps answers fast and documents fresh.

indexing: when documents change load embed store index query: for every question embed search prompt answer
Indexing (top) runs when documents change; the query pipeline (bottom) runs per question and reads the index (grey); only the last step is the model (amber).

The indexing pipeline runs when documents change: it loads them, turns each into a vector of numbers (an embedding) and stores the vectors in an index. The query pipeline runs for every question: it embeds the question with the same model, finds the closest vectors, and passes those documents to the model. Slow, careful work goes in the first; the second must answer in seconds.

python
import numpy as np

def build_index(docs):
    """Indexing pipeline: run once, and again whenever the documents change."""
    vectors = model.encode([d["text"] for d in docs], normalize_embeddings=True)
    return {"docs": docs, "vectors": vectors, "built_from": SNAPSHOT}

INDEX = build_index(DOCS)
print(INDEX["vectors"].shape, "vectors, built from the snapshot of", INDEX["built_from"])
print("length of each vector:", np.linalg.norm(INDEX["vectors"], axis=1).round(3))

Code walkthrough

  • model.encode(...) turns each text into 384 numbers. normalize_embeddings=True makes each vector length 1, so a dot product measures similarity of meaning (cosine similarity, Module 4).
  • build_index keeps vectors beside their documents and records the snapshot. The prints show (9, 384) and lengths of 1.0.
THE INDEX: 9 ROWS, 384 NUMBERS EACH pol-warranty-2026 ... pol-warranty-2024 ... pol-repairs ... ... int-parts-cost ...
The index is one row per document: an ID beside 384 numbers (grey) that only say which texts are close in meaning.

Each document is now a row: an ID and 384 numbers. The index knows nothing about warranties, only which texts are near which. So both pipelines must use the same embedding model: vectors from different models are not comparable.

python
def retrieve(question, user, k=3, index=INDEX):
    """Query pipeline: embed the question, score every document, keep the best k the user may see."""
    q = model.encode(question, normalize_embeddings=True)
    scores = index["vectors"] @ q
    ranked = sorted(zip(scores, index["docs"]), key=lambda pair: -pair[0])
    return [(round(float(s), 3), d) for s, d in ranked if can_see(user, d)][:k]

STANDALONE = "Is a repeated E7 motor overheating fault on the X200 blender covered by the warranty?"
for score, d in retrieve(STANDALONE, "ellie"):
    print(score, d["id"], d["status"])
score, top = retrieve(STANDALONE, "grace", k=1)[0]
print("Grace's top result:", score, top["id"], top["acl"])

Code walkthrough

  • index["vectors"] @ q scores all nine documents in one step; sorted(...) ranks them, best first.
  • The list comprehension drops what the user may not see, then keeps k. STANDALONE is Ellie's question written out in full, from the shared data.
ELLIE'S QUESTION: COSINE SCORES tkt-30412 0.811 staff only pol-warranty-2026 0.788 man-x200-errors 0.741 pol-warranty-2024 0.725 archived pol-repairs 0.621
The staff ticket scores highest but the filter removes it (red); Ellie gets the current terms and error codes (green) and, third, the archived terms (amber).

For Ellie the top three are pol-warranty-2026 (0.788), man-x200-errors (0.741) and pol-warranty-2024 (0.725). For Grace, a member of staff, the top result is tkt-30412 (0.811), the ticket about Ellie's first E7. The permission filter is what keeps it out of Ellie's results. Notice the archived terms in third place: retrieval found them because they match well, not because they are right.

RAG ON WORLD LEADERS (LEWIS ET AL., 2020) 2016 leaders 2018 leaders 2016 index 70% 4% 2018 index 12% 68%
Swapping the index updated the model's knowledge with no retraining: matching index green, mismatched red.

Splitting the pipelines is also how a RAG system learns. In the original RAG paper, Patrick Lewis and colleagues (NeurIPS 2020) swapped a 2016 Wikipedia index for a 2018 one. With the matching index the model named 70% and 68% of world leaders correctly; with the mismatched one, 12% and 4%. No retraining: they replaced the index.

5. Grounding and attribution as requirements (Essential)

An answer Ellie cannot check is worth little, and one quoting the archived terms is a liability. Two properties fix that, and code can check them.

ONE ANSWER, TWO CHECKS X200 motors are covered for 5 years [pol-warranty-2026] grounded the claim is in the document attributed names its source
Grounding means the claim is in the retrieved text; attribution means the answer names that text (green: both hold).

An answer is grounded when every claim is supported by the retrieved documents. It has attribution when each claim names the document that supports it. Grounding stops the model filling gaps from memory; attribution lets Ellie, an agent or your code check the answer in seconds. Grace will not ship an answer without both.

WHAT THE MODEL RECEIVES rules: use only sources, cite [id], else I don't know [pol-warranty-2026] (current) Every Hollin appliance ... [man-x200-errors] (current) E4 means ... [pol-warranty-2024] (archived) Every Hollin appliance ... Question: Is a repeated E7 ... covered by the warranty?
The prompt carries rules (grey), sources with IDs and status (blue, the archived one amber) and Ellie's question.

The prompt carries both. A short set of rules says: use only these sources, cite each claim by its ID in square brackets, and say "I don't know" when the sources do not answer. Each source arrives with its ID and status, so the model can cite it and skip the archived one.

python
RULES = ("You answer questions for Hollin Home Appliances customers. Use only the sources given. "
         "After each claim, cite the source ID in square brackets, like [pol-repairs]. "
         "Ignore sources marked archived. If the sources do not answer the question, reply: I don't know.")

def answer(question, user):
    hits = retrieve(question, user)
    sources = "\n".join(f"[{d['id']}] ({d['status']}) {d['text']}" for _, d in hits)
    reply, reason = ask(llm, [("system", RULES), ("human", f"Sources:\n{sources}\n\nQuestion: {question}")])
    text = reply.text.strip() if reply else ""
    return {"text": text or None, "reason": reason or (None if text else "empty_reply"),
            "retrieved": [d for _, d in hits]}

result = answer(STANDALONE, "ellie")
print(result["text"] or f"no answer: {result['reason']}")

Code walkthrough

  • RULES is the system message, the answer contract in four sentences (Module 9 strengthens it).
  • answer retrieves, writes each source as [id] (status) text, and calls ask. It returns the text, a reason when there is none (provider_error or empty_reply), and the retrieved documents.

Read the reply. Look for the 5-year motor cover and a citation of [pol-warranty-2026]. If it mentions 2 years, the archived terms leaked through despite the rule.

reply from model find [ids] retrieved? and current? pass flag
Code, not the model, decides: each cited ID must be one that was retrieved and is current (green pass, red flag).

The model can still cite an ID it was never given, or the archived terms. So check every answer in code.

python
import re

def check_citations(result):
    """Attribution check: every cited ID must be a retrieved, current document."""
    if result["text"] is None:
        return [f"no answer: {result['reason']}"]
    if "don't know" in result["text"].replace("’", "'"):
        return []
    by_id = {d["id"]: d for d in result["retrieved"]}
    cited = re.findall(r"\[([a-z0-9-]+)\]", result["text"])
    problems = [] if cited else ["no citation"]
    for c in cited:
        if c not in by_id:
            problems.append(f"{c}: never retrieved")
        elif by_id[c]["status"] != "current":
            problems.append(f"{c}: {by_id[c]['status']}")
    return problems

print(check_citations(result) or "passed: every citation is a retrieved, current document")

Code walkthrough

  • A missing answer is reported with its reason, so a provider failure never looks like a pass. An honest "I don't know" passes, since it makes no claim.
  • re.findall(...) pulls every [id] out of the text. Each one must be in the retrieved documents and be current.

Break it: fluent answers with bad citations

Real models make these mistakes now and then. Here they are scripted, so the checker's catch is repeatable.

python
hits = [d for _, d in retrieve(STANDALONE, "ellie")]
replies = {
    "invented ID": "Yes, the motor is covered for 5 years. [pol-warranty-2025]",
    "archived": "The X200 motor is covered for 2 years. [pol-warranty-2024]",
    "no citation": "Yes, blender motors are covered for 5 years.",
    "grounded": "Yes, X200 blender motors are covered for 5 years. [pol-warranty-2026]",
}
for name, text in replies.items():
    verdict = check_citations({"text": text, "reason": None, "retrieved": hits})
    print(f"{name:12}", verdict or "passed")

Code walkthrough

  • Four answers that all read well, each checked against Ellie's real retrieved documents.
  • Output: the invented ID fails as never retrieved, the archived one as archived, the uncited one fails, the grounded one passes.
FOUR FLUENT REPLIES, ONE CHECKER invented ID pol-warranty-2025: never retrieved archived pol-warranty-2024: archived no citation no citation grounded passed
Three confident replies fail the check (red); only the reply citing the current terms passes (green).

All four sound confident. Only attribution makes the difference visible: without IDs there is nothing to check, and the 2-year answer would reach Ellie. A citation proves where a claim came from, not that it is true; whether the cited text supports the claim is Module 9's check.

In production

indexing job on schedule or change index query service per question model answer log cited IDs
In production the indexing job and the query service run apart; logging cited IDs (green) lets you trace any answer back.

Teams run the two pipelines as separate jobs. Indexing runs on a schedule or when a source changes, and freshness depends on it; the query service never waits for it. Mallen and colleagues found that retrieving only for less popular facts improved PopQA accuracy and halved GPT-3 API costs: retrieval is a choice per question. Logging each answer with its cited IDs lets you trace a wrong answer to a document or to the model. Lessons 2 and 3 map where pipelines fail and when RAG is the wrong tool.

What you added to Hollin Answers

  • The first slice of shared data: nine documents, three users, Ellie's cases.
  • An indexing pipeline (build_index) and a query pipeline (retrieve) with a permission filter.
  • The first grounded answer with citations (answer), guarded against provider failures.
  • A citation check (check_citations) that catches invented, archived and missing citations.
  • A rule for Grace: documents are retrieved, repair statuses are looked up live.

Check yourself

  1. Why could a larger model not have answered Ellie's warranty question from memory?
  2. Ellie asks where her repair is. Why is that answered by a database query rather than retrieval?
  3. Give three costs of stuffing every document into the prompt, besides the token bill.
  4. Which work belongs in the indexing pipeline, and why must both pipelines use the same embedding model?
  5. A reply cites [pol-warranty-2026] and passes check_citations. What has that proved, and what has it not?
  1. The terms are private to Hollin, so no training data held them; size helps mainly with facts seen often.
  2. It changes faster than an index is rebuilt, so a copy would be stale.
  3. Any three: it outgrows the window, slows replies, uses mid-prompt facts less reliably, and shows every user every document.
  4. Loading, embedding and storing, which is slow and runs when documents change. Vectors from different models are not comparable.
  5. That the cited document was retrieved and is current; not that it says what the answer claims.

Exercise

Retrieval put the archived 2024 terms in Ellie's top three. Write retrieve_current, which works like retrieve but never returns a document whose status is not "current", and still returns k documents. Then run the check.

python
def retrieve_current(question, user, k=3):
    """Your code: like retrieve(), but skip any document whose status is not "current"."""
    return retrieve(question, user, k)

ids = [d["id"] for _, d in retrieve_current(STANDALONE, "ellie")]
print(ids)
print("PASS" if ids[0] == "pol-warranty-2026" and "pol-warranty-2024" not in ids and len(ids) == 3
      else "Not yet: an archived document is still in the results, or fewer than 3 came back")

Code walkthrough: the starter calls retrieve unchanged, so the archived terms are still third and the check prints Not yet. A working version asks retrieve for every document, filters on status, then keeps the first k; it returns pol-warranty-2026, man-x200-errors and pol-repairs and prints PASS. Module 6 builds the proper version, which ranks current documents above archived ones.

Key terms

TermMeaning
Parametric knowledgeWhat a model absorbed in training, stored in its weights
Retrieval-augmented generation (RAG)Find the relevant text first, then generate an answer from it
Prompt stuffingSending every document with every question
EmbeddingA list of numbers that stands for a text's meaning
Indexing pipelineLoads, embeds and stores documents when they change
Query pipelineEmbeds a question, finds the closest documents and generates the answer
GroundedEvery claim is supported by the retrieved documents
AttributionEach claim names the document that supports it

Sources

  • Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, Hannaneh Hajishirzi, "When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories", ACL 2023 (arXiv December 2022, revised July 2023). https://arxiv.org/abs/2212.10511
  • Patrick Lewis, Ethan Perez, Aleksandra Piktus and others (Facebook AI Research, UCL, NYU), "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", NeurIPS 2020. https://arxiv.org/abs/2005.11401
  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang, "Lost in the Middle: How Language Models Use Long Contexts", TACL 2023. https://arxiv.org/abs/2307.03172