Part 9 · Module 1: RAG Foundations

Lesson 3: Choosing RAG, long context, fine-tuning or tools

22 min read·13 Oct 2026

Module 1 · Lesson 3 · About 45 minutes · Needs: Python dictionaries and lists; what an embedding is, in one sentence (a list of numbers that stands for the meaning of a text)

The situation

On Friday 9 October 2026 Ellie Shaw asks two things in one chat. Her X200 keeps stopping with E7: is it covered? And where is her repair, case 30540?

Kwame asks: "The model takes 131,072 tokens. Why not paste every document into the prompt?" Owen asks what each answer costs. Grace wants a written decision per Hollin source: retrieve it, send it whole, train on it, or look it up live. This lesson makes it, on Ellie's two questions.

Ellie's two questions need two kinds of source: documents for the cover, a live database row for the repair (blue is your system, green the right path).

A real case, first

In November 2024, Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei and Michael Bendersky (Google DeepMind and the University of Michigan) published "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach" at EMNLP 2024. They ran Gemini-1.5-Pro, GPT-4O and GPT-3.5-Turbo on nine public long-document datasets, once with the whole document in the prompt and once with five retrieved chunks of 300 words.

Long context scored higher on average when the model could afford it. RAG cost far less. And for 63% of queries the two gave exactly identical predictions. So they built Self-Route: ask with the retrieved chunks first, with the instruction "Write unanswerable if the query can not be answered based on the provided text", and send the whole document only for queries marked unanswerable. In their words, it "significantly reduces the computation cost while maintaining a comparable performance to LC". You will build a small Self-Route for Hollin at the end of this lesson.

Self-Route tries RAG first and pays for the whole document only when the model says the chunks are not enough (amber is the model's judgement).

What you will be able to do

Measure what long context and RAG each send to the model, in tokens and in cost.
Explain why fine-tuning is the wrong tool for Hollin's facts and the right one for behaviour.
Send structured questions to a database through a tool call instead of a vector search.
Say when agentic search beats an index.
Decide per source with three variables, and combine the options in one router.

1. RAG versus long context (Essential)

Long context is the strongest rival to RAG, because it needs no index at all. If it is good enough for Hollin, most of this course is unnecessary, so test it first.

Long context sends every document Ellie may see with every question; RAG sends only the three a search ranks closest (green).

Long context means sending every document the user may see with every question. RAG sends only the few that a search ranks closest. Both are tried below on Ellie's coverage question, using a slice of the shared corpus: the policies, manuals, bulletins and notices.

python

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

python

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

Code walkthrough

The standard setup cell (course-guide/02). PROVIDER is the one word you change.

python

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough

ask guards every model call: two tries, then no reply and a provider_error reason instead of a crash.

python

%pip install -q sentence-transformers==6.1.0

python

from datetime import date

COURSE_DATE = date(2026, 10, 9)   # "today" in every lesson; never use date.today()
SNAPSHOT = date(2026, 10, 2)      # the day the corpus below was exported

SOURCE = {"policy": "help-centre", "faq": "help-centre", "notice": "help-centre",
          "manual": "manuals", "bulletin": "service-portal", "ticket": "ticket-system",
          "partner": "partner-portal", "internal": "finance-share", "forum": "community-forum"}
AUTHORITY = {"policy": 1.0, "notice": 1.0, "manual": 0.9, "bulletin": 0.9, "partner": 0.8,
             "internal": 0.8, "faq": 0.7, "ticket": 0.3, "forum": 0.1}

def doc(id, type, title, updated, text, acl=("public",), products=(), status="current", authority=None):
    """One document record. acl lists the groups allowed to see it."""
    return {"id": id, "type": type, "title": title, "updated": updated, "text": text,
            "acl": list(acl), "products": list(products), "status": status,
            "source": SOURCE[type], "authority": AUTHORITY[type] if authority is None else authority}

DOCS = []

Code walkthrough

Shared data (course-guide/06), unchanged. doc(...) builds one record; acl lists who may see it.

python

DOCS += [
    doc("pol-warranty-2026", "policy", "Warranty terms (2026)", "2026-03-01",
        "Every Hollin appliance has a 2-year warranty from the date of purchase. Blender motors on the X200 "
        "and X300 Pro are covered for 5 years. The warranty covers manufacturing faults. It does not cover "
        "blade wear, cracked jars, limescale damage or accidental damage. You need proof of purchase to claim."),
    doc("pol-warranty-2024", "policy", "Warranty terms (2024, archived)", "2024-02-10",
        "Every Hollin appliance has a 1-year warranty from the date of purchase. Blender motors are covered "
        "for 2 years. The warranty covers manufacturing faults and cracked jars. You need proof of purchase "
        "to claim.", status="archived"),
    doc("pol-repairs", "policy", "Repairs and turnaround", "2026-05-12",
        "We aim to repair and return an appliance within 10 working days of receiving it. Collection by "
        "courier is free for appliances under warranty. Out-of-warranty repairs carry a £25 inspection fee, "
        "which is deducted from the repair price if you go ahead."),
    doc("pol-spare-parts", "policy", "Ordering spare parts", "2026-04-20",
        "Spare parts are sold at parts.hollin.example. Standard delivery takes 3 to 5 working days and costs "
        "£3.95. Delivery is free on parts orders over £30."),
    doc("pol-privacy", "policy", "Your account data", "2026-02-01",
        "You can download or delete your account data from Settings, then Privacy. Deletion requests are "
        "completed within 30 days."),
    doc("pol-registration", "policy", "Registering your appliance", "2025-11-02",
        "Register your appliance at hollin.example/register so we can contact you about safety notices. "
        "You do not need to register to claim under the warranty."),
]

Code walkthrough

Six public policies. pol-warranty-2024 is archived but still public.

python

DOCS += [
    doc("man-x200-specs", "manual", "X200 Blender specifications", "2025-06-01",
        "The X200 blender has a 1,200 W motor, a 2-litre Tritan jar and three speeds. High speed reaches "
        "22,000 RPM. Model number HL-X200-UK.", products=["X200"]),
    doc("man-x200-errors", "manual", "X200 error codes", "2025-06-01",
        "E4 means the jar lid is not locked: twist the lid until it clicks. E7 means the motor has overheated: "
        "unplug the blender for 20 minutes. E9 means a sensor fault: contact Hollin support.", products=["X200"]),
    doc("man-x200-cleaning", "manual", "X200 cleaning and care", "2025-06-01",
        "Wash the jar and blades in warm soapy water or in the dishwasher. Never put the motor base in water; "
        "wipe it with a damp cloth.", products=["X200"]),
    doc("man-x200-blades", "manual", "X200 blade assembly", "2025-06-01",
        "A grinding noise, or food left in chunks, usually means the blades are worn. Replace the blade "
        "assembly, part BLD-X200, about once a year with daily use.", products=["X200"]),
    doc("man-x300-specs", "manual", "X300 Pro Blender specifications", "2026-04-10",
        "The X300 Pro blender has a 1,500 W motor, a vacuum blending mode and ten speeds. High speed reaches "
        "30,000 RPM. Model number HL-X300-UK.", products=["X300"]),
    doc("man-x300-errors", "manual", "X300 Pro error codes", "2026-04-10",
        "E4 means the vacuum lid seal is loose: take the seal out and press it back into its groove. E12 means "
        "the vacuum pump has failed: contact Hollin support.", products=["X300"]),
    doc("man-k50-kettle", "manual", "K50 Smart Kettle", "2025-09-09",
        "The K50 smart kettle boils 1.7 litres in about four minutes and can keep water warm for up to two "
        "hours. Descale it once a month with citric acid.", products=["K50"]),
    doc("man-k50-app", "manual", "Pairing the K50 with the Hollin Home app", "2025-09-09",
        "To pair the K50 with the Hollin Home app, hold the temperature dial down for 5 seconds until the ring "
        "light flashes white, then choose Add appliance in the app.", products=["K50"]),
]

Code walkthrough

Eight manual pages, each tagged with its product.

python

PARTNERS_AND_STAFF = ("partners", "staff")
DOCS += [
    doc("sb-109", "bulletin", "SB-109: X200 motor coupling", "2026-01-15",
        "Some X200 blenders made before March 2025 have a soft motor coupling that wears early. Fit coupling "
        "kit CPL-X200 at the first repair visit, free of charge.", acl=PARTNERS_AND_STAFF, products=["X200"]),
    doc("sb-112", "bulletin", "SB-112: X300 Pro pump replacement", "2026-06-30",
        "If E12 returns after the seal has been reseated, replace the vacuum pump, part PMP-X300. Allow 35 "
        "minutes of labour.", acl=PARTNERS_AND_STAFF, products=["X300"]),
    doc("sb-114", "bulletin", "SB-114: K50 firmware 2.3", "2026-09-22",
        "K50 kettles on firmware 2.1 may switch off before boiling. Update them to firmware 2.3 through the "
        "Hollin Home app.", acl=PARTNERS_AND_STAFF, products=["K50"]),
    doc("ntc-x200-recall", "notice", "Safety notice: X200 jar recall", "2026-09-14",
        "X200 jars from batch B-2291 may crack when used with hot liquids. Stop using the jar and order a "
        "free replacement at hollin.example/recall. The batch number is printed on the base of the jar.",
        products=["X200"]),
    doc("ntc-repair-centre-move", "notice", "Repair centre move", "2026-09-28",
        "Our repair centre is moving from Cleckheaton to Wakefield. Repairs booked between 1 and 16 October "
        "2026 may take 2 extra working days."),
]

Code walkthrough

Three bulletins for partners and staff only, and two public notices.

python

USERS = {
    "ellie": {"name": "Ellie Shaw", "role": "customer", "groups": {"public"}},
    "grace": {"name": "Grace Okoye", "role": "Head of Customer Support", "groups": {"public", "staff", "partners"}},
    "owen":  {"name": "Owen Fletcher", "role": "Finance Manager", "groups": {"public", "staff", "partners", "finance"}},
    "lucy":  {"name": "Lucy Hart", "role": "technician, Brook Street Repairs", "tenant": "brook-street",
              "groups": {"public", "partners", "partner:brook-street"}},
    "dan":   {"name": "Dan Mercer", "role": "owner, Calder Fix", "tenant": "calder-fix",
              "groups": {"public", "partners", "partner:calder-fix"}},
}

def can_see(user, d):
    return bool(USERS[user]["groups"] & set(d["acl"]))

# (user, question, document IDs that must never reach this user)
ACL_PROBES = [
    ("lucy", "what labour rate does Calder Fix get", {"ptn-calder-rates", "eml-calder-pump"}),
    ("ellie", "what does Hollin pay for a PMP-X300 pump", {"int-parts-cost"}),
    ("ellie", "SB-112", {"sb-112", "eml-calder-pump"}),
    ("grace", "cost price of the FLT-P9-HEPA filter", {"int-parts-cost"}),
]
print({u: sum(can_see(u, d) for d in DOCS) for u in USERS})

CONVERSATION = [
    ("user", "My X200 keeps stopping and shows E7."),
    ("assistant", "E7 means the motor has overheated. Unplug the blender for 20 minutes before using it "
                  "again. [man-x200-errors]"),
    ("user", "I did that last month too. Is it covered?"),
]
STANDALONE = "Is a repeated E7 motor overheating fault on the X200 blender covered by the warranty?"
EXPECTED_ANSWER = ("Yes. Blender motors on the X200 are covered for 5 years under the 2026 warranty terms; "
                   "book a repair with proof of purchase. [pol-warranty-2026] [faq-warranty-claim]")

Code walkthrough

can_see is the permission rule: one shared group is enough. It prints 16 documents for ellie. STANDALONE is "Is it covered?" rewritten to stand alone.

python

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
def count(text):
    return len(embedder.tokenizer(text, verbose=False)["input_ids"])

visible = [d for d in DOCS if can_see("ellie", d)]
vectors = embedder.encode([d["text"] for d in visible], normalize_embeddings=True)

def retrieve(question, k=3):
    scores = vectors @ embedder.encode(question, normalize_embeddings=True)
    return [visible[i] for i in scores.argsort()[::-1][:k]]

def context(docs):
    return "\n\n".join(f"[{d['id']}] {d['text']}" for d in docs)

long_context = context(visible)
rag_context = context(retrieve(STANDALONE))
print(len(DOCS), "documents in this slice;", len(visible), "that Ellie may see")
print("long context:", count(long_context), "tokens")
print("RAG context: ", count(rag_context), "tokens from", [d["id"] for d in retrieve(STANDALONE)])

Code walkthrough

SentenceTransformer(...) loads a small embedding model on the CPU. count uses its tokenizer; the chat model counts a little differently.
visible keeps only documents Ellie may see. retrieve scores each against the question by dot product and returns the top 3.
context joins documents with their IDs so the model can cite them. It prints 19 documents, 16 visible, 784 tokens for long context and 189 for RAG, which ranks pol-warranty-2026 first.

For Ellie's question the long context is 784 tokens and the RAG context 189, counted on this lesson's 19-document slice.

At this size both fit easily. The model's context window, the most tokens it can read at once, is 131,072 for the course's Groq model. Hollin's real systems are far larger than this slice, so the next block scales it up.

python

PRICE_IN = 0.075 / 1_000_000   # US$ per input token, groq:openai/gpt-oss-20b (course-guide/02)
WINDOW = 131_072               # that model's context window, in tokens

for name, ctx in [("long context", long_context), ("RAG top 3", rag_context)]:
    tokens = count(ctx)
    print(f"{name:12} {tokens:6} tokens   ${tokens * PRICE_IN * 1000:.3f} per 1,000 questions")

for growth in (1, 10, 100, 1000):   # simulated: Hollin's sources at 10, 100, 1,000 times this slice
    tokens = count(long_context) * growth
    print(f"slice x{growth:<5} {tokens:9,} tokens   fits in the window: {tokens < WINDOW}")

Code walkthrough

PRICE_IN is the input price from the course's price table, in US dollars as the provider lists it.
The first loop prices both contexts: $0.059 against $0.014 per 1,000 questions, about four times more for long context.
The second loop multiplies the slice (simulated growth). At 1,000 times, 784,000 tokens, it no longer fits.

The slice fits a 131,072-token window easily; at 1,000 times its size (simulated growth) it no longer fits at all (red).

At the course model's input price, long context costs about four times as much per question here, and the gap grows with the corpus.

The money is small at this scale, but it grows with every question and every document. And the window is a hard wall: past it, long context is not an option at all.

Break it: send everything

Long context also sends things you would rather it did not. Search the long context for what it says about motor cover.

python

for d in visible:
    for sentence in d["text"].split(". "):
        if "motor" in sentence.lower() and "year" in sentence:
            print(f"{d['id']:18} {d['status']:9} {sentence}")

Code walkthrough

The loop splits each visible document into sentences and prints those that mention a motor and years, with the document's status.
Two lines print: the current terms say 5 years, the archived 2024 terms say 2 years. Both are public, so both go into the prompt.

Sending everything sends the archived 2-year motor cover (red) next to the current 5 years (green), and leaves the model to choose.

This is how case 30501 happened. Long context does not choose for you; it hands the model both versions and hopes. Retrieval at least gives you a place to filter by status (Module 6 does this). Long context also makes per-user permissions harder, because every user needs their own large prompt.

2. RAG versus fine-tuning (Essential)

"Train the model on our documents" sounds like the permanent fix. For facts that change, it is the slowest and least reliable one, and knowing why saves weeks.

Retrieval changes what the model knows and can cite; fine-tuning changes how it behaves (amber), and needs a training run for every change.

Fine-tuning means continuing to train a model on your own examples, which changes its weights. It is good at behaviour: a tone, a format, a classification. It is poor at knowledge: facts that must be exact, cited and kept current.

In Ovadia et al. (2024), RAG lifted Mistral 7B on new events from 0.481 to 0.875; fine-tuning reached 0.588 at best.

Ovadia and colleagues at Microsoft measured this (arXiv, January 2024). On 910 questions about events after the models' training cut-off, Mistral 7B scored 0.481 alone, 0.588 at best after fine-tuning, and 0.875 with RAG. Their summary: "Fine-tuning is not competitive with RAG."

Now apply Hollin's 8 October update batch to an index.

python

UPDATES = {
    "date": "2026-10-08",
    "upsert": [
        doc("ntc-repair-centre-move", "notice", "Repair centre move", "2026-10-08",
            "Our repair centre is moving from Cleckheaton to Wakefield. Repairs booked between 1 and 16 "
            "October 2026 may take 3 extra working days."),
        doc("sb-115", "bulletin", "SB-115: X300 Pro seal kit", "2026-10-08",
            "When replacing the X300 Pro vacuum pump, also fit the new lid seal SEAL-X300 from the same kit. "
            "Claim both parts on one case.", acl=PARTNERS_AND_STAFF, products=["X300"]),
    ],
    "delete": [("sb-114", "withdrawn: firmware 2.3 caused a new fault; wait for firmware 2.4")],
}
print(len(UPDATES["upsert"]), "upserts,", len(UPDATES["delete"]), "delete")

Code walkthrough

The shared 8 October batch, unchanged: two upserts and one delete.

python

index = {d["id"]: d for d in DOCS}     # the retrieval side: documents by ID
before = index["ntc-repair-centre-move"]["text"].split("may take ")[1]

for d in UPDATES["upsert"]:
    index[d["id"]] = d                 # insert, or replace the old version
for doc_id, reason in UPDATES["delete"]:
    index.pop(doc_id, None)            # withdrawn: gone from the next search

after = index["ntc-repair-centre-move"]["text"].split("may take ")[1]
print("repair notice:", before, "->", after)
print("sb-114 still searchable:", "sb-114" in index, "| sb-115 searchable:", "sb-115" in index)
print("records changed:", len(UPDATES["upsert"]) + len(UPDATES["delete"]), "| model weights changed: 0")

Code walkthrough

index stands in for a search index: one entry per document ID.
An upsert inserts a record, or replaces it if the ID exists; pop removes the withdrawn bulletin.
It prints "2 extra working days" becoming "3", sb-114 gone, sb-115 present, and 3 records changed.

The 8 October batch is three record changes in an index and zero changes to model weights; a fine-tuned model would need retraining.

Three record changes, and the next question sees them. A fine-tuned model would need a new training run to learn "3 days", and nothing guarantees it forgets sb-114. Hollin's plan: retrieval for facts; prompting first, and fine-tuning only if needed, for tone.

3. RAG versus tool calling for structured data (Essential)

Ellie's second question, "where is my repair?", has an exact answer in one database row that changes every minute. Searching text for it is the wrong tool, and the break below shows how.

Text with meaning goes to retrieval; rows with exact fields, such as Ellie's case, go to a query.

Unstructured data is free text: policies, manuals, emails. Structured data is rows with exact fields: case IDs, statuses, dates. Questions about structured data want filters, counts and the latest value. Hollin's cases live in this table, from the shared data.

python

import sqlite3

db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE cases (case_id INTEGER PRIMARY KEY, customer TEXT, product TEXT,
              opened TEXT, status TEXT, partner TEXT, labour_minutes INTEGER)""")
db.executemany("INSERT INTO cases VALUES (?, ?, ?, ?, ?, ?, ?)", [
    (30412, "Ellie Shaw",   "X200", "2026-09-05", "closed",      None,           None),
    (30455, "Amir Hussain", "K50",  "2026-09-10", "closed",      None,           None),
    (30501, "Jo Bennett",   "X300", "2026-09-12", "open",        None,           None),
    (30533, "Sian Lloyd",   "X300", "2026-09-24", "repaired",    "calder-fix",   50),
    (30540, "Ellie Shaw",   "X200", "2026-09-30", "in workshop", "brook-street", None),
    (30547, "Mark Dale",    "P9",   "2026-10-01", "booked",      None,           None),
    (30552, "Nina Patel",   "C70",  "2026-10-06", "booked",      "brook-street", None),
])
print(db.execute("SELECT case_id, status, partner FROM cases WHERE customer = ?", ("Ellie Shaw",)).fetchall())

Code walkthrough

The shared case table, unchanged: seven cases in an in-memory SQLite database. It prints Ellie's two cases; 30540 is "in workshop".

Break it: embed the case table

python

rows = db.execute("SELECT case_id, customer, product, opened, status FROM cases").fetchall()
row_text = [f"Case {c}: {who}, {product}, opened {opened}, status {status}" for c, who, product, opened, status in rows]
row_vectors = embedder.encode(row_text, normalize_embeddings=True)

question = "how many repairs are booked"
best = (row_vectors @ embedder.encode(question, normalize_embeddings=True)).argsort()[::-1][:3]
print("vector search returns:")
for i in best:
    print("  ", row_text[i])
booked = db.execute("SELECT COUNT(*) FROM cases WHERE status = 'booked'").fetchone()[0]
print("SQL returns:", booked, "booked")

Code walkthrough

Each row becomes a sentence and then a vector, as a careless pipeline would do.
Vector search returns three similar-looking rows: two booked cases and one repaired. It cannot count.
COUNT(*) with a WHERE filter returns the exact answer, 2.

Vector search over case rows returns three similar-looking rows and no count (red); SQL returns the exact answer, 2 (green).

Vector search finds text that looks like the question. It cannot add, filter or know which row is newest, and its copy is stale the minute a status changes. For structured data you query the source.

In tool calling the model asks (amber), your code runs the query (blue), and the result goes back to the model to write the answer.

Tool calling lets the model ask your code to run a function. The model sees the function's name and description, replies with a request such as case_status(30540), your code runs it, and the result goes back to the model to write the answer.

python

from langchain_core.tools import tool

CURRENT_CUSTOMER = "Ellie Shaw"   # set by the login, never by the model

@tool
def case_status(case_id: int) -> str:
    """Look up the live status of one of the current customer's repair cases."""
    row = db.execute("SELECT status, partner, opened FROM cases WHERE case_id = ? AND customer = ?",
                     (case_id, CURRENT_CUSTOMER)).fetchone()
    if row is None:
        return f"No case {case_id} for this customer."
    status, partner, opened = row
    return f"Case {case_id}: {status}, opened {opened}, repairer {partner or 'Hollin repair centre'}."

print(case_status.invoke({"case_id": 30540}))
print(case_status.invoke({"case_id": 30533}))   # Sian Lloyd's case, not Ellie's

Code walkthrough

@tool turns a Python function into a LangChain tool; its docstring becomes the description the model reads.
The SQL filters on CURRENT_CUSTOMER, which comes from the login. Ellie's case returns its status; Sian Lloyd's case returns "No case 30533 for this customer."

The tool, not the model, decides whose cases can be read: Ellie's case returns (green), another customer's does not (red).

The permission check lives in your code, not in the prompt. Whatever case number the model asks for, the query only reads the logged-in customer's rows. Now let the model use the tool.

python

from langchain_core.messages import HumanMessage, ToolMessage

with_tools = llm.bind_tools([case_status])
messages = [HumanMessage("Where is my repair, case 30540? Reply in one sentence.")]
reply, reason = ask(with_tools, messages)

if reply is None or not reply.tool_calls:
    print("no tool call:", reason or reply.text)
else:
    call = reply.tool_calls[0]
    result = case_status.invoke(call["args"])
    print("model asked for:", call["name"], call["args"])
    print("your code returned:", result)
    messages += [reply, ToolMessage(result, tool_call_id=call["id"])]
    final, reason = ask(with_tools, messages)
    print("answer:", final.text if final else f"no answer ({reason})")

Code walkthrough

bind_tools tells the model the tool exists. ask sends Ellie's question through the guard.
If the reply holds a tool call, your code runs it with the model's arguments and appends the result as a ToolMessage, tied to the call by its ID.
The second ask lets the model write the answer. Look for case_id 30540 in the request and "in workshop" in the answer; if either call fails twice, you see no answer and the reason instead.

4. RAG versus agentic search (Recommended)

Some questions cannot be answered with one search. Knowing when to let a model search step by step saves you from building an index that cannot keep up.

Agentic search loops search, read and decide until it has enough, with a cap on the steps (amber is the model deciding).

Agentic search gives the model search tools and lets it loop: search, read, decide what to look for next, stop when it has enough. A cap on the number of steps keeps it from wandering. Coding assistants work this way. Cline's Nick Baumann explained in May 2025 why Cline reads files and follows imports instead of indexing: "An index, by definition, is a snapshot frozen in time."

An index is a snapshot that is fast and predictable; exploring is live and better at multi-step questions, but slower and costlier.

The trade is plain. An index answers in one fast, cheap, predictable step, but only from what it held at the last sync. Exploring is live and handles multi-step questions, but it is slower, costs more per question and varies from run to run. Hollin's documents change weekly or monthly and its questions come in volume, so it indexes them. Module 10 adds an agentic loop for the hard multi-step questions.

5. Hybrid strategies and three deciding variables (Recommended)

Real systems rarely pick one option. They pick per source, and route each question to the right one, which is why Grace wants the decision written down.

Three variables decide most cases: how big a source is, how often it changes, and how varied its questions are; the help centre (blue) is large, weekly and varied.

Three variables decide most cases. Corpus size: does it fit a prompt? Update frequency: does it change faster than you can re-index? Query diversity: are questions repeated or varied? A fourth fact, whether the data is structured, decides how a live lookup is done.

The decision runs in order: too fast to index means a live lookup, small and repeated means send it whole, otherwise RAG.

python

# Kwame's estimates for Hollin's full systems (simulated):
# (structured, size in tokens, minutes between changes, question variety)
SOURCES = {
    "help-centre":    (False,   600_000,  7 * 24 * 60, "varied"),
    "manuals":        (False, 2_000_000, 30 * 24 * 60, "varied"),
    "service-portal": (False,    40_000, 30 * 24 * 60, "varied"),
    "ticket-system":  (False, 5_000_000,           60, "varied"),
    "parts-shop":     (True,      3_000, 30 * 24 * 60, "varied"),
    "case-database":  (True,  1_000_000,            1, "repeated"),
}
REINDEX_MINUTES = 15   # the fastest the indexer can refresh (assumed)

def decide(structured, size, changes_every, variety):
    if changes_every < REINDEX_MINUTES:
        return "tool call" if structured else "live search"
    if size < 20_000 and variety == "repeated":
        return "long context"
    return "RAG"

for name, variables in SOURCES.items():
    print(f"{name:15} {decide(*variables)}")

Code walkthrough

SOURCES holds one row per Hollin source: structured or not, size, minutes between changes, and whether questions repeat. The sizes are simulated estimates.
decide applies the rules in order: too fast to index means a live lookup; small and repeated means send it whole; everything else is RAG.
Five sources print RAG; case-database prints tool call. The parts price list is structured but changes monthly, so it is indexed as a table.

Run on Hollin's sources (sizes simulated), decide() sends five to RAG and the case database to a live tool call.

One message can need both paths: a router sends the cover question to retrieval and the repair question to the tool.

A router sends each question, or each part of it, down the right path. Ellie's message needs both: retrieval for the cover, the tool for case 30540. The second hybrid is Self-Route from the opening case, which mixes RAG and long context per question.

Hollin's Self-Route: three documents first, Ellie's 16 visible documents only when the model says unanswerable, and a clear 'none' if the provider fails twice.

python

RAG_PROMPT = ("Answer from the documents below and cite their IDs in square brackets. Write unanswerable "
              "if the query can not be answered based on the provided text.\n\n{docs}\n\nQuestion: {q}")
LC_PROMPT = ("Answer from the documents below and cite their IDs in square brackets. If they do not "
             "answer the question, say you do not know.\n\n{docs}\n\nQuestion: {q}")

def self_route(q):
    """Try RAG first; fall back to the long context only when the model says unanswerable."""
    docs = context(retrieve(q))
    reply, reason = ask(llm, RAG_PROMPT.format(docs=docs, q=q))
    used = count(docs)
    if reply is not None and reply.text.strip() and "unanswerable" not in reply.text.lower():
        return {"route": "RAG", "tokens": used, "answer": reply.text}
    if reply is None:
        return {"route": "none", "tokens": used, "answer": None, "reason": reason}
    reply, reason = ask(llm, LC_PROMPT.format(docs=long_context, q=q))
    used += count(long_context)
    if reply is None:
        return {"route": "none", "tokens": used, "answer": None, "reason": reason}
    return {"route": "long context", "tokens": used, "answer": reply.text}

for q in [STANDALONE, "how do I descale the X300 Pro"]:
    r = self_route(q)
    print(f"{r['route']:12} {r['tokens']:5} tokens | {q}")
    print("   ", r["answer"] or r["reason"])

Code walkthrough

RAG_PROMPT carries the paper's own instruction, "Write unanswerable if...". LC_PROMPT is the fallback and asks for "I do not know" instead.
self_route asks with three documents. A non-empty reply without "unanswerable" is kept.
An empty reply or "unanswerable" retries with all 16 visible documents; two errors return route none with the reason.
Look for Ellie's question staying on RAG with [pol-warranty-2026] cited, and the X300 Pro descaling question (nothing in the corpus answers it) falling through to long context, which should then say it does not know.

In production

Li and colleagues answered between 57% (GPT-4O) and 77% (Gemini-1.5-Pro) of queries on the cheap path (their Table 1), cutting cost by 65% and 39% with comparable scores. Two cautions carry over. The fallback must respect permissions: "the whole context" means what this user may see. And "answerable" is not proof: Module 9 checks citations, Module 11 measures each route. Some providers discount a repeated prompt prefix (prompt caching), which narrows the gap for long context over fixed documents; check your provider's pricing page.

Li et al. (2024) report Self-Route cut cost by 65% on Gemini-1.5-Pro and 39% on GPT-4O, with comparable scores to long context.

What you added to Hollin Answers

A measured comparison of long context and RAG on Ellie's question: 784 against 189 tokens.
The 8 October update applied as record changes, with no retraining.
case_status, a tool that reads only the logged-in customer's cases.
A written decision per source: five to RAG, the case database to a live tool call.
A small Self-Route that pays for long context only when retrieval is not enough.

Check yourself

The slice fits the window easily. Give two reasons to use RAG anyway.

Cost grows with every document (four times more here), and long context sends both the current and the archived terms.

A colleague wants to fine-tune the model on the help centre so it "knows" the warranty. What goes wrong on 8 October?

An index changes three records; a fine-tuned model needs another training run and may still repeat SB-114.

Why does vector search fail on "how many repairs are booked"?

It returns similar-looking rows, not a count, and its copy goes stale. SQL returned 2.

In case_status, why does the customer name come from CURRENT_CUSTOMER and not from the model?

The model can be tricked into asking for any case. Filtering on the login keeps other customers' cases unreadable.

Which of the three variables sends the case database to a tool call?

Update frequency: it changes every minute, faster than the indexer can refresh.

Exercise

Grace adds two sources (simulated). Her escalation checklist is about 1,500 tokens, changes once a year, and agents ask it the same few questions. The community forum is about 300,000 tokens, changes daily, and gets varied questions. Predict each decision, then run the check. Then find the smallest change to the forum's numbers that makes it a live search.

python

SOURCES["escalation-checklist"] = (False, 1_500, 365 * 24 * 60, "repeated")
SOURCES["community-forum"] = (False, 300_000, 24 * 60, "varied")

assert decide(*SOURCES["escalation-checklist"]) == "long context"
assert decide(*SOURCES["community-forum"]) == "RAG"
assert decide(False, 300_000, 10, "varied") == "live search"
print("checks passed")

Code walkthrough

Two new rows go into SOURCES; each assert stops with an error if decide returns something else.
The last line is the answer to the second part: a forum that changes every 10 minutes, faster than the 15-minute re-index, needs a live search.

Key terms

Term	Meaning
Long context	Sending every document the user may see with each question
Context window	The most tokens a model can read at once
Self-Route	Ask with retrieved chunks first; use long context only when the model says "unanswerable"
Fine-tuning	Further training on your examples, which changes the model's weights
Structured data	Rows with exact fields, answered by queries
Tool calling	The model requests a function; your code runs it and returns the result
Agentic search	The model searches, reads and decides in a capped loop
Router	Sends each question, or part of one, to the right path

Sources

Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky, "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach", EMNLP 2024 Industry Track, November 2024. https://aclanthology.org/2024.emnlp-industry.66/ (arXiv version: https://arxiv.org/abs/2407.16833)
Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha (Microsoft), "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs", arXiv, revised 30 January 2024. https://arxiv.org/abs/2312.05934
Nick Baumann (Cline), "Why Cline Doesn't Index Your Codebase (And Why That's a Good Thing)", 27 May 2025. https://cline.bot/blog/why-cline-doesnt-index-your-codebase-and-why-thats-a-good-thing
Groq, "Supported models" and "Rate limits"; prices and the context window as recorded in course-guide/02 (checked 9 October 2026). https://console.groq.com/docs/models