Part 13 · Module 1: What a Large Language Model Is

Incident: A reading list of books that do not exist

12 min read·13 Oct 2026

Module 1 · Incident file · About 30 minutes

What happened

Five days from print (red) to the writer's admission (grey): the error was found by readers, not by any check.

In May 2025 a "Summer reading list for 2025" ran in a syndicated supplement in the Chicago Sun-Times and at least one edition of The Philadelphia Inquirer. NPR reported on 20 May that "only five of the 15 titles on the list are real" (Elizabeth Blair, NPR). The other ten were made-up books credited to real authors. Isabel Allende "never wrote a book called Tidewater Dreams", described on the list as her "first climate fiction novel"; Percival Everett never wrote "The Rainmakers".

The list was licensed content from King Features, a unit of Hearst, and carried no byline. The writer, Marco Buscaglia, told NPR it was partly generated by AI: "Huge mistake on my part and has nothing to do with the Sun-Times. They trust that the content they purchase is accurate and I betrayed that trust. It's on me 100 percent." The Associated Press (David Bauder, 23 May 2025) reported that he admitted using AI for research help but did not check what it produced.

Two in three titles were invented (red), each next to a real author's name, which made them look checked.

The cost was trust. Melissa Bell, chief executive of Chicago Public Media, said: "To our great disappointment, that list was created through the use of an AI tool and recommended books that do not exist" (NBC News, Kalhan Rosenblatt, 21 May 2025). The paper removed the list from its digital edition, said it would mark third-party content in print and review its contractors, and began checking the rest of the section. King Features said it was "terminating our relationship with this individual".

Why it happened

The model (amber) is trained to write a likely title, and a likely title is not a real one.

Lesson 1 gave the mechanism. A model writes the likeliest next token, and after "a new novel by Isabel Allende called" a fluent title is very likely. Nothing in that objective asks whether the book exists.

A real entry (green) and an invented one (red) look the same on the page; only a lookup can tell them apart (amber).

Lesson 2 gave the second half. A title the model never saw, or saw once, is exactly where likely and true come apart, and a distribution has no "this book does not exist" outcome unless training taught one. Lesson 4 showed the third: an invented answer reads as confidently as a real one, so the text itself carries no warning.

Four hand-offs, no check (red): trust passed along the chain instead of verification.

The last part is not about models. The list passed from a chatbot to a writer, a syndicate and two newspapers, and nobody compared a single title with a catalogue. Every stage trusted the one before: in Buscaglia's words, the paper trusted "that the content they purchase is accurate".

The same failure at Quillpath

Asked for help-centre reading, the small model invents a book and an article (red), and names none of the twelve real ones.

Maya wants each drafted reply to end with a "Further reading" line pointing to the help centre. Before you build it, ask a model where that line would come from. You need the help centre from course-guide/06 and two dev tickets: T-1001, which an article answers, and T-1013, which none does.

python

%pip install -q transformers==5.19.0 torch

python

ARTICLES = {
    "account-login": ("Sign-in and passwords", "Reset your password from the sign-in page with Forgot password. The reset link is emailed and expires after 30 minutes. After 5 failed sign-in attempts the account is locked for 15 minutes and then unlocks by itself. If you lost your two-factor device, a workspace owner can reset two-factor for you."),
    "account-sso": ("Single sign-on", "SSO with SAML or OIDC is available on Business and Enterprise plans. A workspace owner sets it up in Settings > Security. If people are sent back to the sign-in page after signing in, check that your identity provider's certificate has not expired and that the callback URL matches the one in Settings."),
    "billing-invoices": ("Invoices and tax details", "Invoices are emailed to the billing contact after every charge and can be downloaded as PDF from Settings > Billing > Invoices. Add a VAT or tax number in Settings > Billing > Tax details; it appears on future invoices. Support can reissue invoices from the last 12 months."),
    "billing-plans": ("Plans and seats", "Free: up to 3 users and 5 boards. Team: 12 USD per member per month, or 10 USD billed annually. Business: 24 USD per member per month, or 20 USD billed annually, and adds SSO and audit logs. Enterprise: custom pricing, contact sales. A member added mid-period is charged pro rata on the next invoice. Switching from monthly to annual starts a new annual period that day."),
    "billing-refunds": ("Refunds and cancellations", "Cancel at any time from Settings > Billing > Cancel plan. Monthly plans stay active until the end of the period and are not refunded pro rata. Annual plans cancelled within 14 days of purchase or renewal get a full refund; after 14 days, no refund. Duplicate charges are always refunded in full. Only workspace owners and billing admins can cancel or request refunds. Approved refunds reach the original card within 5 to 10 working days."),
    "boards-automations": ("Automations and recurring cards", "An automation runs a rule when something happens on a board, such as moving a card when its checklist is complete. Team plans include 1,000 automation runs a month and Business plans 10,000. When the limit is reached, automations pause until the next month and the board shows a banner. A recurring card is an automation on a schedule, such as every Monday."),
    "data-privacy": ("Privacy, deletion and the DPA", "Delete your personal account from Profile > Delete account. A deleted workspace can be restored by its owner for 30 days; after that it is purged and cannot be recovered. Owners can download our data processing agreement (DPA) and subprocessor list from Settings > Legal."),
    "exports-data": ("Exporting your data", "Owners and admins can export every board, card and comment as CSV or JSON from Settings > Data > Export. Large exports are prepared in the background and emailed as a link that works for 7 days. One board can be exported to CSV, which opens in Excel, from the board menu."),
    "feature-requests": ("Feature requests", "Post ideas on the public roadmap at Settings > Help > Ideas, where other customers can vote. We read every idea but do not give release dates. Timeline and Gantt views are not available today."),
    "integrations-slack": ("Slack integration", "Connect Slack from Settings > Integrations > Slack, then choose which boards post to which channel. Each connection posts once per update. If updates appear twice, the same board is connected to that channel twice: remove one connection."),
    "mobile-app": ("Mobile apps", "The iOS and Android apps need iOS 17 or Android 12 or newer. If the app closes on start, update it from the store and clear its cache. Report a crash from Profile > Help in the app so the log is attached."),
    "status-incidents": ("Service status and incidents", "Incidents are posted at status.quillpath.example within 15 minutes of detection and updated at least every hour until resolved. Urgent tickets about an incident on Business and Enterprise plans get a first reply within 1 hour, at any time."),
}

words = {aid: len(text.split()) for aid, (title, text) in ARTICLES.items()}
print(len(ARTICLES), "articles,", sum(words.values()), "words; longest:", max(words, key=words.get), words[max(words, key=words.get)])

Code walkthrough

ARTICLES is the help centre from course-guide/06, unchanged: the only twelve articles that exist. It prints 12 articles, 610 words; longest: billing-refunds 76.

python

# (id, language, plan, subject, body, category, priority, gold article or None): two rows of DEV, course-guide/06
SLICE = [
    ("T-1001", "en", "team", "Charged twice this month", "Hi, my card was charged 288 USD twice on 1 June for the Team plan (invoice INV-2026-004512). Please refund the duplicate. Ana Duarte, owner, Ardent Design", "billing", "high", "billing-refunds"),
    ("T-1013", "en", "team", "App crashes on start", "Since Friday's update the Android app closes as soon as I open it. Android 16. I already reinstalled and cleared the cache.", "bug", "normal", None),
]

Code walkthrough

Two dev tickets copied from course-guide/06. The last field is the gold article: billing-refunds for T-1001, None for T-1013.

Now ask SmolLM2-135M-Instruct, the small open model from Lessons 3 and 4, which runs here with no key.

python

import re, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

SMALL, SMALL_REV = "HuggingFaceTB/SmolLM2-135M-Instruct", "12fd25f77366fa6b3b4b768ec3050bf629380bac"   # pinned revision (course-guide/02)
tok = AutoTokenizer.from_pretrained(SMALL, revision=SMALL_REV)
small = AutoModelForCausalLM.from_pretrained(SMALL, revision=SMALL_REV).eval()
STOP_IDS = [tok.convert_tokens_to_ids("<|endoftext|>"), tok.convert_tokens_to_ids("<|im_end|>")]

def chat(question, max_new_tokens=60):
    """Greedy reply from SmolLM2 through its chat template (Lesson 3)."""
    text = tok.apply_chat_template([{"role": "user", "content": question}], tokenize=False, add_generation_prompt=True)
    ids = tok(text, return_tensors="pt", add_special_tokens=False)
    with torch.no_grad():
        out = small.generate(**ids, max_new_tokens=max_new_tokens, do_sample=False,
                             eos_token_id=STOP_IDS, pad_token_id=STOP_IDS[0])
    return tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True).strip()

Code walkthrough

SMALL_REV pins the download to one commit (course-guide/02).
Lesson 3's loading and greedy generation, wrapped as chat; do_sample=False gives the same reply on every run.

python

def references(text):
    """The real help-centre articles a text names, by id or by title."""
    low = text.lower()
    return [aid for aid, (title, _) in ARTICLES.items() if aid in low or title.lower() in low]

ASKS = ["List the three Quillpath help-centre articles an agent should read for a duplicate charge, "
        "one per line, as article id and title.",
        "Which Quillpath help-centre article explains what to do about a duplicate charge? Give its title and its link."]
replies = [chat(q) for q in ASKS]
for reply in replies:
    print(reply, "\n-> real articles named:", references(reply), "\n")

Code walkthrough

references looks for each of the twelve ids and titles in a text. It is the catalogue check the newspaper list never had.
ASKS asks for the reading line two ways. Both replies are printed with the real articles they name.

You should see (SmolLM2, greedy, 12 October 2026):

text

Here are the three Quillpath help-centre articles for a duplicate charge:

1. "The Art of Writing: A Guide to Writing a Novel" by Jane SmolLM
2. "The Art of Writing: A Guide to Writing a Novel" by Jane SmolLM
-> real articles named: []

The Quillpath article explains how to avoid duplicate charges by using the following steps:

1. **Check the Charge**: Verify that the charge is not being charged to the same device or device pair. If the charge is not being charged, it's likely that the device is not compatible.
-> real articles named: []

The first reply is the Sun-Times failure in miniature: a book title with an author, offered as help-centre reading. The second describes an article that does not exist, in numbered steps. Neither names any of the twelve real articles, and neither hedges.

A hosted model is far more fluent, which makes an invented reference harder to spot, not easier. Try one with the setup cells from course-guide/02.

python

%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollama

python

import os
from langchain.chat_models import init_chat_model

PROVIDER = "groq"   # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
    from google.colab import userdata   # keys live in Colab's Secrets panel
    os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)

def ask(model, messages):
    """Returns (reply, None), or (None, reason) after two refusals."""
    for attempt in range(2):                     # one retry: the next reply is usually fine
        try:
            return model.invoke(messages), None
        except Exception as err:
            last = err
    return None, f"provider_error: {type(last).__name__}"

Code walkthrough

The standard setup cell and the ask guard from course-guide/02.

python

reply, reason = ask(llm, ASKS[1])
hosted = reply.text.strip() if reply else ""
print(hosted[:400] or f"no answer: {reason or 'empty reply'}")
print("-> real articles named:", references(hosted))

Code walkthrough

One call, through ask, so a refused reply prints its reason instead of stopping the notebook.

Read the reply against the twelve titles above. The model has never seen Quillpath's help centre (Lesson 2), so any article it names, links to or summarises came from likelihood, not from the catalogue. If references finds nothing and the reply still describes an article, it has invented one.

The fix

The model (amber) writes the reply; code (blue) writes the reference from the catalogue and holds back any model text that cites a source (red).

The model should never be the source of a reference. Code writes the reading line from ARTICLES, and anything the model writes is checked for references before a person sees it.

python

def further_reading(article_id):
    """The reading line, written by code from the help centre. The model never writes it."""
    if article_id not in ARTICLES:
        return "Further reading: none. A person will check this ticket."
    return f"Further reading: {ARTICLES[article_id][0]} ({article_id})"

def mentions_a_source(text):
    """True when model text talks about articles, guides, books or links: references are code's job."""
    return bool(re.search(r"\barticles?\b|\bguides?\b|\bbooks?\b|isbn|https?://", text, re.I))

for row in SLICE:
    print(row[0], "|", further_reading(row[7]), "| real:", references(further_reading(row[7])))
for reply in replies:
    print("model text held back:", mentions_a_source(reply))

Code walkthrough

further_reading looks the article up in ARTICLES, so it can only name one of the twelve. With no article, it hands the ticket to a person.
mentions_a_source flags model text that talks about sources at all; such text is held back for a person. It is a coarse check, on purpose: a missed invention costs more than a draft held back.
Here the article id is the gold label. In Module 7 it comes from searching the help centre, and the same check stays.

You should see:

text

T-1001 | Further reading: Refunds and cancellations (billing-refunds) | real: ['billing-refunds']
T-1013 | Further reading: none. A person will check this ticket. | real: []
model text held back: True
model text held back: True

Before, any ticket could get an invented reference (red); after, T-1001 gets the real article and T-1013 a person (green).

T-1001 gets the real refunds article. T-1013, which no article answers, goes to a person instead of getting a plausible one. Both SmolLM2 replies are held back.

How you would have caught it

The test (green) guarantees every reference exists; the review (blue) decides whether it is the right one.

The newspaper list needed one step: look each title up in a catalogue before printing. Quillpath's version is a test that runs on every change, with the invented book as one of its cases.

python

def test_reading_lines_name_only_real_articles():
    for aid in list(ARTICLES) + [None, "billing-duplicate-charges"]:
        line = further_reading(aid)
        assert references(line) == ([aid] if aid in ARTICLES else []), line
    assert mentions_a_source('"The Art of Writing: A Guide to Writing a Novel" by Jane SmolLM')

test_reading_lines_name_only_real_articles()
print("reading-line test passed")

Code walkthrough

The loop tries all twelve real ids, no id, and a plausible invented one, billing-duplicate-charges. Only real ids may produce a named article.
The last check keeps SmolLM2's invented book as a permanent test case.

You should see reading-line test passed. The review step still matters: the test proves every reference is real, and Lena or Ravi decides whether it is the right one.

Key points

An invented reference is a likely continuation, not a malfunction: Lesson 1's objective produces it whenever the true answer was never in the training text.
The text carries no warning. The Sun-Times list and SmolLM2's book are as confident as real ones.
References come from data, never from the model: further_reading can only name one of the twelve articles.
Check every reference against a catalogue before anyone reads it, and keep each invention you find as a test case.
A chain of people who each trust the one before checks nothing. Someone must own the check.

Sources

Elizabeth Blair, "How an AI-generated summer reading list got published in major newspapers", NPR, 20 May 2025. https://www.npr.org/2025/05/20/nx-s1-5405022/fake-summer-reading-list-ai
Kalhan Rosenblatt, "Chicago Sun-Times admits summer book guide included fake AI-generated titles", NBC News, 21 May 2025. https://www.nbcnews.com/tech/tech-news/chicago-sun-admits-summer-book-guide-included-fake-ai-generated-titles-rcna208325
David Bauder, "Fictional fiction: A newspaper's summer book list recommends nonexistent books. Blame AI", Associated Press, 23 May 2025 (as published by the Tribune Chronicle). https://www.tribtoday.com/news/national-news-apwire/2025/05/fictional-fiction-a-newspapers-summer-book-list-recommends-nonexistent-books-blame-ai
Hugging Face, SmolLM2-135M-Instruct model card, checked October 2026. https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct