Incident: Search answers that cited the wrong source

Module 1 · Incident file · About 30 minutes

What happened

AI search tools are retrieval systems: they search the web, read what they find and answer with citations. On 6 March 2025 Klaudia Jaźwińska and Aisvarya Chandrasekar of the Tow Center for Digital Journalism published a test of eight of them in the Columbia Journalism Review, "AI Search Has a Citation Problem" (CJR, 2025). It followed their earlier study of ChatGPT, published in November 2024.

The method was simple to check. They took ten articles from each of 20 news publishers, gave each tool a direct excerpt, and asked for the article's headline, publisher, date and URL: 1,600 queries in all. An answer counted as correct only when the article, the publisher and the URL were all right.

"Collectively, they provided incorrect answers to more than 60 percent of queries." Perplexity answered 37 percent incorrectly; Grok 3, 94 percent. ChatGPT "incorrectly identified 134 articles" but "signaled a lack of confidence just fifteen times out of its two hundred responses". The authors found the tools "generally bad at declining to answer questions they couldn't answer accurately". More than half of Gemini's and Grok 3's responses "cited fabricated or broken URLs"; for Grok 3, "154 citations led to error pages". DeepSeek misattributed the source 115 times out of 200. The paid tiers, Perplexity Pro and Grok 3, answered more prompts correctly than the free ones but "paradoxically also demonstrated higher error rates": the authors put it down to a "tendency to provide definitive, but wrong, answers" rather than declining.

The cost is trust, on both sides: readers sent to a page that does not exist or to the wrong publisher, and publishers whose work is credited to someone else. Only OpenAI and Microsoft replied to the authors, and neither addressed the specific findings. The authors note that answers vary between runs and that they recorded one answer per excerpt.

Why it happened

The study measures outcomes; it does not say how each product builds its citations, and this file does not guess. What it shows is the shape Lesson 1 warned about. A citation is text the model writes, like any other text. Unless code compares it with what was actually retrieved, it can name a page that was never fetched, a page that does not exist, or a real page that says something else.

The second half is the missing "I don't know". ChatGPT hedged 15 times in 200 answers while getting 134 wrong. A system that always answers, and always attaches a source, turns every retrieval miss into a confident, cited error. The source line makes the answer look checked, which is exactly what Magesh and colleagues called misgrounded in Lesson 2.

The same failure at Hollin

Kwame's August prototype showed each answer with a "Source" link built from whatever ID the model wrote. Nothing compared it with the retrieved documents. This replay uses the three documents the Module 1 lab retrieves for Ellie's question, and four scripted replies of the kinds the study found, so it needs no key and gives the same output every time.

def doc(id, type, title, updated, text, acl=("public",), products=(), status="current"):
    """The shared data's record, cut to the fields this replay reads."""
    return {"id": id, "title": title, "text": text, "status": status}

DOCS = [
    doc("pol-warranty-2026", "policy", "Warranty terms (2026)", "2026-03-01",
        "Every Hollin appliance has a 2-year warranty from the date of purchase. Blender motors on the X200 "
        "and X300 Pro are covered for 5 years. The warranty covers manufacturing faults. It does not cover "
        "blade wear, cracked jars, limescale damage or accidental damage. You need proof of purchase to claim."),
    doc("pol-warranty-2024", "policy", "Warranty terms (2024, archived)", "2024-02-10",
        "Every Hollin appliance has a 1-year warranty from the date of purchase. Blender motors are covered "
        "for 2 years. The warranty covers manufacturing faults and cracked jars. You need proof of purchase "
        "to claim.", status="archived"),
    doc("man-x200-errors", "manual", "X200 error codes", "2025-06-01",
        "E4 means the jar lid is not locked: twist the lid until it clicks. E7 means the motor has overheated: "
        "unplug the blender for 20 minutes. E9 means a sensor fault: contact Hollin support.", products=["X200"]),
    doc("man-x200-specs", "manual", "X200 Blender specifications", "2025-06-01",
        "The X200 blender has a 1,200 W motor, a 2-litre Tritan jar and three speeds. High speed reaches "
        "22,000 RPM. Model number HL-X200-UK.", products=["X200"]),
]
BY_ID = {d["id"]: d for d in DOCS}
RETRIEVED = ["pol-warranty-2026", "man-x200-errors", "man-x200-specs"]   # Ellie's top 3 in the Module 1 lab

Code walkthrough

• Four documents from the shared data, texts unchanged. The archived 2024 terms are in the corpus but not among Ellie's three.
• RETRIEVED is what the lab's retrieve returned for Ellie's question, so this file needs no embedding model.

import re

REPLIES = {   # scripted: the kinds of reply the Tow Center study found, written for Hollin
    "grounded": "Yes. X200 motors are covered for 5 years [pol-warranty-2026]. "
                "E7 means the motor overheated: unplug it for 20 minutes [man-x200-errors].",
    "made-up source": "Yes. X200 motors are covered for 5 years [pol-warranty-2025].",
    "never retrieved": "Yes. Cracked jars are covered too [pol-warranty-2024].",
    "misread source": "E7 means the motor overheated: unplug it for 30 minutes [man-x200-errors].",
}

def prototype(reply):
    """August's display: trust whatever ID the model wrote and link to it."""
    links = [f"hollin.example/docs/{c}" + ("" if c in BY_ID else "  (page not found)")
             for c in re.findall(r"\[([a-z0-9-]+)\]", reply)]
    return {"shown to Ellie": True, "links": links}

for name, reply in REPLIES.items():
    print(f"{name:16}", prototype(reply))

Code walkthrough

• REPLIES are scripted, not model output: one grounded answer and three failures.
• prototype builds a link for every bracketed ID and shows every answer. Only the made-up ID visibly fails, and only when someone clicks it.

You should see:

grounded         {'shown to Ellie': True, 'links': ['hollin.example/docs/pol-warranty-2026', 'hollin.example/docs/man-x200-errors']}
made-up source   {'shown to Ellie': True, 'links': ['hollin.example/docs/pol-warranty-2025  (page not found)']}
never retrieved  {'shown to Ellie': True, 'links': ['hollin.example/docs/pol-warranty-2024']}
misread source   {'shown to Ellie': True, 'links': ['hollin.example/docs/man-x200-errors']}

All four reach Ellie. The made-up ID is Hollin's broken URL. The archived terms are a real page, so the link works, and it tells her cracked jars are covered, which has not been true since 1 March 2026. The misread answer cites the right page and doubles the waiting time. None of them says "I don't know".

The fix

Check every reply in code before anyone sees it, and replace a failed reply with an honest hand-over. Two checks: the lab's check_citations (every ID must be one of the retrieved, current documents), and a new, deliberately simple support check: every number in a cited sentence must appear in the document it cites.

def check_citations(text, retrieved):
    """The Module 1 lab's check, cut to text and IDs: cited IDs must be retrieved and current."""
    cited = re.findall(r"\[([a-z0-9-]+)\]", text)
    problems = [] if cited else ["no citation"]
    for c in cited:
        if c not in retrieved:
            problems.append(f"{c}: never retrieved")
        elif BY_ID[c]["status"] != "current":
            problems.append(f"{c}: {BY_ID[c]['status']}")
    return problems

def numbers_supported(text):
    """Every number in a cited sentence must also appear in each document that sentence cites."""
    problems = []
    for sentence in re.split(r"(?<=[.!?])\s+", text):
        cited = [c for c in re.findall(r"\[([a-z0-9-]+)\]", sentence) if c in BY_ID]
        claim = set(re.findall(r"\d+", re.sub(r"\[[^\]]*\]", "", sentence)))
        for c in cited:
            missing = claim - set(re.findall(r"\d+", BY_ID[c]["text"]))
            if missing:
                problems.append(f"{c} does not say {sorted(missing)}")
    return problems

Code walkthrough

• check_citations is the lab's rule, taking the retrieved IDs as a list.
• numbers_supported splits the reply into sentences, removes the bracketed IDs, and compares the numbers left with the numbers in each cited document. "X200" counts as 200, which the cited documents also contain.

HANDOVER = "I can pass your question to a member of the support team."

def gate(reply, retrieved=RETRIEVED):
    """Show a reply only if both checks pass; otherwise hand over, and keep the reasons for the log."""
    problems = check_citations(reply, retrieved) + numbers_supported(reply)
    return {"shown": HANDOVER if problems else reply, "problems": problems}

for name, reply in REPLIES.items():
    out = gate(reply)
    print(f"{name:16}", "shown as written" if not out["problems"] else f"handed over: {out['problems']}")

Code walkthrough

• HANDOVER is the course's one offer sentence; Module 9 keeps it in the package as answer.HANDOVER.
• gate runs both checks; any problem replaces the reply with HANDOVER and keeps the reasons for the log.

You should see:

grounded         shown as written
made-up source   handed over: ['pol-warranty-2025: never retrieved']
never retrieved  handed over: ['pol-warranty-2024: never retrieved']
misread source   handed over: ["man-x200-errors does not say ['30']"]

Three bad answers become hand-overs and the grounded one goes through. The number check is crude: it misses a wrong word and would flag a correct sum. Module 9 builds the proper claim check. But it costs nothing to run and catches the cheapest kind of misreading.

How you would have caught it

The Tow Center's method is itself the test: known questions, known right sources, and a score for whether each answer names them. Hollin can run the same thing on every commit with scripted bad replies, and in production by logging every gate decision and watching the share handed over.

def test_gate():
    assert gate(REPLIES["grounded"])["problems"] == []
    for name in ("made-up source", "never retrieved", "misread source"):
        assert gate(REPLIES[name])["shown"] == HANDOVER, name

test_gate()
log = [gate(r)["problems"] for r in REPLIES.values()]
print(f"handed over: {sum(bool(p) for p in log)} of {len(log)} replies")

Code walkthrough

• test_gate is a test in the lab's style: it passes silently, or stops at the first failure with its name.
• log keeps every gate decision; in production the same count per day becomes an alert when it jumps.

You should see handed over: 3 of 4 replies. A rising hand-over rate means retrieval or the model got worse; a rate of zero for weeks means the checks may have stopped running.

Key points

• A citation is text the model writes. Check it against what was retrieved before showing it; the Tow Center found more than 60 percent of answers misidentified their source.
• A link that resolves is not a source that supports the claim: check status and, at least crudely, content.
• A system that never says "I don't know" turns every miss into a confident, cited error. Make the hand-over the default when a check fails.
• Test the checks with scripted bad replies, and log every gate decision so you can see the rate change.

Sources

• Klaudia Jaźwińska and Aisvarya Chandrasekar, Tow Center for Digital Journalism, "AI Search Has a Citation Problem", Columbia Journalism Review, 6 March 2025, fetched 12 October 2026. https://www.cjr.org/tow_center/we-compared-eight-ai-search-engines-theyre-all-bad-at-citing-news.php
• Varun Magesh and others, "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools", arXiv, 30 May 2024 (Lesson 2's case, for "misgrounded"). https://arxiv.org/abs/2405.20362