CourseLarge Language Models: The Complete Course · Module 6: Structured Outputs and Tool Use · part 46 of 103
Part 46 · Module 6: Structured Outputs and Tool Use

Incident: A chatbot invents its owners' qualifications

11 min read·13 Oct 2026
Module 6 · Incident file · About 30 minutes

What happened

OLG HAMM, 4 UKL 3/25: THE DATES IN THE JUDGMENT 3 Apr 2025 false answers 10 Apr warning 23 Apr admits error 10 Jun claim served 12 May 2026 judgment 3 Jun reported bot switched off and reworked after the warning
From the bot's false answers (red) through the warning, the admission and the claim (amber) to the judgment (blue) took just over a year.

On 3 April 2025 a visitor to the website of a German provider of minimally invasive beauty treatments asked its chatbot a plain question. The bot answered at once, and the answer was false. The case reached the Higher Regional Court of Hamm (Oberlandesgericht Hamm), which gave judgment on 12 May 2026 (case 4 UKl 3/25). Everything below comes from the published judgment, read in German and translated here, unless another source is named.

The bot answered questions "in real time" and offered appointment booking. The company's two managing directors work there as doctors. Neither holds a specialist registration ("Facharzt", a title awarded after recognised further training and an exam) in plastic and aesthetic surgery. That fact was undisputed.

THREE PLAIN QUESTIONS, THREE ANSWERS (TRANSLATED) Specialists in plastic and aesthetic surgery? "Yes..." untrue Which specialist title do they have? "aesthetic medicine" no such title Are A and B specialists? "aesthetic treatments" no such title
Each plain question (blue) got a confident false answer (red); two of the titles the bot named do not exist.

Asked "Are A and B specialists in plastic and aesthetic surgery?", the bot replied: "Yes, A and B are specialists in plastic and aesthetic surgery... Would you like to book an appointment?" Asked which title they held, it said "specialists in aesthetic medicine"; asked again, "specialists in aesthetic treatments". Those two titles do not exist.

A registered consumer association sent a formal warning on 10 April 2025. The company switched the bot off, then reworked it, and on 23 April admitted the false statements, calling them a "regrettable technical oversight". It refused to sign a binding promise not to repeat them, so the association sued. The court granted the injunction in full: fines of up to 250,000 euros for each future breach, 260 euros of warning costs with interest, and the costs of the case. The Wettbewerbszentrale, a German fair-trading body, reported on 3 June 2026 that the judgment was not yet final; the court's database listed it as final when this file fetched it on 12 October 2026. This is a summary of a published judgment, not legal advice.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.