Lesson 1: What NLP is, and the inbox we will fix
52 min read·14 Oct 2026
Module 1 · Lesson 1 · About 45 minutes · Needs: basic Python (strings, lists, dictionaries, functions); no NLP
The situation
Monday 7 September 2026, project week 1. Rosa Mensah owns Tidewell Cycles, a bike shop and workshop in Bristol. Every email, web-form message and review lands in one inbox, read part-time by Ines Duarte. Since deliveries to Germany, Spain and France began on 1 July, the inbox has nearly doubled, and four in ten messages are not in English. Rosa has hired you to fix it. First: what is the problem, what does the inbox hold, and how good is a very simple answer? Tidewell is made up; the case below is real.
A real case, first
In January 1966 Joseph Weizenbaum of MIT published "ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine" in Communications of the ACM. ELIZA looked for keywords in what a person typed and turned the sentence round with fixed rules. Its best-known script, DOCTOR, imitated a psychotherapist.
ELIZA understood nothing, yet people talked to it as if it did. In Computer Power and Human Reason (1976) Weizenbaum wrote that this disturbed him. The lesson for Tidewell: keyword rules can look clever on the messages you tried, and looking clever is not being right. So this lesson writes ELIZA-style rules, then measures them on messages they were not written for.
What you will be able to do
- Explain what natural language processing is, using one inbox message.
- Match five common uses of NLP to Rosa's five jobs.
- Place rules, statistics, neural networks and LLMs on one timeline.
- Trace a message through the pipeline from raw text to a routing decision.
- Load the shared data, check its fingerprint and score a baseline.
1. What natural language processing is (Essential)
Every job Rosa wants done starts with text a person wrote. If a program cannot get from that text to something it can act on, nothing else works. The running example is a message from a customer, Megan Price, on 15 April 2026.
You read her sentence and know at once: the battery is failing, she is asking about the warranty, and she is not happy. Python knows none of that. To Python the sentence is a string: characters in a row, with no meaning attached.
Natural language processing (NLP) is the set of methods that turn text written by people into something a program can use: a label, a list of fields, a ranking, another text. For Megan's sentence that is a small record: a warranty fault, about a Volt 2 Step-Through, negative, for Dev in the workshop.
python
sentence = ("My e-bike (Volt 2 Step-Through) is only four months old and the battery "
"runs flat very quickly. Is this covered by the warranty?")
print(type(sentence).__name__, "of", len(sentence), "characters")
print(list(sentence[:10]))
words = sentence.split()
print(len(words), "words:", words[:6])
print("battery" in sentence, "Akku" in sentence)Code walkthrough
type(...).__name__names the type;lencounts characters.list(sentence[:10])shows the first ten characters: this is all Python holds.sentence.split()cuts at spaces; note'(Volt', bracket attached. Splitting text well is Module 2."battery" in sentenceis a substring test. It finds "battery" but not "Akku", German for battery.
text
str of 128 characters
['M', 'y', ' ', 'e', '-', 'b', 'i', 'k', 'e', ' ']
23 words: ['My', 'e-bike', '(Volt', '2', 'Step-Through)', 'is']
True FalseThat last line is the subject in miniature. A substring test matches characters, not meaning. NLP gets closer to meaning step by step, and measures how close each step gets.
2. The Tidewell inbox, the five jobs and the shared data (Essential)
You cannot improve an inbox you have not looked at.
Ines forwards repair bookings and warranty faults to Dev Malhotra in the workshop and product questions to Callum Reid, who runs the web shop. She answers the rest herself.
Spring months brought 200 to 270 messages; July brought 460. One part-time reader cannot keep up.
Rosa wants five jobs done: route every message to the right person; pull out the order number, product, fault and date; tell her each Monday what customers are unhappy about; suggest the right help-centre article while Ines replies; and handle German, Spanish and French.
All Tidewell data is simulated: made by code with a fixed seed, so every learner gets the same messages. Paste these four blocks from the shared-data page (course-guide/06) unchanged; you need not read them yet.
python
from datetime import date, timedelta
COURSE_DATE = date(2026, 10, 14) # facts, versions and model names checked on this day
DATA_AS_OF = date(2026, 8, 31) # the inbox snapshot: count days to this, never to the clock
WINDOW = (date(2026, 3, 2), date(2026, 8, 30)) # 26 weeks of messages, Monday to Sunday
SEED = 20260831
EVENTS = { # fixed dates the data and the lessons share
"spam_wave": (date(2026, 4, 6), date(2026, 4, 19)),
"battery_notice": date(2026, 6, 9), # email to Volt 2 owners about pack batch VB-0426
"eu_shipping": date(2026, 7, 1), # Germany, Spain and France
"summer_sale": (date(2026, 7, 15), date(2026, 7, 28)),
"carrier_change": date(2026, 8, 10), # ParcelNorth -> SwiftDrop
}
BATCH = "VB-0426"
STAFF = ["Rosa", "Ines", "Dev", "Callum", "Amara"]
PRODUCTS = [ # (sku, name, price in GBP, kind)
("TW-VOLT2-BLK", "Volt 2", 2199, "ebike"),
("TW-VOLT2-STP", "Volt 2 Step-Through", 2249, "ebike"),
("TW-HARB-GRN", "Harbour", 649, "bike"),
("TW-HKID-20R", "Harbour Kids 20", 329, "bike"),
("TW-CAUS-GRY", "Causeway", 1149, "bike"),
("TW-PANN-SET", "Commuter Pannier Set", 89, "accessory"),
("TW-BATT-500", "Volt 2 battery pack (500 Wh)", 549, "accessory"),
("TW-LOCK-DLK", "LockSure D-lock", 45, "accessory"),
("TW-LITE-BCN", "Beacon Light Set", 39, "accessory"),
("TW-SERV-BAS", "Basic Service", 65, "service"),
("TW-SERV-FUL", "Full Service", 129, "service"),
("TW-REPR-PUN", "Puncture Repair", 18, "service"),
("TW-DIAG-EBK", "E-bike Diagnostic", 45, "service"),
("TW-GIFT-CRD", "Gift Card", 50, "gift"), # sold from 25 to 200 GBP; 50 is the usual
]
INTENTS = ["order_status", "returns_refund", "repair_booking", "product_question",
"warranty_fault", "praise", "spam"]
LABELS = { # Ines's written definitions, used for every label in the course
"order_status": "Where an order is, delivery problems, missing items, invoices and order documents.",
"returns_refund": "Sending something back, exchanges without a fault, cancellations, refunds and wrong charges.",
"repair_booking": "Booking, moving or asking the price of a service or repair for wear and tear.",
"product_question": "Questions before or after buying: sizes, stock, range, compatibility, delivery areas.",
"warranty_fault": "A product that is faulty or arrived damaged, including the battery batch VB-0426.",
"praise": "Thanks, compliments and good reviews, with no request that needs action.",
"spam": "Unsolicited selling, phishing and scams.",
}
CUSTOMERS = [ # (first, last, gender, language, home town); all fictional
("Joan", "Hartley", "f", "en", "Clifton"), ("Oliver", "Bennett", "m", "en", "Bristol"),
("Amelia", "Clarke", "f", "en", "Bath"), ("Mohammed", "Khan", "m", "en", "Bristol"),
("Aisha", "Begum", "f", "en", "Bedminster"), ("Daniel", "O'Connor", "m", "en", "Bristol"),
("Priya", "Patel", "f", "en", "Swindon"), ("Arjun", "Sharma", "m", "en", "Bristol"),
("Grace", "Okafor", "f", "en", "Cardiff"), ("Emeka", "Eze", "m", "en", "Bristol"),
("Hannah", "Lewis", "f", "en", "Stroud"), ("Tom", "Hughes", "m", "en", "Gloucester"),
("Siobhan", "Murphy", "f", "en", "Bristol"), ("Liam", "Walsh", "m", "en", "Bath"),
("Mei", "Chen", "f", "en", "Bristol"), ("Wei", "Zhang", "m", "en", "Clifton"),
("Fatima", "Hussain", "f", "en", "Bristol"), ("Yusuf", "Ali", "m", "en", "Bedminster"),
("Zara", "Ahmed", "f", "en", "Taunton"), ("Jack", "Robinson", "m", "en", "Bristol"),
("Ella", "Morgan", "f", "en", "Cardiff"), ("Rhys", "Evans", "m", "en", "Cardiff"),
("Megan", "Price", "f", "en", "Weston-super-Mare"), ("Ben", "Carter", "m", "en", "Bristol"),
("Lucy", "Ward", "f", "en", "Bath"), ("Kofi", "Asante", "m", "en", "Bristol"),
("Ama", "Boateng", "f", "en", "Bristol"), ("Olusegun", "Adeyemi", "m", "en", "Swindon"),
("Funmi", "Adebayo", "f", "en", "Bristol"), ("Raj", "Gupta", "m", "en", "Gloucester"),
("Neha", "Kapoor", "f", "en", "Bristol"), ("Tomasz", "Nowak", "m", "en", "Bristol"),
("Agnieszka", "Kowalska", "f", "en", "Stroud"), ("Stefan", "Ionescu", "m", "en", "Bristol"),
("Ioana", "Popescu", "f", "en", "Bath"), ("Hiroshi", "Tanaka", "m", "en", "Bristol"),
("Lukas", "Becker", "m", "de", "Berlin"), ("Anna", "Schmidt", "f", "de", "Hamburg"),
("Jonas", "Weber", "m", "de", "München"), ("Lea", "Fischer", "f", "de", "Köln"),
("Mehmet", "Yılmaz", "m", "de", "Berlin"), ("Elif", "Demir", "f", "de", "Leipzig"),
("Felix", "Wagner", "m", "de", "Hamburg"), ("Sophie", "Müller", "f", "de", "München"),
("Javier", "García", "m", "es", "Madrid"), ("Lucía", "Fernández", "f", "es", "Sevilla"),
("Alejandro", "Martín", "m", "es", "Valencia"), ("Carmen", "Ruiz", "f", "es", "Madrid"),
("Mateo", "López", "m", "es", "Barcelona"), ("Valentina", "Torres", "f", "es", "Bilbao"),
("Youssef", "El Amrani", "m", "es", "Barcelona"), ("Nuria", "Puig", "f", "es", "Barcelona"),
("Hugo", "Martin", "m", "fr", "Paris"), ("Camille", "Dubois", "f", "fr", "Lyon"),
("Lucas", "Bernard", "m", "fr", "Nantes"), ("Chloé", "Lefèvre", "f", "fr", "Bordeaux"),
("Karim", "Benali", "m", "fr", "Lille"), ("Amina", "Diallo", "f", "fr", "Paris"),
("Théo", "Moreau", "m", "fr", "Lyon"), ("Léa", "Laurent", "f", "fr", "Nantes"),
]
print(len(PRODUCTS), "products,", len(CUSTOMERS), "customers,", len(INTENTS), "intents")
print("women:", sum(c[2] == "f" for c in CUSTOMERS), "men:", sum(c[2] == "m" for c in CUSTOMERS))
print({lang: sum(c[3] == lang for c in CUSTOMERS) for lang in ["en", "de", "es", "fr"]})
print((DATA_AS_OF - WINDOW[0]).days, "days from the first message day to DATA_AS_OF")Code walkthrough Block 1: dates, the fourteen products (prices in pounds), Ines's intent definitions (LABELS) and sixty customers.
text
14 products, 60 customers, 7 intents
women: 30 men: 30
{'en': 36, 'de': 8, 'es': 8, 'fr': 8}
182 days from the first message day to DATA_AS_OFpython
import random, re
# Templates: one dict per wording. Keys en, de, es, fr hold the same message in each language
# (English-only wordings have just "en"). {UPPER} slots become gold entities; {lower} slots do not.
T = {
"order_status": [
{"s": "neu", "en": "Hi, I placed an order on {DATE} (order {ORDER_ID}, item: {PRODUCT}) and it still says processing. When will it ship?",
"de": "Hallo, ich habe am {DATE} bestellt (Bestellung {ORDER_ID}, Artikel: {PRODUCT}) und der Status steht immer noch auf in Bearbeitung. Wann wird sie verschickt?",
"es": "Hola, hice un pedido el {DATE} (pedido {ORDER_ID}, artículo: {PRODUCT}) y sigue en preparación. ¿Cuándo lo enviaréis?",
"fr": "Bonjour, j'ai passé commande le {DATE} (commande {ORDER_ID}, article : {PRODUCT}) et elle est toujours en préparation. Quand sera-t-elle expédiée ?"},
{"s": "neg", "en": "Order {ORDER_ID} was due on {DATE} and nothing has arrived. The {carrier} tracking has not changed for three days. Where is my parcel ({PRODUCT})?",
"de": "Bestellung {ORDER_ID} sollte am {DATE} ankommen, aber es ist nichts da. Die Sendungsverfolgung von {carrier} hat sich seit drei Tagen nicht geändert. Wo ist mein Paket ({PRODUCT})?",
"es": "El pedido {ORDER_ID} tenía que llegar el {DATE} y no ha llegado nada. El seguimiento de {carrier} no cambia desde hace tres días. ¿Dónde está mi paquete ({PRODUCT})?",
"fr": "La commande {ORDER_ID} devait arriver le {DATE} et rien n'est arrivé. Le suivi {carrier} n'a pas bougé depuis trois jours. Où est mon colis ({PRODUCT}) ?"},
{"s": "neu", "en": "Could you send me the tracking link for order {ORDER_ID}? The confirmation email never came through to {email}.",
"de": "Könnten Sie mir den Tracking-Link für Bestellung {ORDER_ID} schicken? Die Bestätigung ist nie bei {email} angekommen.",
"es": "¿Me podéis enviar el enlace de seguimiento del pedido {ORDER_ID}? El correo de confirmación nunca llegó a {email}.",
"fr": "Pouvez-vous m'envoyer le lien de suivi de la commande {ORDER_ID} ? L'e-mail de confirmation n'est jamais arrivé sur {email}."},
{"s": "neg", "from": "2026-08-10", "w": 3, "en": "Third time {carrier} has said 'attempted delivery' when I was in all day. Order {ORDER_ID}. This never happened with your old courier."},
{"s": "neu", "en": "any update on {ORDER_ID}? need the {PRODUCT} before {DATE} if poss"},
],
"returns_refund": [
{"s": "neu", "en": "I'd like to return the {PRODUCT} from order {ORDER_ID}. It's unused and still in the box. How do I send it back?",
"de": "Ich möchte {PRODUCT} aus Bestellung {ORDER_ID} zurückgeben. Unbenutzt und noch originalverpackt. Wie schicke ich es zurück?",
"es": "Quiero devolver {PRODUCT} del pedido {ORDER_ID}. Está sin usar y en su caja. ¿Cómo lo envío?",
"fr": "Je voudrais retourner {PRODUCT} de la commande {ORDER_ID}. Il n'a pas servi et il est encore dans sa boîte. Comment le renvoyer ?"},
{"s": "neg", "en": "I sent the return for {ORDER_ID} back on {DATE} and still no refund of {MONEY}. How long does this take?",
"de": "Ich habe die Rücksendung für {ORDER_ID} am {DATE} abgeschickt und noch keine Erstattung über {MONEY} bekommen. Wie lange dauert das?",
"es": "Envié la devolución del pedido {ORDER_ID} el {DATE} y todavía no me habéis devuelto {MONEY}. ¿Cuánto tarda?",
"fr": "J'ai renvoyé la commande {ORDER_ID} le {DATE} et je n'ai toujours pas reçu le remboursement de {MONEY}. Combien de temps faut-il ?"},
{"s": "neu", "en": "The {KIDBIKE} is the wrong size for my daughter. Can I swap it or get a refund? Order {ORDER_ID}.",
"de": "{KIDBIKE} ist für meine Tochter die falsche Größe. Kann ich umtauschen oder bekomme ich das Geld zurück? Bestellung {ORDER_ID}.",
"es": "{KIDBIKE} es de una talla equivocada para mi hija. ¿Puedo cambiarlo o me devolvéis el dinero? Pedido {ORDER_ID}.",
"fr": "{KIDBIKE} n'est pas à la bonne taille pour ma fille. Puis-je l'échanger ou être remboursé ? Commande {ORDER_ID}."},
{"s": "neg", "en": "You charged me {MONEY} twice for order {ORDER_ID}. Please refund the duplicate payment."},
{"s": "neu", "en": "want to cancel order {ORDER_ID} before it ships, changed my mind about the {PRODUCT}. refund to my card pls"},
],
"repair_booking": [
{"s": "neu", "en": "Can I book a {SERVICE} for {DATE}? It's for my bike ({BIKE}). I can drop it at the workshop in the morning.",
"de": "Kann ich für den {DATE} einen Termin für {SERVICE} buchen? Es geht um mein Rad ({BIKE}). Ich kann es morgens in die Werkstatt bringen.",
"es": "¿Puedo reservar {SERVICE} para el {DATE}? Es para mi bici ({BIKE}). Puedo dejarla en el taller por la mañana.",
"fr": "Puis-je réserver {SERVICE} pour le {DATE} ? C'est pour mon vélo ({BIKE}). Je peux le déposer à l'atelier le matin."},
{"s": "neu", "en": "My bike ({BIKE}) needs a repair: {wear}. Can I bring it in or send it to you? I live in {LOC}.",
"de": "Mein Rad ({BIKE}) muss repariert werden: {wear}. Kann ich es vorbeibringen oder einschicken? Ich wohne in {LOC}.",
"es": "Mi bici ({BIKE}) necesita una reparación: {wear}. ¿Puedo llevarla o enviárosla? Vivo en {LOC}.",
"fr": "Mon vélo ({BIKE}) a besoin d'une réparation : {wear}. Puis-je l'apporter ou vous l'envoyer ? J'habite à {LOC}."},
{"s": "neu", "en": "Please can I move my {SERVICE} booking ({ticket}) from {DATE} to the following week? Something came up.",
"de": "Kann ich meinen Termin für {SERVICE} ({ticket}) vom {DATE} auf die Woche danach verschieben? Mir ist etwas dazwischengekommen.",
"es": "¿Puedo cambiar mi cita de {SERVICE} ({ticket}) del {DATE} a la semana siguiente? Me ha surgido algo.",
"fr": "Puis-je déplacer mon rendez-vous {SERVICE} ({ticket}) du {DATE} à la semaine suivante ? J'ai un empêchement."},
{"s": "neu", "en": "hiya, how much is a {SERVICE} and can u fit me in {DATE}? bike is a {BIKE}, cheers"},
{"s": "neu", "en": "I'd like to book the {SERVICE} for my {BIKE}. Is {MONEY} still the price? Any day after {DATE} works."},
],
"product_question": [
{"s": "neu", "en": "Does the {BIKE} come in a smaller frame? I'm 158 cm tall.",
"de": "Gibt es {BIKE} auch mit einem kleineren Rahmen? Ich bin 1,58 m groß.",
"es": "¿{BIKE} viene en una talla de cuadro más pequeña? Mido 1,58 m.",
"fr": "{BIKE} existe-t-il avec un cadre plus petit ? Je mesure 1,58 m."},
{"s": "neu", "en": "What range does the {EBIKE} get on one charge, and is the {ACCESSORY} compatible with it?",
"de": "Welche Reichweite hat {EBIKE} mit einer Akkuladung, und passt {ACCESSORY} dazu?",
"es": "¿Qué autonomía tiene {EBIKE} con una carga, y es compatible con {ACCESSORY}?",
"fr": "Quelle autonomie a {EBIKE} avec une charge, et {ACCESSORY} est-il compatible ?"},
{"s": "neu", "en": "Is the {PRODUCT} in stock? The website says {MONEY} but I can't add it to the basket.",
"de": "Ist {PRODUCT} auf Lager? Auf der Website steht {MONEY}, aber ich kann es nicht in den Warenkorb legen.",
"es": "¿Tenéis {PRODUCT} en stock? En la web pone {MONEY} pero no puedo añadirlo a la cesta.",
"fr": "Est-ce que {PRODUCT} est en stock ? Le site indique {MONEY} mais je n'arrive pas à l'ajouter au panier."},
{"s": "neu", "en": "Do you deliver to {LOC}? And do bikes arrive fully built or do I need to assemble them?"},
{"s": "neu", "en": "quick q: whats the difference between the {SERVICE} and the {SERVICE2}?"},
],
"warranty_fault": [
{"s": "neg", "en": "My e-bike ({EBIKE}) is only four months old and {fault}. Is this covered by the warranty?",
"de": "Mein E-Bike ({EBIKE}) ist erst vier Monate alt und {fault}. Ist das von der Garantie abgedeckt?",
"es": "Mi bici eléctrica ({EBIKE}) solo tiene cuatro meses y {fault}. ¿Lo cubre la garantía?",
"fr": "Mon vélo électrique ({EBIKE}) n'a que quatre mois et {fault}. Est-ce couvert par la garantie ?"},
{"s": "neg", "from": "2026-06-09", "w": 4, "en": "I got your email about battery batch {batch}. My pack has that code and {fault}. What happens now?",
"de": "Ich habe Ihre E-Mail zur Akku-Charge {batch} bekommen. Mein Akku hat diesen Code und {fault}. Wie geht es jetzt weiter?",
"es": "Recibí vuestro correo sobre el lote de baterías {batch}. Mi batería tiene ese código y {fault}. ¿Qué hago ahora?",
"fr": "J'ai reçu votre e-mail sur le lot de batteries {batch}. Ma batterie porte ce code et {fault}. Que se passe-t-il maintenant ?"},
{"s": "neg", "en": "The {PRODUCT} arrived with {damage}. I've attached photos. Can you send a replacement?",
"de": "{PRODUCT} kam mit {damage} an. Fotos sind im Anhang. Können Sie Ersatz schicken?",
"es": "{PRODUCT} llegó con {damage}. Adjunto fotos. ¿Podéis enviarme uno nuevo?",
"fr": "{PRODUCT} est arrivé avec {damage}. Je joins des photos. Pouvez-vous m'en envoyer un autre ?"},
{"s": "neg", "en": "Spoke snapped on my {BIKE} after two rides. That's not normal wear, surely? Bought {DATE}, order {ORDER_ID}."},
{"s": "neg", "en": "{EBIKE} battery dead again. Second pack in a month. I want this sorted properly this time."},
],
"praise": [
{"s": "pos", "en": "Just wanted to say thank you to {PERSON} in the workshop. My bike ({BIKE}) rides like new after the {SERVICE}.",
"de": "Ich wollte mich nur bei {PERSON} in der Werkstatt bedanken. Mein Rad ({BIKE}) fährt sich nach {SERVICE} wie neu.",
"es": "Solo quería dar las gracias a {PERSON} del taller. Mi bici ({BIKE}) va como nueva después de {SERVICE}.",
"fr": "Je voulais juste remercier {PERSON} de l'atelier. Mon vélo ({BIKE}) roule comme neuf après {SERVICE}."},
{"s": "pos", "en": "Love my new bike ({BIKE}). Fast delivery to {LOC} and it was easy to set up. Five stars.",
"de": "Ich liebe mein neues Rad ({BIKE}). Schnelle Lieferung nach {LOC} und leicht aufzubauen. Fünf Sterne.",
"es": "Me encanta mi nueva bici ({BIKE}). Entrega rápida a {LOC} y fácil de montar. Cinco estrellas.",
"fr": "J'adore mon nouveau vélo ({BIKE}). Livraison rapide à {LOC} et facile à monter. Cinq étoiles."},
{"s": "pos", "en": "Great service from {PERSON}, who sorted my problem in one email. Thank you!",
"de": "Super Service von {PERSON}, das Problem war mit einer E-Mail gelöst. Vielen Dank!",
"es": "Un servicio genial de {PERSON}, que me lo resolvió en un solo correo. ¡Gracias!",
"fr": "Super service de la part de {PERSON}, qui a réglé mon problème en un seul e-mail. Merci !"},
{"s": "pos", "en": "the {ACCESSORY} is brilliant, fits perfectly. cheers"},
{"s": "pos", "en": "Honestly the best bike shop in {LOC}. Friendly, quick and fair prices."},
],
"spam": [
{"s": "neu", "en": "Dear partner, we offer guaranteed first-page search ranking for your online store. Reply today for a {MONEY} discount."},
{"s": "neu", "en": "Exclusive partnership opportunity: invest in our crypto bike-sharing token and earn 300% returns. Contact {email} now."},
{"s": "neu", "en": "Your mailbox storage is full. Verify your account within 24 hours at http://mail-verify.example/login to avoid suspension."},
{"s": "neu", "en": "Hello, I am an influencer with 50k followers. Send me a free {BIKE} and I will post about your brand."},
{"s": "neu", "en": "Congratulations! You have been selected for a {MONEY} gift voucher. Click http://prize-claim.example to claim."},
],
}
PHRASES = { # paired fillers for the {lower} slots: (en, de, es, fr)
"fault": [("the battery will not charge", "der Akku lädt nicht mehr", "la batería no carga", "la batterie ne se charge plus"),
("the motor cuts out on hills", "der Motor setzt am Berg aus", "el motor se para en las cuestas", "le moteur se coupe dans les montées"),
("the display shows error E07", "das Display zeigt Fehler E07", "la pantalla muestra el error E07", "l'écran affiche l'erreur E07"),
("the battery runs flat very quickly", "der Akku ist sehr schnell leer", "la batería se agota muy rápido", "la batterie se vide très vite")],
"wear": [("the back tyre has a puncture", "der Hinterreifen hat einen Platten", "la rueda trasera está pinchada", "le pneu arrière est crevé"),
("the gears keep slipping", "die Gangschaltung springt ständig", "los cambios saltan todo el rato", "les vitesses sautent sans arrêt"),
("the brake pads are worn out", "die Bremsbeläge sind abgenutzt", "las pastillas de freno están gastadas", "les plaquettes de frein sont usées"),
("the chain squeaks", "die Kette quietscht", "la cadena chirría", "la chaîne grince")],
"damage": [("a cracked mudguard", "einem gebrochenen Schutzblech", "un guardabarros roto", "un garde-boue fendu"),
("a bent front wheel", "einem verbogenen Vorderrad", "la rueda delantera doblada", "une roue avant voilée"),
("a broken clip", "einem kaputten Clip", "un clip roto", "un clip cassé")],
}
SUBJECTS = {"order_status": ["Where is my order?", "Wo bleibt meine Bestellung?", "¿Dónde está mi pedido?", "Où est ma commande ?"],
"returns_refund": ["Return and refund", "Rücksendung", "Devolución", "Retour"],
"repair_booking": ["Booking a service", "Werkstatttermin", "Cita en el taller", "Rendez-vous atelier"],
"product_question": ["Product question", "Frage zum Produkt", "Pregunta sobre un producto", "Question sur un produit"],
"warranty_fault": ["Fault with my bike", "Defekt", "Avería", "Panne"],
"praise": ["Thank you!", "Danke!", "¡Gracias!", "Merci !"],
"spam": ["Partnership opportunity"] * 4}
FORM_TOPIC = {"order_status": "Order enquiry", "returns_refund": "Returns", "repair_booking": "Workshop",
"product_question": "Product question", "warranty_fault": "Faulty item", "praise": "Feedback", "spam": "Other"}
CHANNELS = {"order_status": (5, 4, 1), "returns_refund": (6, 4, 0), "repair_booking": (4, 6, 0), # email, web form, review
"product_question": (4, 6, 0), "warranty_fault": (5, 3, 2), "praise": (3, 1, 6), "spam": (1, 0, 0)}
LANGS = ["en", "de", "es", "fr"]
MONTHS = {"en": "January February March April May June July August September October November December",
"de": "Januar Februar März April Mai Juni Juli August September Oktober November Dezember",
"es": "enero febrero marzo abril mayo junio julio agosto septiembre octubre noviembre diciembre",
"fr": "janvier février mars avril mai juin juillet août septembre octobre novembre décembre"}
TOWNS = {"en": ["Bristol", "Bath", "Clifton", "Cardiff", "Stroud"], "de": ["Berlin", "Hamburg", "München", "Köln"],
"es": ["Madrid", "Barcelona", "Valencia", "Sevilla"], "fr": ["Paris", "Lyon", "Nantes", "Lille"]}
SPAMMERS = [("Growth Partners", "offers@growth-partners.example"), ("Crypto Ride", "team@cryptoride.example"),
("Mail Admin", "admin@mail-verify.example"), ("Prize Desk", "win@prize-claim.example")]
def fmt_date(d, lang, rng):
m = MONTHS[lang].split()[d.month - 1]
styles = {"en": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}", f"{m} {d.day}{'th' if 10 < d.day < 14 else {1: 'st', 2: 'nd', 3: 'rd'}.get(d.day % 10, 'th')}"],
"de": [f"{d.day}. {m}", f"{d.day:02d}.{d.month:02d}."], "es": [f"{d.day} de {m}", f"{d.day}/{d.month}"],
"fr": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}/{d.year}"]}[lang]
return rng.choice(styles) if rng else styles[0]
def fmt_money(gbp, lang, rng):
styles = {"en": [f"£{gbp:,}", f"£{gbp:,}.00", f"{gbp} pounds"], "de": [f"{gbp:,} £".replace(",", "."), f"{gbp} Pfund"],
"es": [f"{gbp:,} libras".replace(",", "."), f"{gbp} £"], "fr": [f"{gbp:,} £".replace(",", " "), f"{gbp} livres"]}[lang]
return rng.choice(styles) if rng else styles[0]
def typo(word, rng):
i = rng.randrange(1, len(word) - 1)
return rng.choice([word[:i] + word[i + 1] + word[i] + word[i + 2:], word[:i] + word[i + 1:], word[:i] + word[i] + word[i:]])
print(sum(len(v) for v in T.values()), "wordings,", sum("de" in t for v in T.values() for t in v), "of them in all four languages")Code walkthrough Block 2: every wording, and helpers for dates, prices and typos.
text
35 wordings, 18 of them in all four languagespython
import html, unicodedata
POOLS = {"PRODUCT": ["bike", "ebike", "accessory"], "BIKE": ["bike", "ebike"], "EBIKE": ["ebike"],
"ACCESSORY": ["accessory"], "SERVICE": ["service"], "SERVICE2": ["service"], "KIDBIKE": ["kids"]}
EXTRA = {"greet": {"en": ["Hi,", "Hello,", "Hello Tidewell,", "Dear Sir or Madam,"], "de": ["Hallo,", "Guten Tag,"],
"es": ["Hola,", "Buenos días,"], "fr": ["Bonjour,", "Madame, Monsieur,"]},
"sign": {"en": ["Thanks,", "Many thanks,", "Cheers,"], "de": ["Danke,", "Viele Grüße,"],
"es": ["Gracias,", "Un saludo,"], "fr": ["Merci,", "Cordialement,"]},
"phone": {"en": "You can call me on ", "de": "Sie erreichen mich unter ", "es": "Me podéis llamar al ",
"fr": "Vous pouvez m'appeler au "},
"quote": {"en": ("On ", ", Tidewell Cycles <help@tidewell.example> wrote:"), "de": ("Am ", " schrieb Tidewell Cycles <help@tidewell.example>:"),
"es": ("El ", ", Tidewell Cycles <help@tidewell.example> escribió:"), "fr": ("Le ", ", Tidewell Cycles <help@tidewell.example> a écrit :")},
"again": {"en": "Sorry, sending this again as I have not heard back. ", "de": "Entschuldigung, ich schicke das noch einmal. ",
"es": "Perdón, lo envío otra vez porque no he recibido respuesta. ", "fr": "Désolé, je renvoie ce message car je n'ai pas eu de réponse. "}}
LABEL_OF = {"PRODUCT": "PRODUCT", "BIKE": "PRODUCT", "EBIKE": "PRODUCT", "ACCESSORY": "PRODUCT", "SERVICE": "PRODUCT",
"SERVICE2": "PRODUCT", "KIDBIKE": "PRODUCT", "ORDER_ID": "ORDER_ID", "DATE": "DATE", "MONEY": "MONEY",
"LOC": "LOC", "PERSON": "PERSON"}
def fill(template, values, lang, rng):
"""Template text -> list of (text, label or None) pieces, formatting dates and money for `lang`."""
pieces = []
for part in re.split(r"(\{\w+\})", template):
if not (part.startswith("{") and part.endswith("}")):
pieces.append((part, None)) if part else None
continue
slot = part[1:-1]
v = values[slot]
if slot == "DATE":
v = fmt_date(v, lang, rng)
elif slot == "MONEY":
v = fmt_money(v, lang, rng)
elif slot in ("fault", "wear", "damage"):
v = PHRASES[slot][v][LANGS.index(lang)]
pieces.append((v, LABEL_OF.get(slot)))
return pieces
def email_of(first, last):
"""'Mehmet', 'Yılmaz' -> 'mehmet.yilmaz@...'; one fixed address per customer, on a reserved .example domain."""
plain = unicodedata.normalize("NFKD", f"{first}.{last}".lower().replace("ı", "i").replace(" ", "").replace("'", ""))
plain = "".join(ch for ch in plain if ch.isascii())
return plain + "@" + ["post.example", "mailbox.example", "inbox.example"][len(first + last) % 3]
def make_messages(n=1800, seed=SEED):
rng = random.Random(seed)
days = [WINDOW[0] + timedelta(i) for i in range((WINDOW[1] - WINDOW[0]).days + 1)]
base = {"order_status": 26, "returns_refund": 16, "repair_booking": 17, "product_question": 18,
"warranty_fault": 11, "praise": 6, "spam": 6}
def weights(d):
w = dict(base)
w["spam"] *= 8 if EVENTS["spam_wave"][0] <= d <= EVENTS["spam_wave"][1] else 1
w["warranty_fault"] *= 3.5 if EVENTS["battery_notice"] <= d <= date(2026, 7, 31) else 2 if d.month == 8 else 1
w["order_status"] *= (2.5 if EVENTS["summer_sale"][0] <= d <= EVENTS["summer_sale"][1] else 1) * (1.6 if d >= EVENTS["carrier_change"] else 1)
w["returns_refund"] *= 1.3 if date(2026, 7, 29) <= d <= date(2026, 8, 15) else 1
vol = (0.6 if d.weekday() >= 5 else 1.0) * (1.5 if d >= EVENTS["eu_shipping"] else 1.0)
return [w[i] * vol for i in INTENTS]
W = [weights(d) for d in days]
n_dup = round(n * 0.03)
picks = sorted(rng.choices(range(len(days)), weights=[sum(w) for w in W], k=n - n_dup))
by_kind = {k: [p for p in PRODUCTS if p[3] == k] for k in ["bike", "ebike", "accessory", "service"]}
by_kind["kids"] = [p for p in PRODUCTS if p[1] == "Harbour Kids 20"]
out = []
for di in picks:
d = days[di]
intent = rng.choices(INTENTS, weights=W[di])[0]
lang = "en" if intent == "spam" else rng.choices(LANGS, weights=[95, 2, 1.5, 1.5] if d < EVENTS["eu_shipping"] else [58, 16, 14, 12])[0]
options = [t for t in T[intent] if lang in t and date.fromisoformat(t.get("from", "2026-01-01")) <= d]
tmpl = rng.choices(options, weights=[t.get("w", 1) for t in options])[0]
if intent == "spam":
cust, email = rng.choice(SPAMMERS)
first, last, gender, town = cust, "", None, None
else:
first, last, gender, _, town = rng.choice([c for c in CUSTOMERS if c[3] == lang])
email = email_of(first, last)
v = {}
for slot, kinds in POOLS.items():
pool = [p for k in kinds for p in by_kind[k]]
pick = rng.choice([p for p in pool if p[1] != v.get("SERVICE")])
v[slot] = pick[1]
v[slot + "_sku"], v[slot + "_price"] = pick[0], pick[2]
main = next((s for s in POOLS if "{" + s + "}" in tmpl["en"]), None)
future = intent == "repair_booking" or "before {DATE}" in tmpl["en"]
v["DATE"] = d + timedelta(rng.randint(2, 14)) if future else d - timedelta(rng.randint(3, 20))
v["MONEY"] = v[main + "_price"] if main else rng.choice([50, 100, 250, 500])
v["ORDER_ID"] = f"TW-{100000 + di * 120 + rng.randrange(120)}"
lives_there = "live in {LOC}" in tmpl["en"] or "shop in {LOC}" in tmpl["en"]
v["LOC"] = town if lives_there and town else rng.choice(TOWNS[lang])
v["PERSON"] = rng.choice(["Dev", "Dev", "Ines", "Callum"]) if intent == "praise" else first
v.update(carrier="SwiftDrop" if d >= EVENTS["carrier_change"] else "ParcelNorth", email=email, batch=BATCH,
ticket=f"RJ-2026-{300 + di * 2 + rng.randrange(3):04d}", fault=rng.randrange(4), wear=rng.randrange(4),
damage=rng.randrange(3))
core = fill(tmpl[lang], v, lang, rng)
en_ref = "".join(t for t, _ in fill(tmpl["en"], v, "en", None)) if lang != "en" else None
channel = rng.choices(["email", "web form", "review"], weights=CHANNELS[intent])[0]
channel = "web form" if channel == "review" and tmpl["s"] == "neu" else channel # reviews praise or complain
pieces, g = list(core), 0
if channel == "email":
if rng.random() < 0.7:
pieces, g = [(rng.choice(EXTRA["greet"][lang]) + "\n\n", None)] + pieces, 1
if rng.random() < 0.15:
pieces += [("\n" + EXTRA["phone"][lang] + rng.choice(["07700 900", "07700900", "+44 7700 900"]) + f"{rng.randrange(1000):03d}.", None)]
if lang == "en" and intent in ("repair_booking", "order_status") and rng.random() < 0.3:
pieces += [(f"\nMy postcode is BS{rng.randint(1, 16)} {rng.randint(1, 9)}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}.", None)]
if rng.random() < 0.7 and intent != "spam":
pieces += [("\n\n" + rng.choice(EXTRA["sign"][lang]) + "\n", None), (first if rng.random() < 0.6 else f"{first} {last}", "PERSON")]
if rng.random() < 0.15:
pieces += [("\n" + {"en": "Sent from my phone", "de": "Von meinem Handy gesendet", "es": "Enviado desde mi móvil", "fr": "Envoyé de mon téléphone"}[lang], None)]
if rng.random() < 0.25 and intent != "spam":
q = EXTRA["quote"][lang]
prev = fmt_date(d - timedelta(rng.randint(1, 6)), lang, None) + (" de 2026" if lang == "es" else " 2026")
pieces += [("\n\n> " + q[0], None), (prev, "DATE"), (q[1] + "\n> Thanks for your message. We will reply within one working day.", None)]
r = rng.random
if r() < 0.15: # typos
for _ in range(rng.randint(1, 2)):
i = rng.randrange(len(pieces))
if pieces[i][1] is None:
words = pieces[i][0].split(" ")
j = rng.randrange(len(words))
if len(words[j]) >= 5 and words[j].isalpha():
words[j] = typo(words[j], rng)
pieces[i] = (" ".join(words), None)
case = r()
if case < 0.03:
pieces = [(t.upper(), l) for t, l in pieces]
elif case < 0.09:
pieces = [(t.lower(), l) for t, l in pieces]
if r() < 0.08: # emoji after the main text
pieces.insert(g + len(core), (" " + rng.choice(["😡", "😤", "🚲"] if tmpl["s"] == "neg" else ["🙂", "👍", "🚲"]), None))
if r() < 0.05: # odd whitespace
i = next((i for i, (t, l) in enumerate(pieces) if l is None and " " in t), None)
if i is not None:
t = pieces[i][0]
pieces[i] = (t.replace(" ", " ", 1) if r() < 0.5 else t.replace(" ", " ", 1), None)
if channel == "web form" and r() < 0.6: # HTML from the form
pieces = [("<p>", None)] + [(html.escape(t).replace("\n", "<br>\n"), None) if l is None else (t, l) for t, l in pieces] + [("</p>", None)]
if r() < 0.04: # mojibake: UTF-8 bytes read as cp1252
try:
pieces = [(t.encode("utf-8").decode("cp1252"), l) for t, l in pieces]
except UnicodeDecodeError:
pass
body, spans, pos = "", [], 0
for t, l in pieces:
if l:
spans.append((pos, pos + len(t), l))
body, pos = body + t, pos + len(t)
sent = tmpl["s"]
stars = None
if channel == "review":
stars = {"pos": rng.choice([4, 5]), "neu": 3, "neg": rng.choice([1, 2])}[sent]
if r() < 0.08:
stars = {"pos": rng.choice([1, 2]), "neu": rng.choice([1, 5]), "neg": 5}[sent]
subject = {"email": SUBJECTS[intent][LANGS.index(lang)], "web form": "Web form: " + FORM_TOPIC[intent],
"review": "Review: " + (v[main] if main else "Tidewell Cycles")}[channel]
out.append({"date": d.isoformat(), "channel": channel, "language": lang, "customer": f"{first} {last}".strip(),
"gender": gender, "email": email, "subject": subject, "body": body, "intent": intent, "sentiment": sent,
"stars": stars, "entities": spans, "sku": v[main + "_sku"] if main else None,
"order_id": v["ORDER_ID"] if "{ORDER_ID}" in tmpl["en"] else None, "duplicate_of": None,
"core": "".join(t for t, _ in core), "en_ref": en_ref, "label_flipped": False})
originals = [m for m in out if m["intent"] != "spam"]
for m in rng.sample(originals, n_dup): # near-duplicates: the same customer writes again
again, cut = EXTRA["again"][m["language"]], 3 if m["body"].startswith("<p>") else 0 # inside the form's HTML
dup = dict(m, body=m["body"][:cut] + again + m["body"][cut:], entities=[(a + len(again), b + len(again), l) for a, b, l in m["entities"]],
date=min(date.fromisoformat(m["date"]) + timedelta(rng.randint(0, 2)), WINDOW[1]).isoformat(), duplicate_of=m)
out.append(dup)
out.sort(key=lambda m: (m["date"], m["duplicate_of"] is not None))
for i, m in enumerate(out, 1):
m["id"] = f"TW-M{i:04d}"
for m in out:
if m["duplicate_of"] is not None:
m["duplicate_of"] = m["duplicate_of"]["id"]
for m in rng.sample(out, round(n * 0.02)): # deliberate wrong labels, for error analysis
m["intent"], m["label_flipped"] = rng.choice([i for i in INTENTS if i != m["intent"]]), True
split_of = {}
for intent in INTENTS: # stratified 70 / 15 / 15, duplicates follow their original
ids = [m["id"] for m in out if m["intent"] == intent and m["duplicate_of"] is None]
rng.shuffle(ids)
a, b = round(len(ids) * 0.70), round(len(ids) * 0.85)
split_of.update({x: "train" if k < a else "dev" if k < b else "test" for k, x in enumerate(ids)})
for m in out:
m["split"] = split_of[m["duplicate_of"] or m["id"]]
m["period"] = "before_july" if m["date"] < "2026-07-01" else "from_july"
return out
MESSAGES = make_messages()
print(len(MESSAGES), "messages;", MESSAGES[0]["date"], "to", MESSAGES[-1]["date"])
print(MESSAGES[0]["id"], "|", MESSAGES[0]["subject"], "|", MESSAGES[0]["body"][:80].replace("\n", " "))Code walkthrough Block 3: the generator of the 1,800 messages.
text
1800 messages; 2026-03-02 to 2026-08-30
TW-M0001 | Web form: Workshop | <p>Mon vélo (Causeway) a besoin d'une réparation : les plaquettes de freipython
import re
def unmark(marked):
"""'[[Volt 2|PRODUCT]] died' -> ('Volt 2 died', [(0, 6, 'PRODUCT')])."""
text, spans, pos = "", [], 0
for m in re.finditer(r"\[\[(.+?)\|([A-Z_]+)\]\]|([^\[]+|\[)", marked):
if m.group(1):
spans.append((len(text), len(text) + len(m.group(1)), m.group(2)))
text += m.group(1)
else:
text += m.group(3)
return text, spans
# (language, kind, intent, second intent, sentiment, text with [[entity|LABEL]] marks, English reference or None)
HARD = [
("en", "sarcasm", "warranty_fault", None, "neg", "Oh brilliant, the [[Volt 2|PRODUCT]] battery died halfway up the hill again. Really loving my [[£2,199|MONEY]] paperweight.", None),
("en", "two_intents", "order_status", "returns_refund", "neu", "Two things: where is order [[TW-118734|ORDER_ID]]? And I want to send back the [[LockSure D-lock|PRODUCT]] from my last order, it's too heavy.", None),
("en", "very_short", "returns_refund", None, "neg", "refund??", None),
("en", "very_short", "order_status", None, "neu", "where's my bike", None),
("en", "slang", "repair_booking", None, "neu", "yo the gears on my [[Causeway|PRODUCT]] are proper knackered, can u lot sort it [[sat|DATE]]?", None),
("en", "plain", "praise", None, "pos", "[[Dev|PERSON]] fixed my wheel in ten minutes while I waited. Legend.", None),
("en", "sarcasm", "order_status", None, "neg", "Great to see my parcel has been 'out for delivery' for six days. Very scenic route, I assume. Order [[TW-121090|ORDER_ID]].", None),
("en", "negation", "product_question", None, "neu", "I don't want the [[Volt 2|PRODUCT]], I want something without a battery for my commute. Is the [[Harbour|PRODUCT]] any good on hills?", None),
("en", "plain", "warranty_fault", None, "neg", "The display on my [[Volt 2 Step-Through|PRODUCT]] keeps showing E07 and the motor won't engage. Bought it on [[14 March|DATE]].", None),
("en", "plain", "spam", None, "neu", "Hi [[Rosa|PERSON]], loved your shop! We help bike brands get 10x more reviews. Can I send over a quick proposal?", None),
("en", "very_long", "warranty_fault", None, "neu", "Hello, I hope you are well. I'm writing about the email you sent on [[9 June|DATE]] about battery batch VB-0426. I checked the label under my pack as the email said, and it does say VB-0426. To be honest the battery had been fine until about a fortnight ago, when it started dropping from 80% to nothing in a couple of miles. I use the bike every day to take my son to school and then ride on to work, so I can't really be without it for long. Could you tell me how the swap works, whether I need to bring the whole bike in or just the battery, and roughly how long it takes? My order number was [[TW-103552|ORDER_ID]] if that helps. Many thanks, [[Margaret Ellis|PERSON]]", None),
("en", "codes", "product_question", None, "neu", "is TW-BATT-500 the same pack as the one in the [[Volt 2|PRODUCT]] or a newer version?", None),
("en", "plain", "returns_refund", None, "neu", "I'd like to return the [[Commuter Pannier Set|PRODUCT]]. The clips don't fit my rack. Order [[TW-115902|ORDER_ID]].", None),
("en", "plain", "repair_booking", None, "neu", "Can I bring my [[Harbour Kids 20|PRODUCT]] in for a [[Basic Service|PRODUCT]] next week? Any day but [[Wednesday|DATE]].", None),
("en", "mixed", "praise", None, "neu", "Bike is lovely, delivery was a nightmare. Overall happy though, thanks [[Ines|PERSON]].", None),
("en", "slang", "order_status", None, "neg", "any news on my order mate? [[TW-122417|ORDER_ID]]. been ages", None),
("en", "plain", "spam", None, "neu", "Your Tidewell Cycles account has been suspended. Confirm your payment details at tidewell-secure.example within 24 hours.", None),
("en", "very_short", "warranty_fault", None, "neg", "battery dead. again.", None),
("en", "plain", "repair_booking", None, "neu", "Is the [[Full Service|PRODUCT]] still [[£129|MONEY]]? If so I'll book it for the [[2nd of September|DATE]].", None),
("en", "plain", "product_question", None, "neu", "Will a [[LockSure D-lock|PRODUCT]] fit the frame of the [[Causeway|PRODUCT]] or is the tube too fat?", None),
("en", "sarcasm", "returns_refund", None, "neg", "Thanks so much for refunding me in 'five working days'. It's been fifteen. Order [[TW-112006|ORDER_ID]], [[£89|MONEY]].", None),
("en", "two_intents", "warranty_fault", "repair_booking", "neg", "My [[Volt 2|PRODUCT]] motor is cutting out, and while it's in could you also do a [[Basic Service|PRODUCT]]?", None),
("en", "plain", "praise", None, "pos", "Five stars for the [[Harbour|PRODUCT]]. Comfy, fast enough, and the bell is very loud. Ha.", None),
("en", "plain", "order_status", None, "neg", "Ordered on [[3/7|DATE]], still waiting on [[TW-119880|ORDER_ID]]. SwiftDrop says 'exception'. What does that mean?", None),
("en", "plain", "product_question", None, "neu", "Do you price match? Saw the [[Volt 2|PRODUCT]] cheaper elsewhere.", None),
("en", "sarcasm", "order_status", None, "neg", "Love how SwiftDrop left my new bike in the bin store. Really secure. Order [[TW-124118|ORDER_ID]].", None),
("en", "very_long", "returns_refund", None, "neu", "Hello, I bought the [[Harbour Kids 20|PRODUCT]] for my nephew's birthday on [[18 July|DATE]] during your summer sale and paid [[£279|MONEY]]. Unfortunately my sister had already bought him a bike, which nobody told me about until the party. The bike is still in its box, completely unused, and I have the receipt. I understand that sale items can sometimes be treated differently, so I wanted to check before I post it back whether I can still get a full refund, and whether you send a returns label or I need to arrange the courier myself. I would also like to know how long the refund normally takes to appear on my card. Thank you for your help. [[Peter Lowe|PERSON]]", None),
("en", "slang", "repair_booking", None, "neu", "brakes squealing like a banshee lol. book me in?", None),
("en", "plain", "spam", None, "neu", "We are a logistics company offering cheap shipping from overseas factories. Minimum order 500 units. Message us for prices.", None),
("en", "plain", "warranty_fault", None, "neg", "Box arrived soaked and the [[Causeway|PRODUCT]] rear mech is bent. Photos attached.", None),
("en", "very_short", "product_question", None, "neu", "kids helmets?", None),
("en", "plain", "praise", None, "pos", "Shout out to [[Callum|PERSON]] for talking me out of the expensive model. Saved me [[£500|MONEY]] and I love the bike.", None),
("en", "two_intents", "order_status", "product_question", "neu", "Has order [[TW-120577|ORDER_ID]] shipped yet? Also, does the [[Beacon Light Set|PRODUCT]] come with batteries?", None),
("en", "negation", "repair_booking", None, "neu", "This is not a complaint, the bike is great, I just need to know when my [[Full Service|PRODUCT]] slot is.", None),
("en", "very_short", "returns_refund", None, "neg", "Charged twice. [[£45|MONEY]] and [[£45|MONEY]]. Fix it.", None),
("en", "sarcasm", "warranty_fault", None, "neg", "Fourth puncture this month on the 'puncture resistant' tyres. Impressive.", None),
("en", "plain", "spam", None, "neu", "Earn passive income from your bikes! Tokenise your fleet with BikeCoin today.", None),
("en", "plain", "praise", None, "pos", "cheers for sorting the refund so quick, really appreciate it", None),
("en", "very_long", "order_status", None, "neg", "Hi there, I'm getting a bit worried about my order [[TW-123390|ORDER_ID]]. I ordered the [[Volt 2 Step-Through|PRODUCT]] on [[22 July|DATE]] in the sale, and the confirmation said it would arrive within five working days. On [[12 August|DATE]] I got a text from a courier I've never heard of, SwiftDrop, saying they had tried to deliver it, but I was at home all day and nobody knocked. Since then the tracking page just says 'awaiting next attempt'. I've tried calling the number on the text but it goes to a recorded message. Could you check with them what is going on and when I can expect it? I need it for my new job, which starts on [[1 September|DATE]]. Thanks, [[Nadia Hussein|PERSON]]", None),
("en", "slang", "repair_booking", None, "neu", "can i get a puncture fixed same day if i drop it in before 9?", None),
("de", "plain", "order_status", None, "neg", "Meine Bestellung [[TW-121554|ORDER_ID]] ist seit einer Woche unterwegs. Wo bleibt sie?", "My order TW-121554 has been on its way for a week. Where is it?"),
("de", "plain", "warranty_fault", None, "neg", "Der Akku von meinem [[Volt 2|PRODUCT]] lädt nicht mehr. Das Rad habe ich erst seit [[Juli|DATE]].", "The battery on my Volt 2 no longer charges. I have only had the bike since July."),
("de", "sarcasm", "order_status", None, "neg", "Toll, drei Zustellversuche und ich war jedes Mal zu Hause. Super Service von SwiftDrop.", "Great, three delivery attempts and I was at home every time. Super service from SwiftDrop."),
("de", "plain", "returns_refund", None, "neu", "Ich möchte das [[Commuter Pannier Set|PRODUCT]] zurückschicken. Wie bekomme ich ein Rücksendeetikett?", "I would like to send back the Commuter Pannier Set. How do I get a returns label?"),
("de", "plain", "product_question", None, "neu", "Kann man das [[Volt 2|PRODUCT]] auch in [[Deutschland|LOC]] reparieren lassen, oder muss es nach [[Bristol|LOC]]?", "Can the Volt 2 also be repaired in Germany, or does it have to go to Bristol?"),
("de", "code_switch", "repair_booking", None, "neu", "Hallo, mein [[Causeway|PRODUCT]] hat einen flat tyre, kann ich das bei euch fixen lassen oder soll ich es selbst machen?", "Hello, my Causeway has a flat tyre, can I get it fixed by you or should I do it myself?"),
("de", "plain", "praise", None, "pos", "Vielen Dank, das Rad ist heute angekommen und es ist wunderschön!", "Thank you very much, the bike arrived today and it is beautiful!"),
("de", "two_intents", "order_status", "warranty_fault", "neg", "Wo ist meine Bestellung [[TW-122960|ORDER_ID]]? Und ich will das [[LockSure D-lock|PRODUCT]] umtauschen, der Schlüssel klemmt.", "Where is my order TW-122960? And I want to exchange the LockSure D-lock, the key sticks."),
("de", "very_short", "order_status", None, "neu", "Rechnung?", "Invoice?"),
("de", "plain", "repair_booking", None, "neu", "Können Sie mein E-Bike am [[Samstag|DATE]] durchsehen? Die Bremsen schleifen.", "Can you look over my e-bike on Saturday? The brakes are rubbing."),
("de", "very_long", "warranty_fault", None, "neu", "Guten Tag, ich habe am [[14. Juli|DATE]] ein [[Volt 2 Step-Through|PRODUCT]] bei Ihnen gekauft. Seit gestern zeigt das Display den Fehler E07 und der Motor unterstützt nicht mehr. Ich habe den Akku vollständig geladen und das Rad neu gestartet, aber nichts hat geholfen. Bitte teilen Sie mir mit, wie die Garantie funktioniert und ob ich das Rad nach [[Großbritannien|LOC]] schicken muss. Mit freundlichen Grüßen, [[Sabine Krause|PERSON]]", "Good afternoon, on 14 July I bought a Volt 2 Step-Through from you. Since yesterday the display has shown error E07 and the motor no longer assists. I have fully charged the battery and restarted the bike, but nothing has helped. Please tell me how the warranty works and whether I have to send the bike to Great Britain. Kind regards, Sabine Krause"),
("de", "plain", "spam", None, "neu", "Sehr geehrte Damen und Herren, wir bieten Ihnen günstige Werbung auf 500 Fahrrad-Blogs an. Interesse?", "Dear Sir or Madam, we offer you cheap advertising on 500 cycling blogs. Interested?"),
("de", "slang", "warranty_fault", None, "neg", "Moin, Rad is da, aber der Lenker ist total schief montiert. Was nu?", "Hi, the bike is here, but the handlebars are fitted completely crooked. What now?"),
("de", "sarcasm", "returns_refund", None, "neg", "Super, die Erstattung von [[549 £|MONEY]] dauert jetzt schon drei Wochen. Ganz toll.", "Great, the refund of £549 has now taken three weeks. Just wonderful."),
("es", "plain", "order_status", None, "neu", "Hola, ¿cuándo llega mi pedido [[TW-121871|ORDER_ID]]? Ya ha pasado una semana.", "Hello, when will my order TW-121871 arrive? A week has already gone by."),
("es", "plain", "warranty_fault", None, "neg", "La batería de mi [[Volt 2|PRODUCT]] se agota en diez minutos. ¿Está en garantía?", "The battery of my Volt 2 runs out in ten minutes. Is it under warranty?"),
("es", "code_switch", "order_status", None, "neu", "Hi, my order llegó pero falta el cargador. Can you send it?", "Hi, my order arrived but the charger is missing. Can you send it?"),
("es", "plain", "returns_refund", None, "neu", "Quiero devolver la [[Harbour|PRODUCT]], es demasiado grande para mí. ¿Me devolvéis el dinero?", "I want to return the Harbour, it is too big for me. Will you give me my money back?"),
("es", "sarcasm", "order_status", None, "neg", "Genial, el repartidor dejó la bici en la calle bajo la lluvia. Gracias, de verdad.", "Great, the delivery driver left the bike in the street in the rain. Thanks, really."),
("es", "plain", "product_question", None, "neu", "¿La [[Volt 2 Step-Through|PRODUCT]] sirve para alguien de 1,55 m?", "Is the Volt 2 Step-Through suitable for someone who is 1.55 m tall?"),
("es", "plain", "repair_booking", None, "neu", "Necesito cambiar las pastillas de freno. ¿Tenéis hueco el [[jueves|DATE]]?", "I need to change the brake pads. Do you have a slot on Thursday?"),
("es", "plain", "praise", None, "pos", "¡Muchísimas gracias a [[Ines|PERSON]] por responder en español! Todo resuelto.", "Many thanks to Ines for replying in Spanish! All sorted."),
("es", "two_intents", "order_status", "returns_refund", "neg", "Mi pedido [[TW-122204|ORDER_ID]] no ha llegado y además quiero cancelar el [[Beacon Light Set|PRODUCT]] que pedí aparte.", "My order TW-122204 has not arrived and I also want to cancel the Beacon Light Set that I ordered separately."),
("es", "very_short", "product_question", None, "neu", "¿Hacéis envíos a [[Canarias|LOC]]?", "Do you ship to the Canary Islands?"),
("es", "plain", "spam", None, "neu", "Gana [[5.000 euros|MONEY]] al mes desde casa. Haz clic aquí.", "Earn 5,000 euros a month from home. Click here."),
("es", "very_long", "warranty_fault", None, "neg", "Buenas tardes. Compré una [[Causeway|PRODUCT]] el [[20 de julio|DATE]] y me llegó el [[29 de julio|DATE]]. Desde el primer día el cambio trasero hace un ruido muy raro y las marchas saltan cuando subo cuestas. La he llevado a una tienda de bicis aquí en [[Valencia|LOC]] y me dicen que el desviador viene doblado de fábrica. No quiero pagar una reparación por algo que vino mal. ¿Me podéis decir qué hago? ¿La tengo que enviar a [[Inglaterra|LOC]]? Un saludo, [[Marta Gil|PERSON]]", "Good afternoon. I bought a Causeway on 20 July and it arrived on 29 July. From the first day the rear gears have made a very strange noise and the gears slip when I ride uphill. I have taken it to a bike shop here in Valencia and they tell me the derailleur came bent from the factory. I don't want to pay for a repair for something that arrived faulty. Can you tell me what to do? Do I have to send it to England? Best wishes, Marta Gil"),
("es", "slang", "praise", None, "neu", "Tío, la bici mola mazo, pero el sillín es durísimo jaja", "Mate, the bike is really cool, but the saddle is rock hard haha"),
("fr", "plain", "order_status", None, "neg", "Bonjour, ma commande [[TW-122691|ORDER_ID]] est bloquée en transit depuis six jours. Pouvez-vous vérifier ?", "Hello, my order TW-122691 has been stuck in transit for six days. Can you check?"),
("fr", "plain", "warranty_fault", None, "neg", "Mon [[Volt 2|PRODUCT]] affiche l'erreur E07 et le moteur ne marche plus.", "My Volt 2 shows error E07 and the motor no longer works."),
("fr", "sarcasm", "order_status", None, "neg", "Bravo, livré chez le voisin du dessus sans me prévenir. Très pro.", "Well done, delivered to the upstairs neighbour without telling me. Very professional."),
("fr", "plain", "returns_refund", None, "neu", "Je souhaite renvoyer le [[LockSure D-lock|PRODUCT]] : il est trop lourd pour moi. Comment faire ?", "I would like to send back the LockSure D-lock: it is too heavy for me. How do I do that?"),
("fr", "plain", "product_question", None, "neu", "Le [[Harbour|PRODUCT]] est-il livré monté ou faut-il l'assembler ?", "Does the Harbour come assembled or does it need to be put together?"),
("fr", "code_switch", "warranty_fault", None, "neu", "Salut, the bike est arrivé mais the front light ne s'allume pas. C'est normal ?", "Hi, the bike has arrived but the front light does not switch on. Is that normal?"),
("fr", "plain", "repair_booking", None, "neu", "Est-ce que je peux prendre rendez-vous pour un [[Full Service|PRODUCT]] le [[3 septembre|DATE]] ?", "Can I make an appointment for a Full Service on 3 September?"),
("fr", "plain", "praise", None, "pos", "Merci à toute l'équipe, le vélo est parfait et la livraison à [[Lyon|LOC]] a été rapide.", "Thanks to the whole team, the bike is perfect and the delivery to Lyon was quick."),
("fr", "very_short", "returns_refund", None, "neg", "Remboursement ???", "Refund ???"),
("fr", "two_intents", "warranty_fault", "product_question", "neg", "Ma batterie ne charge plus et je voudrais aussi savoir si vous vendez une deuxième batterie ([[Volt 2 battery pack (500 Wh)|PRODUCT]]).", "My battery no longer charges and I would also like to know whether you sell a second battery (Volt 2 battery pack (500 Wh))."),
("fr", "plain", "spam", None, "neu", "Félicitations ! Vous avez gagné un téléphone neuf. Cliquez ici pour le recevoir.", "Congratulations! You have won a new phone. Click here to receive it."),
("fr", "slang", "praise", None, "neu", "Franchement, le vélo est trop bien, mais le livreur a galéré pour trouver chez moi lol", "Honestly, the bike is great, but the delivery driver struggled to find my place lol"),
("fr", "very_long", "warranty_fault", None, "neu", "Bonjour, j'ai acheté un [[Volt 2 Step-Through|PRODUCT]] le [[16 juillet|DATE]] pendant les soldes, pour [[1 999 £|MONEY]]. Depuis une semaine, la batterie se vide très vite : je fais à peine dix kilomètres alors qu'avant j'en faisais cinquante. J'ai lu sur votre site qu'il y avait un problème avec certains lots de batteries, mais le numéro sur la mienne n'est pas VB-0426. Est-ce que ma batterie est quand même couverte par la garantie, et comment faire pour l'échanger depuis la [[France|LOC]] ? Cordialement, [[Hélène Rousseau|PERSON]]", "Hello, I bought a Volt 2 Step-Through on 16 July during the sale, for £1,999. For a week the battery has been running down very quickly: I barely manage ten kilometres when before I did fifty. I read on your website that there was a problem with some battery batches, but the number on mine is not VB-0426. Is my battery still covered by the warranty, and how do I exchange it from France? Kind regards, Hélène Rousseau"),
]
HARD_TEST = []
for k, (lang, kind, intent, second, sent, marked, en) in enumerate(HARD, 1):
text, spans = unmark(marked)
HARD_TEST.append({"id": f"HT{k:02d}", "language": lang, "kind": kind, "intent": intent, "second_intent": second,
"sentiment": sent, "text": text, "entities": spans, "en_ref": en})
from collections import Counter
print(len(HARD_TEST), "hand-written messages:", dict(Counter(h["language"] for h in HARD_TEST)))
print(dict(Counter(h["intent"] for h in HARD_TEST)))
print(dict(Counter(h["kind"] for h in HARD_TEST)))
print(HARD_TEST[0]["text"], HARD_TEST[0]["entities"])Code walkthrough Block 4: HARD_TEST, 80 hand-written messages.
text
80 hand-written messages: {'en': 40, 'de': 14, 'es': 13, 'fr': 13}
{'warranty_fault': 16, 'order_status': 18, 'returns_refund': 10, 'repair_booking': 10, 'praise': 10, 'product_question': 9, 'spam': 7}
{'sarcasm': 9, 'two_intents': 6, 'very_short': 8, 'slang': 7, 'plain': 37, 'negation': 2, 'very_long': 6, 'codes': 1, 'mixed': 1, 'code_switch': 3}
Oh brilliant, the Volt 2 battery died halfway up the hill again. Really loving my £2,199 paperweight. [(18, 24, 'PRODUCT'), (82, 88, 'MONEY')]A fingerprint is a short code computed from the data. Change one character and it changes completely, so matching fingerprints mean identical data.
python
import hashlib, json
def fingerprint(*datasets):
"""The first 12 hex digits of a SHA-256 hash of the data: equal fingerprints mean identical data."""
blob = json.dumps(datasets, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
return hashlib.sha256(blob).hexdigest()[:12]
print("messages ", fingerprint(MESSAGES))
print("hard_test", fingerprint(HARD_TEST))
edited = [dict(m) for m in MESSAGES]
edited[0]["id"] = "TW-M0000" # change one character: TW-M0001 becomes TW-M0000
print("edited ", fingerprint(edited))Code walkthrough
json.dumpswrites the data as one text, keys sorted;sha256(...).hexdigest()[:12]hashes it and keeps twelve characters.- Your first two lines must read
294611361162and63c80b0396bc; if not, paste the blocks again. editedcopies the messages with one character changed. Its fingerprint shares nothing with the original.
text
messages 294611361162
hard_test 63c80b0396bc
edited 1c42b2e57f34Now find Megan's message.
python
from collections import Counter
RUNNING = next(m for m in MESSAGES if m["id"] == "TW-M0327")
for field in ["id", "date", "channel", "customer", "intent", "sentiment", "sku", "order_id"]:
print(f"{field:10}", RUNNING[field])
print(RUNNING["body"])
print("per month:", dict(sorted(Counter(m["date"][:7] for m in MESSAGES).items())))Code walkthrough
next(...)stops at the first message whose id isTW-M0327.intentandsentimentare gold labels: the answers from Ines's definitions, which every model is scored against.bodyis the raw text as it arrived.Counter(m["date"][:7] ...)counts messages per month.
text
id TW-M0327
date 2026-04-15
channel email
customer Megan Price
intent warranty_fault
sentiment neg
sku TW-VOLT2-STP
order_id None
Hello Tidewell,
My e-bike (Volt 2 Step-Through) is only four months old and the battery runs flat very quickly. Is this covered by the warranty? 🚲
You can call me on 07700900201.
Thanks,
Megan Price
per month: {'2026-03': 211, '2026-04': 260, '2026-05': 203, '2026-06': 267, '2026-07': 460, '2026-08': 399}order_id is None: Megan gave no order number, so Dev will have to ask. Extraction cannot find what is not there.
3. Where NLP is used: search, routing, extraction, moderation, translation (Essential)
NLP is not one task. The same few kinds of output appear in almost every product that handles text, and the kind you need decides how you measure it.
Search ranks documents for a query. Routing, or classification, picks one label from a fixed list. Extraction pulls fields out of text. Moderation flags text that must not pass, such as spam. Translation turns one language into another. Here are crude versions of all five, run on Megan's message.
python
import re
TITLES = {"KB04": "What our warranty covers", "KB06": "Caring for your e-bike battery",
"KB10": "Tracking your parcel and our courier"}
text = RUNNING["body"]
words = set(re.findall(r"[a-z0-9-]+", text.lower()))
route = "Dev" if words & {"battery", "faulty", "warranty"} else "Ines"
found = {"phone": re.findall(r"07700 ?\d{6}", text),
"product": [name for sku, name, price, kind in PRODUCTS if name in text]}
spam = bool(words & {"crypto", "proposal", "suspended"})
article = max(TITLES, key=lambda k: len(words & set(TITLES[k].lower().split())))
language = "en" if words & {"the", "and", "is"} else "not en"
print("routing :", route)
print("extraction:", found)
print("moderation:", "spam" if spam else "not spam")
print("search :", article, TITLES[article])
print("translate?:", language)Code walkthrough
TITLESholds three help-centre titles from the shared data.re.findall(...)pulls out words (regular expressions are Lesson 3);setkeeps each once.words & {...}is set intersection, non-empty when any listed word appears: the router and the spam check.- The extraction line finds a phone number shaped
07700plus six digits, and every product name in the text. max(TITLES, key=...)picks the title sharing most words with the message: a tiny search engine. The last check guesses the language from common English words.
text
routing : Dev
extraction: {'phone': ['07700900201'], 'product': ['Volt 2', 'Volt 2 Step-Through']}
moderation: not spam
search : KB06 Caring for your e-bike battery
translate?: enTwo answers are wrong. Extraction found "Volt 2" inside "Volt 2 Step-Through". Search chose battery care when Megan asks about the warranty. Crude methods fail quietly; you only notice when you look.
4. Rules, statistics, neural networks and LLMs: four eras, one toolkit (Essential)
No method has replaced the others. Each era added a tool, and engineers still choose among all four by cost and measured results.
Rules are patterns people write by hand, as Weizenbaum did. Statistics counts how words behave in labelled examples. Neural networks learn their own representations of words from large amounts of text. Large language models (LLMs) are very large neural networks that can be asked to do a task in plain words.
A rule says "battery means warranty fault" because a person believes it. Statistics checks the belief against the labelled training messages.
python
train = [m for m in MESSAGES if m["split"] == "train"]
for word in ["battery", "refund", "service", "order"]:
rows = [m for m in train if re.search(rf"\b{word}\b", m["body"].lower())]
intent, n = Counter(m["intent"] for m in rows).most_common(1)[0]
print(f"{word:8} in {len(rows):3} training messages, most often {intent} ({n / len(rows):.2f})")Code walkthrough
trainkeeps the 1,262 training messages; scoring always uses messages a method did not learn from.\bin the pattern marks a word boundary, so only whole words match.most_common(1)gives the commonest intent among those messages;n / len(rows)is its share.
text
battery in 154 training messages, most often warranty_fault (0.73)
refund in 114 training messages, most often returns_refund (0.98)
service in 110 training messages, most often repair_booking (0.50)
order in 321 training messages, most often order_status (0.58)"Refund" is a near-certain cue (0.98); "service" is a coin toss (0.50), because people also praise or criticise customer service. Module 4 turns counts like these into a trained classifier.
Later eras cost more to run or explain, so the course scores each on the same test set and keeps the cheapest that does the job.
5. The NLP pipeline from raw text to a decision (Essential)
Almost every NLP system is a chain of small steps. Seeing the chain tells you where a mistake came from and which module fixes it.
The pipeline: clean the raw text; split it into tokens, the units later steps count; turn tokens into features, the evidence a decision uses; let a model, here a list of rules, pick a label; act on it.
python
import html
def clean(raw):
text = html.unescape(re.sub(r"<[^>]+>", " ", raw)) # drop web-form HTML tags
return " ".join(text.lower().split()) # one case, single spaces
def tokens(text):
return re.findall(r"[a-z0-9']+", text)
RULES = [("spam", ["proposal", "crypto", "suspended", "seo", "investment"]),
("warranty_fault", ["faulty", "broken", "battery", "error", "damaged", "snapped"]),
("returns_refund", ["return", "refund", "send back", "cancel"]),
("repair_booking", ["service", "book", "repair", "puncture"]),
("order_status", ["order", "delivery", "parcel", "arrived"]),
("praise", ["thank", "brilliant", "love", "great"])]
ROUTE = {"warranty_fault": "Dev", "repair_booking": "Dev", "product_question": "Callum"}
def decide(toks):
joined = " ".join(toks)
for intent, cues in RULES:
hits = [c for c in cues if c in joined]
if hits:
return intent, hits
return "product_question", [] # no rule fired
cleaned = clean(RUNNING["body"])
toks = tokens(cleaned)
intent, hits = decide(toks)
print(cleaned[:58], "...")
print(len(toks), "tokens:", toks[:7])
print("cues:", hits, "| intent:", intent, "| route to:", ROUTE.get(intent, "Ines"))Code walkthrough
cleanswaps HTML tags for spaces, turns codes such as'back into characters (html.unescape), lower-cases, and squeezes whitespace.tokenskeeps runs of letters, digits and apostrophes, so "e-bike" becomeseandbike.RULESis ordered: the first rule with a cue wins, so spam is checked first.decidereturns that intent and its cues, orproduct_questionif none fire.ROUTE.get(intent, "Ines")maps the intent to a person, Ines by default.
text
hello tidewell, my e-bike (volt 2 step-through) is only fo ...
36 tokens: ['hello', 'tidewell', 'my', 'e', 'bike', 'volt', '2']
cues: ['battery'] | intent: warranty_fault | route to: DevEach step passes its mistakes on: a missed HTML tag becomes a token, and a rule may fire on it. This trace is what you read when a message goes to the wrong person.
6. Baseline first: measuring before building (Essential)
The rules routed Megan correctly, but they were written with messages like hers in mind. Before building better, measure the simplest answers on messages nobody tuned for.
A baseline is the score of the simplest reasonable method. The majority baseline always answers with the most common label; a model that cannot beat it has learned nothing. The score here is accuracy, the share labelled correctly.
python
def accuracy(gold, predicted):
return sum(g == p for g, p in zip(gold, predicted)) / len(gold)
majority = Counter(m["intent"] for m in train).most_common(1)[0][0]
def majority_predict(texts):
return [majority for _ in texts]
test = [m for m in MESSAGES if m["split"] == "test"]
print("always say:", majority)
print("template test:", round(accuracy([m["intent"] for m in test], majority_predict([m["body"] for m in test])), 3), f"({len(test)} messages)")
print("HARD_TEST :", round(accuracy([h["intent"] for h in HARD_TEST], majority_predict([h["text"] for h in HARD_TEST])), 3), f"({len(HARD_TEST)} messages)")Code walkthrough
accuracypairs gold labels with predictions (zip), counts matches and divides by the total.majorityis the commonest training intent.majority_predict(texts)returns one label per text: every intent model in the course has this shape,predict(texts) -> list[str].testis the template test split, 271 generated messages held out from training.
text
always say: order_status
template test: 0.247 (271 messages)
HARD_TEST : 0.225 (80 messages)Always saying order_status is right 22.5% of the time on HARD_TEST: the floor. Now score the keyword rules from Section 5.
python
def keyword_predict(texts):
return [decide(tokens(clean(t)))[0] for t in texts]
pred = keyword_predict([h["text"] for h in HARD_TEST])
print("template test:", round(accuracy([m["intent"] for m in test], keyword_predict([m["body"] for m in test])), 3))
print("HARD_TEST :", round(accuracy([h["intent"] for h in HARD_TEST], pred), 3))
for lang in ["en", "de", "es", "fr"]:
pairs = [(h["intent"], p) for h, p in zip(HARD_TEST, pred) if h["language"] == lang]
print(f" {lang}: {sum(g == p for g, p in pairs)} of {len(pairs)} right")Code walkthrough
keyword_predictruns clean, tokens and decide on each text and keeps the intent.- The loop scores the
HARD_TESTmessages one language at a time.
text
template test: 0.661
HARD_TEST : 0.4
en: 25 of 40 right
de: 1 of 14 right
es: 3 of 13 right
fr: 3 of 13 rightThe rules score 0.661 on template messages and 0.4 on hand-written ones. Generated messages repeat their wordings, which flatters rules shaped around them. That is why HARD_TEST is the course's headline number.
By language: 25 of 40 English messages right, and 7 of 40 in German, Spanish or French, four of those only because the fallback label happened to be the answer.
Break it: one German battery
Run this and you will see the rules miss a German battery fault; then one patch, and what it buys.
python
akku = next(h for h in HARD_TEST if h["text"].startswith("Der Akku"))
print(akku["text"])
print("gold:", akku["intent"], "| sketch:", keyword_predict([akku["text"]])[0])
RULES[1][1].append("akku") # patch the warranty rule with one German word
print("after the patch:", keyword_predict([akku["text"]])[0])
print("HARD_TEST now:", round(accuracy([h["intent"] for h in HARD_TEST], keyword_predict([h["text"] for h in HARD_TEST])), 3))
RULES[1][1].remove("akku")Code walkthrough
RULES[1][1]is the cue list of thewarranty_faultrule;.append("akku")adds one word.- The last line removes it, so later cells see the original rules.
text
Der Akku von meinem Volt 2 lädt nicht mehr. Das Rad habe ich erst seit Juli.
gold: warranty_fault | sketch: product_question
after the patch: warranty_fault
HARD_TEST now: 0.425"Der Akku von meinem Volt 2 lädt nicht mehr" (my Volt 2's battery no longer charges) has no English cue, so the fallback fires. One word fixes it and lifts the score from 0.4 to 0.425: one message in 80. Every further fix is another word in another language. That treadmill is why Module 4 learns cues from data.
These are the first rows of the course's comparison table; every later method must beat them on the same 80 messages.
7. The course setup cell and one optional LLM call (Essential)
Most of this course needs no key; hosted LLMs arrive in Module 10. Meet the setup cell now so it is familiar later.
The setup cell builds one model object, llm, for the provider you choose: Groq or Gemini in Colab, Ollama on your own computer. Store a free key in Colab's Secrets panel (course-guide/02). No key? Skip the next four cells; nothing after them needs them.
python
%pip install -q -U langchain langgraph langchain-groq langchain-google-genai langchain-ollamapython
import os
from langchain.chat_models import init_chat_model
PROVIDER = "groq" # "groq" or "gemini" in Colab; "ollama" on your own computer
MODELS = {"groq": "groq:openai/gpt-oss-20b", "gemini": "google_genai:gemini-3.5-flash-lite", "ollama": "ollama:llama3.2:3b"}
KEY = {"groq": "GROQ_API_KEY", "gemini": "GOOGLE_API_KEY", "ollama": None}[PROVIDER]
if KEY and KEY not in os.environ:
from google.colab import userdata # keys live in Colab's Secrets panel
os.environ[KEY] = userdata.get(KEY)
llm = init_chat_model(MODELS[PROVIDER], temperature=0, max_retries=6)Code walkthrough
PROVIDERis the one word you change;MODELSnames a model per provider (checked 14 October 2026).- The
ifblock copies your key from Colab's Secrets into an environment variable. init_chat_model(...)builds the model object, sending nothing;max_retries=6retries on "too many requests".
Providers sometimes reject a reply, so every call goes through ask.
python
def ask(model, messages):
"""Returns (reply, None), or (None, reason) after two refusals."""
for attempt in range(2): # one retry: the next reply is usually fine
try:
return model.invoke(messages), None
except Exception as err:
last = err
return None, f"provider_error: {type(last).__name__}"Code walkthrough model.invoke(messages) sends the messages. After a second error ask returns None and a reason such as provider_error: BadRequestError, which is scored as wrong.
The optional call asks the model to triage Megan's message from Ines's definitions. It sends only the core sentence, without her name or number, and is off until you set TRY_LLM = True.
python
TRY_LLM = False # set to True to send one message (needs the setup cell above)
def triage(model, text):
definitions = "\n".join(f"{name}: {meaning}" for name, meaning in LABELS.items())
messages = [("system", "You sort a bike shop's inbox. Reply with one intent name only.\n" + definitions),
("user", text)]
reply, reason = ask(model, messages)
return reply.text.strip() if reply else reason
if TRY_LLM:
print("LLM:", triage(llm, RUNNING["core"]), "| gold:", RUNNING["intent"])
else:
print("Not sent. Set TRY_LLM = True to try it.")Code walkthrough
messagesis a list of (role, text) pairs:systemsets the task with Ines's seven definitions,useris the text to sort.reply.text.strip()is the answer; ifaskgave up, the reason comes back instead.RUNNING["core"]is the message without greeting, phone or signature.
text
Not sent. Set TRY_LLM = True to try it.One answer on one message proves little; Module 10 scores LLMs on all 80 messages, with cost.
In production
Teams running text systems tend to work this way. They keep a small hand-labelled test set that looks like real traffic and distrust scores on generated data. They record the simplest baseline first, so each new model is judged against a number. They keep rules where rules are exact, such as order-number formats, and learned models where wording varies. And they log the evidence behind each decision, so a misrouted message can be explained.
What you added to the Tidewell inbox
- The shared data, loaded and fingerprinted: 1,800 messages and the 80
HARD_TESTmessages. - Rosa's five jobs, traced on Megan's message.
- A pipeline sketch with the
predict(texts)shape. - The first scores on
HARD_TEST: majority baseline 0.225, keyword rules 0.4. - The setup cell and the
askguard.
Check yourself
- What does Python hold when it reads Megan's sentence?
- Which NLP use is each: choosing a help article; finding an order number; flagging phishing?
- Which score would you report to Rosa for the keyword rules, 0.661 or 0.4, and why?
- Why compute a majority baseline nobody would ship?
- Why is adding "akku" not a good way to handle German?
- A string of 128 characters, and no meaning.
- Search; extraction; moderation.
- 0.4 on
HARD_TEST. Template messages repeat wordings and flatter the rules; hand-written ones are closer to real mail. - It is the floor: a model that cannot beat 0.225 has learned nothing.
- Each word fixes one message in one language; learning cues from labelled data scales.
Exercise
Write a router of at most three rules that beats the majority baseline on HARD_TEST. Edit MY_RULES, run the check, and try to beat 0.4.
python
MY_RULES = [("warranty_fault", ["battery", "broken", "faulty"]),
("returns_refund", ["refund", "return"]),
("order_status", ["order", "parcel", "delivery"])]
def my_predict(texts):
out = []
for t in texts:
low = clean(t)
out.append(next((intent for intent, cues in MY_RULES if any(c in low for c in cues)), majority))
return out
score = accuracy([h["intent"] for h in HARD_TEST], my_predict([h["text"] for h in HARD_TEST]))
print("my router on HARD_TEST:", round(score, 3))
assert len(MY_RULES) <= 3, "use at most three rules"
assert score > 0.225, "not yet better than always saying order_status"
print("passed: better than the majority baseline")Code walkthrough my_predict reuses clean and falls back to the majority label; the two assert lines are the check.
text
my router on HARD_TEST: 0.325
passed: better than the majority baselineKey terms
| Term | Meaning |
|---|---|
| Natural language processing (NLP) | Methods that turn human text into labels, fields, rankings or other text |
| String | Text as a computer holds it: characters with no meaning attached |
| Routing | Picking one label from a fixed list (classification) |
| Extraction | Pulling fields such as an order number out of text |
| Token | A unit of text later steps count, often a word |
| Feature | Evidence a decision uses, such as "battery appears" |
| Gold label | The correct answer, from Ines's definitions |
| Baseline | The score of the simplest reasonable method |
| Accuracy | The share of messages labelled correctly |
| Fingerprint | A short hash: equal fingerprints mean identical data |
Sources
- Joseph Weizenbaum, "ELIZA: A Computer Program for the Study of Natural Language Communication Between Man and Machine", Communications of the ACM 9(1), January 1966. https://doi.org/10.1145/365153.365168
- Joseph Weizenbaum, Computer Power and Human Reason: From Judgment to Calculation, W. H. Freeman, 1976.
- Python Software Foundation, "Text Sequence Type: str", Python 3 documentation. https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str
- Groq, "Supported models" (checked 14 October 2026, per course-guide/02). https://console.groq.com/docs/models
- Google, "Gemini models" (checked 14 October 2026, per course-guide/02). https://ai.google.dev/gemini-api/docs/models