Lesson 3: Regular expressions that find order numbers and dates
Module 1 · Lesson 3 · About 45 minutes · Needs: basic Python (strings, lists, loops, functions); no NLP
The situation
It is Wednesday 9 September 2026. Ines opens message TW-M0435 from Kofi Asante, who wants a tracking link. Before she can answer she copies five things by hand: his order number, email, phone, postcode and the date of Tidewell's last reply. She does that about thirty times a day, and Dev wants the same fields from every repair booking.
Rosa asks two questions: how much of that copying can code do, and how often will the code be wrong? This lesson writes the patterns and then measures them against the gold labels in the shared data.
A real case, first
In 1968 Ken Thompson published "Regular Expression Search Algorithm" in Communications of the ACM. It described a way to search text with regular expressions: short patterns that describe a whole family of strings, such as "TW, a hyphen, then six digits". Thompson built the method into the text editors he wrote.
One editor command, g/re/p (globally, find a regular expression, print the matching lines), became a separate Unix tool named after it: grep. Regular expressions have been the standard way to find patterns in text ever since, and Python's re module carries the same idea.
Thompson's patterns find text by its shape, not its meaning. An order number has a shape, and so does a postcode. But a pattern finds only the shapes you thought of, and this lesson ends by measuring exactly that.
What you will be able to do
- Read and write patterns built from literals, classes, quantifiers, anchors, boundaries, groups and flags.
- Choose between
search,findall,finditerandsub. - Fix a greedy pattern that swallows a message.
- Extract order numbers, emails, phones, postcodes, dates and prices, and normalise dates and prices.
- Measure a pattern's precision and recall against gold labels.
1. Literals, character classes and quantifiers (Essential)
Three kinds of piece build every pattern in this lesson; learn them and you can read most patterns at work.
First, paste blocks 1, 2 and 3 from course-guide/06 unchanged, then check the fingerprint. Nothing needs installing: re comes with Python.
from datetime import date, timedelta
COURSE_DATE = date(2026, 10, 14) # facts, versions and model names checked on this day
DATA_AS_OF = date(2026, 8, 31) # the inbox snapshot: count days to this, never to the clock
WINDOW = (date(2026, 3, 2), date(2026, 8, 30)) # 26 weeks of messages, Monday to Sunday
SEED = 20260831
EVENTS = { # fixed dates the data and the lessons share
"spam_wave": (date(2026, 4, 6), date(2026, 4, 19)),
"battery_notice": date(2026, 6, 9), # email to Volt 2 owners about pack batch VB-0426
"eu_shipping": date(2026, 7, 1), # Germany, Spain and France
"summer_sale": (date(2026, 7, 15), date(2026, 7, 28)),
"carrier_change": date(2026, 8, 10), # ParcelNorth -> SwiftDrop
}
BATCH = "VB-0426"
STAFF = ["Rosa", "Ines", "Dev", "Callum", "Amara"]
PRODUCTS = [ # (sku, name, price in GBP, kind)
("TW-VOLT2-BLK", "Volt 2", 2199, "ebike"),
("TW-VOLT2-STP", "Volt 2 Step-Through", 2249, "ebike"),
("TW-HARB-GRN", "Harbour", 649, "bike"),
("TW-HKID-20R", "Harbour Kids 20", 329, "bike"),
("TW-CAUS-GRY", "Causeway", 1149, "bike"),
("TW-PANN-SET", "Commuter Pannier Set", 89, "accessory"),
("TW-BATT-500", "Volt 2 battery pack (500 Wh)", 549, "accessory"),
("TW-LOCK-DLK", "LockSure D-lock", 45, "accessory"),
("TW-LITE-BCN", "Beacon Light Set", 39, "accessory"),
("TW-SERV-BAS", "Basic Service", 65, "service"),
("TW-SERV-FUL", "Full Service", 129, "service"),
("TW-REPR-PUN", "Puncture Repair", 18, "service"),
("TW-DIAG-EBK", "E-bike Diagnostic", 45, "service"),
("TW-GIFT-CRD", "Gift Card", 50, "gift"), # sold from 25 to 200 GBP; 50 is the usual
]
INTENTS = ["order_status", "returns_refund", "repair_booking", "product_question",
"warranty_fault", "praise", "spam"]
LABELS = { # Ines's written definitions, used for every label in the course
"order_status": "Where an order is, delivery problems, missing items, invoices and order documents.",
"returns_refund": "Sending something back, exchanges without a fault, cancellations, refunds and wrong charges.",
"repair_booking": "Booking, moving or asking the price of a service or repair for wear and tear.",
"product_question": "Questions before or after buying: sizes, stock, range, compatibility, delivery areas.",
"warranty_fault": "A product that is faulty or arrived damaged, including the battery batch VB-0426.",
"praise": "Thanks, compliments and good reviews, with no request that needs action.",
"spam": "Unsolicited selling, phishing and scams.",
}
CUSTOMERS = [ # (first, last, gender, language, home town); all fictional
("Joan", "Hartley", "f", "en", "Clifton"), ("Oliver", "Bennett", "m", "en", "Bristol"),
("Amelia", "Clarke", "f", "en", "Bath"), ("Mohammed", "Khan", "m", "en", "Bristol"),
("Aisha", "Begum", "f", "en", "Bedminster"), ("Daniel", "O'Connor", "m", "en", "Bristol"),
("Priya", "Patel", "f", "en", "Swindon"), ("Arjun", "Sharma", "m", "en", "Bristol"),
("Grace", "Okafor", "f", "en", "Cardiff"), ("Emeka", "Eze", "m", "en", "Bristol"),
("Hannah", "Lewis", "f", "en", "Stroud"), ("Tom", "Hughes", "m", "en", "Gloucester"),
("Siobhan", "Murphy", "f", "en", "Bristol"), ("Liam", "Walsh", "m", "en", "Bath"),
("Mei", "Chen", "f", "en", "Bristol"), ("Wei", "Zhang", "m", "en", "Clifton"),
("Fatima", "Hussain", "f", "en", "Bristol"), ("Yusuf", "Ali", "m", "en", "Bedminster"),
("Zara", "Ahmed", "f", "en", "Taunton"), ("Jack", "Robinson", "m", "en", "Bristol"),
("Ella", "Morgan", "f", "en", "Cardiff"), ("Rhys", "Evans", "m", "en", "Cardiff"),
("Megan", "Price", "f", "en", "Weston-super-Mare"), ("Ben", "Carter", "m", "en", "Bristol"),
("Lucy", "Ward", "f", "en", "Bath"), ("Kofi", "Asante", "m", "en", "Bristol"),
("Ama", "Boateng", "f", "en", "Bristol"), ("Olusegun", "Adeyemi", "m", "en", "Swindon"),
("Funmi", "Adebayo", "f", "en", "Bristol"), ("Raj", "Gupta", "m", "en", "Gloucester"),
("Neha", "Kapoor", "f", "en", "Bristol"), ("Tomasz", "Nowak", "m", "en", "Bristol"),
("Agnieszka", "Kowalska", "f", "en", "Stroud"), ("Stefan", "Ionescu", "m", "en", "Bristol"),
("Ioana", "Popescu", "f", "en", "Bath"), ("Hiroshi", "Tanaka", "m", "en", "Bristol"),
("Lukas", "Becker", "m", "de", "Berlin"), ("Anna", "Schmidt", "f", "de", "Hamburg"),
("Jonas", "Weber", "m", "de", "München"), ("Lea", "Fischer", "f", "de", "Köln"),
("Mehmet", "Yılmaz", "m", "de", "Berlin"), ("Elif", "Demir", "f", "de", "Leipzig"),
("Felix", "Wagner", "m", "de", "Hamburg"), ("Sophie", "Müller", "f", "de", "München"),
("Javier", "García", "m", "es", "Madrid"), ("Lucía", "Fernández", "f", "es", "Sevilla"),
("Alejandro", "Martín", "m", "es", "Valencia"), ("Carmen", "Ruiz", "f", "es", "Madrid"),
("Mateo", "López", "m", "es", "Barcelona"), ("Valentina", "Torres", "f", "es", "Bilbao"),
("Youssef", "El Amrani", "m", "es", "Barcelona"), ("Nuria", "Puig", "f", "es", "Barcelona"),
("Hugo", "Martin", "m", "fr", "Paris"), ("Camille", "Dubois", "f", "fr", "Lyon"),
("Lucas", "Bernard", "m", "fr", "Nantes"), ("Chloé", "Lefèvre", "f", "fr", "Bordeaux"),
("Karim", "Benali", "m", "fr", "Lille"), ("Amina", "Diallo", "f", "fr", "Paris"),
("Théo", "Moreau", "m", "fr", "Lyon"), ("Léa", "Laurent", "f", "fr", "Nantes"),
]
print(len(PRODUCTS), "products,", len(CUSTOMERS), "customers,", len(INTENTS), "intents")
print("women:", sum(c[2] == "f" for c in CUSTOMERS), "men:", sum(c[2] == "m" for c in CUSTOMERS))
print({lang: sum(c[3] == lang for c in CUSTOMERS) for lang in ["en", "de", "es", "fr"]})
print((DATA_AS_OF - WINDOW[0]).days, "days from the first message day to DATA_AS_OF")Code walkthrough
- Block 1 of course-guide/06: dates, products, customers and Ines's labels.
Output
14 products, 60 customers, 7 intents
women: 30 men: 30
{'en': 36, 'de': 8, 'es': 8, 'fr': 8}
182 days from the first message day to DATA_AS_OFimport random, re
# Templates: one dict per wording. Keys en, de, es, fr hold the same message in each language
# (English-only wordings have just "en"). {UPPER} slots become gold entities; {lower} slots do not.
T = {
"order_status": [
{"s": "neu", "en": "Hi, I placed an order on {DATE} (order {ORDER_ID}, item: {PRODUCT}) and it still says processing. When will it ship?",
"de": "Hallo, ich habe am {DATE} bestellt (Bestellung {ORDER_ID}, Artikel: {PRODUCT}) und der Status steht immer noch auf in Bearbeitung. Wann wird sie verschickt?",
"es": "Hola, hice un pedido el {DATE} (pedido {ORDER_ID}, artículo: {PRODUCT}) y sigue en preparación. ¿Cuándo lo enviaréis?",
"fr": "Bonjour, j'ai passé commande le {DATE} (commande {ORDER_ID}, article : {PRODUCT}) et elle est toujours en préparation. Quand sera-t-elle expédiée ?"},
{"s": "neg", "en": "Order {ORDER_ID} was due on {DATE} and nothing has arrived. The {carrier} tracking has not changed for three days. Where is my parcel ({PRODUCT})?",
"de": "Bestellung {ORDER_ID} sollte am {DATE} ankommen, aber es ist nichts da. Die Sendungsverfolgung von {carrier} hat sich seit drei Tagen nicht geändert. Wo ist mein Paket ({PRODUCT})?",
"es": "El pedido {ORDER_ID} tenía que llegar el {DATE} y no ha llegado nada. El seguimiento de {carrier} no cambia desde hace tres días. ¿Dónde está mi paquete ({PRODUCT})?",
"fr": "La commande {ORDER_ID} devait arriver le {DATE} et rien n'est arrivé. Le suivi {carrier} n'a pas bougé depuis trois jours. Où est mon colis ({PRODUCT}) ?"},
{"s": "neu", "en": "Could you send me the tracking link for order {ORDER_ID}? The confirmation email never came through to {email}.",
"de": "Könnten Sie mir den Tracking-Link für Bestellung {ORDER_ID} schicken? Die Bestätigung ist nie bei {email} angekommen.",
"es": "¿Me podéis enviar el enlace de seguimiento del pedido {ORDER_ID}? El correo de confirmación nunca llegó a {email}.",
"fr": "Pouvez-vous m'envoyer le lien de suivi de la commande {ORDER_ID} ? L'e-mail de confirmation n'est jamais arrivé sur {email}."},
{"s": "neg", "from": "2026-08-10", "w": 3, "en": "Third time {carrier} has said 'attempted delivery' when I was in all day. Order {ORDER_ID}. This never happened with your old courier."},
{"s": "neu", "en": "any update on {ORDER_ID}? need the {PRODUCT} before {DATE} if poss"},
],
"returns_refund": [
{"s": "neu", "en": "I'd like to return the {PRODUCT} from order {ORDER_ID}. It's unused and still in the box. How do I send it back?",
"de": "Ich möchte {PRODUCT} aus Bestellung {ORDER_ID} zurückgeben. Unbenutzt und noch originalverpackt. Wie schicke ich es zurück?",
"es": "Quiero devolver {PRODUCT} del pedido {ORDER_ID}. Está sin usar y en su caja. ¿Cómo lo envío?",
"fr": "Je voudrais retourner {PRODUCT} de la commande {ORDER_ID}. Il n'a pas servi et il est encore dans sa boîte. Comment le renvoyer ?"},
{"s": "neg", "en": "I sent the return for {ORDER_ID} back on {DATE} and still no refund of {MONEY}. How long does this take?",
"de": "Ich habe die Rücksendung für {ORDER_ID} am {DATE} abgeschickt und noch keine Erstattung über {MONEY} bekommen. Wie lange dauert das?",
"es": "Envié la devolución del pedido {ORDER_ID} el {DATE} y todavía no me habéis devuelto {MONEY}. ¿Cuánto tarda?",
"fr": "J'ai renvoyé la commande {ORDER_ID} le {DATE} et je n'ai toujours pas reçu le remboursement de {MONEY}. Combien de temps faut-il ?"},
{"s": "neu", "en": "The {KIDBIKE} is the wrong size for my daughter. Can I swap it or get a refund? Order {ORDER_ID}.",
"de": "{KIDBIKE} ist für meine Tochter die falsche Größe. Kann ich umtauschen oder bekomme ich das Geld zurück? Bestellung {ORDER_ID}.",
"es": "{KIDBIKE} es de una talla equivocada para mi hija. ¿Puedo cambiarlo o me devolvéis el dinero? Pedido {ORDER_ID}.",
"fr": "{KIDBIKE} n'est pas à la bonne taille pour ma fille. Puis-je l'échanger ou être remboursé ? Commande {ORDER_ID}."},
{"s": "neg", "en": "You charged me {MONEY} twice for order {ORDER_ID}. Please refund the duplicate payment."},
{"s": "neu", "en": "want to cancel order {ORDER_ID} before it ships, changed my mind about the {PRODUCT}. refund to my card pls"},
],
"repair_booking": [
{"s": "neu", "en": "Can I book a {SERVICE} for {DATE}? It's for my bike ({BIKE}). I can drop it at the workshop in the morning.",
"de": "Kann ich für den {DATE} einen Termin für {SERVICE} buchen? Es geht um mein Rad ({BIKE}). Ich kann es morgens in die Werkstatt bringen.",
"es": "¿Puedo reservar {SERVICE} para el {DATE}? Es para mi bici ({BIKE}). Puedo dejarla en el taller por la mañana.",
"fr": "Puis-je réserver {SERVICE} pour le {DATE} ? C'est pour mon vélo ({BIKE}). Je peux le déposer à l'atelier le matin."},
{"s": "neu", "en": "My bike ({BIKE}) needs a repair: {wear}. Can I bring it in or send it to you? I live in {LOC}.",
"de": "Mein Rad ({BIKE}) muss repariert werden: {wear}. Kann ich es vorbeibringen oder einschicken? Ich wohne in {LOC}.",
"es": "Mi bici ({BIKE}) necesita una reparación: {wear}. ¿Puedo llevarla o enviárosla? Vivo en {LOC}.",
"fr": "Mon vélo ({BIKE}) a besoin d'une réparation : {wear}. Puis-je l'apporter ou vous l'envoyer ? J'habite à {LOC}."},
{"s": "neu", "en": "Please can I move my {SERVICE} booking ({ticket}) from {DATE} to the following week? Something came up.",
"de": "Kann ich meinen Termin für {SERVICE} ({ticket}) vom {DATE} auf die Woche danach verschieben? Mir ist etwas dazwischengekommen.",
"es": "¿Puedo cambiar mi cita de {SERVICE} ({ticket}) del {DATE} a la semana siguiente? Me ha surgido algo.",
"fr": "Puis-je déplacer mon rendez-vous {SERVICE} ({ticket}) du {DATE} à la semaine suivante ? J'ai un empêchement."},
{"s": "neu", "en": "hiya, how much is a {SERVICE} and can u fit me in {DATE}? bike is a {BIKE}, cheers"},
{"s": "neu", "en": "I'd like to book the {SERVICE} for my {BIKE}. Is {MONEY} still the price? Any day after {DATE} works."},
],
"product_question": [
{"s": "neu", "en": "Does the {BIKE} come in a smaller frame? I'm 158 cm tall.",
"de": "Gibt es {BIKE} auch mit einem kleineren Rahmen? Ich bin 1,58 m groß.",
"es": "¿{BIKE} viene en una talla de cuadro más pequeña? Mido 1,58 m.",
"fr": "{BIKE} existe-t-il avec un cadre plus petit ? Je mesure 1,58 m."},
{"s": "neu", "en": "What range does the {EBIKE} get on one charge, and is the {ACCESSORY} compatible with it?",
"de": "Welche Reichweite hat {EBIKE} mit einer Akkuladung, und passt {ACCESSORY} dazu?",
"es": "¿Qué autonomía tiene {EBIKE} con una carga, y es compatible con {ACCESSORY}?",
"fr": "Quelle autonomie a {EBIKE} avec une charge, et {ACCESSORY} est-il compatible ?"},
{"s": "neu", "en": "Is the {PRODUCT} in stock? The website says {MONEY} but I can't add it to the basket.",
"de": "Ist {PRODUCT} auf Lager? Auf der Website steht {MONEY}, aber ich kann es nicht in den Warenkorb legen.",
"es": "¿Tenéis {PRODUCT} en stock? En la web pone {MONEY} pero no puedo añadirlo a la cesta.",
"fr": "Est-ce que {PRODUCT} est en stock ? Le site indique {MONEY} mais je n'arrive pas à l'ajouter au panier."},
{"s": "neu", "en": "Do you deliver to {LOC}? And do bikes arrive fully built or do I need to assemble them?"},
{"s": "neu", "en": "quick q: whats the difference between the {SERVICE} and the {SERVICE2}?"},
],
"warranty_fault": [
{"s": "neg", "en": "My e-bike ({EBIKE}) is only four months old and {fault}. Is this covered by the warranty?",
"de": "Mein E-Bike ({EBIKE}) ist erst vier Monate alt und {fault}. Ist das von der Garantie abgedeckt?",
"es": "Mi bici eléctrica ({EBIKE}) solo tiene cuatro meses y {fault}. ¿Lo cubre la garantía?",
"fr": "Mon vélo électrique ({EBIKE}) n'a que quatre mois et {fault}. Est-ce couvert par la garantie ?"},
{"s": "neg", "from": "2026-06-09", "w": 4, "en": "I got your email about battery batch {batch}. My pack has that code and {fault}. What happens now?",
"de": "Ich habe Ihre E-Mail zur Akku-Charge {batch} bekommen. Mein Akku hat diesen Code und {fault}. Wie geht es jetzt weiter?",
"es": "Recibí vuestro correo sobre el lote de baterías {batch}. Mi batería tiene ese código y {fault}. ¿Qué hago ahora?",
"fr": "J'ai reçu votre e-mail sur le lot de batteries {batch}. Ma batterie porte ce code et {fault}. Que se passe-t-il maintenant ?"},
{"s": "neg", "en": "The {PRODUCT} arrived with {damage}. I've attached photos. Can you send a replacement?",
"de": "{PRODUCT} kam mit {damage} an. Fotos sind im Anhang. Können Sie Ersatz schicken?",
"es": "{PRODUCT} llegó con {damage}. Adjunto fotos. ¿Podéis enviarme uno nuevo?",
"fr": "{PRODUCT} est arrivé avec {damage}. Je joins des photos. Pouvez-vous m'en envoyer un autre ?"},
{"s": "neg", "en": "Spoke snapped on my {BIKE} after two rides. That's not normal wear, surely? Bought {DATE}, order {ORDER_ID}."},
{"s": "neg", "en": "{EBIKE} battery dead again. Second pack in a month. I want this sorted properly this time."},
],
"praise": [
{"s": "pos", "en": "Just wanted to say thank you to {PERSON} in the workshop. My bike ({BIKE}) rides like new after the {SERVICE}.",
"de": "Ich wollte mich nur bei {PERSON} in der Werkstatt bedanken. Mein Rad ({BIKE}) fährt sich nach {SERVICE} wie neu.",
"es": "Solo quería dar las gracias a {PERSON} del taller. Mi bici ({BIKE}) va como nueva después de {SERVICE}.",
"fr": "Je voulais juste remercier {PERSON} de l'atelier. Mon vélo ({BIKE}) roule comme neuf après {SERVICE}."},
{"s": "pos", "en": "Love my new bike ({BIKE}). Fast delivery to {LOC} and it was easy to set up. Five stars.",
"de": "Ich liebe mein neues Rad ({BIKE}). Schnelle Lieferung nach {LOC} und leicht aufzubauen. Fünf Sterne.",
"es": "Me encanta mi nueva bici ({BIKE}). Entrega rápida a {LOC} y fácil de montar. Cinco estrellas.",
"fr": "J'adore mon nouveau vélo ({BIKE}). Livraison rapide à {LOC} et facile à monter. Cinq étoiles."},
{"s": "pos", "en": "Great service from {PERSON}, who sorted my problem in one email. Thank you!",
"de": "Super Service von {PERSON}, das Problem war mit einer E-Mail gelöst. Vielen Dank!",
"es": "Un servicio genial de {PERSON}, que me lo resolvió en un solo correo. ¡Gracias!",
"fr": "Super service de la part de {PERSON}, qui a réglé mon problème en un seul e-mail. Merci !"},
{"s": "pos", "en": "the {ACCESSORY} is brilliant, fits perfectly. cheers"},
{"s": "pos", "en": "Honestly the best bike shop in {LOC}. Friendly, quick and fair prices."},
],
"spam": [
{"s": "neu", "en": "Dear partner, we offer guaranteed first-page search ranking for your online store. Reply today for a {MONEY} discount."},
{"s": "neu", "en": "Exclusive partnership opportunity: invest in our crypto bike-sharing token and earn 300% returns. Contact {email} now."},
{"s": "neu", "en": "Your mailbox storage is full. Verify your account within 24 hours at http://mail-verify.example/login to avoid suspension."},
{"s": "neu", "en": "Hello, I am an influencer with 50k followers. Send me a free {BIKE} and I will post about your brand."},
{"s": "neu", "en": "Congratulations! You have been selected for a {MONEY} gift voucher. Click http://prize-claim.example to claim."},
],
}
PHRASES = { # paired fillers for the {lower} slots: (en, de, es, fr)
"fault": [("the battery will not charge", "der Akku lädt nicht mehr", "la batería no carga", "la batterie ne se charge plus"),
("the motor cuts out on hills", "der Motor setzt am Berg aus", "el motor se para en las cuestas", "le moteur se coupe dans les montées"),
("the display shows error E07", "das Display zeigt Fehler E07", "la pantalla muestra el error E07", "l'écran affiche l'erreur E07"),
("the battery runs flat very quickly", "der Akku ist sehr schnell leer", "la batería se agota muy rápido", "la batterie se vide très vite")],
"wear": [("the back tyre has a puncture", "der Hinterreifen hat einen Platten", "la rueda trasera está pinchada", "le pneu arrière est crevé"),
("the gears keep slipping", "die Gangschaltung springt ständig", "los cambios saltan todo el rato", "les vitesses sautent sans arrêt"),
("the brake pads are worn out", "die Bremsbeläge sind abgenutzt", "las pastillas de freno están gastadas", "les plaquettes de frein sont usées"),
("the chain squeaks", "die Kette quietscht", "la cadena chirría", "la chaîne grince")],
"damage": [("a cracked mudguard", "einem gebrochenen Schutzblech", "un guardabarros roto", "un garde-boue fendu"),
("a bent front wheel", "einem verbogenen Vorderrad", "la rueda delantera doblada", "une roue avant voilée"),
("a broken clip", "einem kaputten Clip", "un clip roto", "un clip cassé")],
}
SUBJECTS = {"order_status": ["Where is my order?", "Wo bleibt meine Bestellung?", "¿Dónde está mi pedido?", "Où est ma commande ?"],
"returns_refund": ["Return and refund", "Rücksendung", "Devolución", "Retour"],
"repair_booking": ["Booking a service", "Werkstatttermin", "Cita en el taller", "Rendez-vous atelier"],
"product_question": ["Product question", "Frage zum Produkt", "Pregunta sobre un producto", "Question sur un produit"],
"warranty_fault": ["Fault with my bike", "Defekt", "Avería", "Panne"],
"praise": ["Thank you!", "Danke!", "¡Gracias!", "Merci !"],
"spam": ["Partnership opportunity"] * 4}
FORM_TOPIC = {"order_status": "Order enquiry", "returns_refund": "Returns", "repair_booking": "Workshop",
"product_question": "Product question", "warranty_fault": "Faulty item", "praise": "Feedback", "spam": "Other"}
CHANNELS = {"order_status": (5, 4, 1), "returns_refund": (6, 4, 0), "repair_booking": (4, 6, 0), # email, web form, review
"product_question": (4, 6, 0), "warranty_fault": (5, 3, 2), "praise": (3, 1, 6), "spam": (1, 0, 0)}
LANGS = ["en", "de", "es", "fr"]
MONTHS = {"en": "January February March April May June July August September October November December",
"de": "Januar Februar März April Mai Juni Juli August September Oktober November Dezember",
"es": "enero febrero marzo abril mayo junio julio agosto septiembre octubre noviembre diciembre",
"fr": "janvier février mars avril mai juin juillet août septembre octobre novembre décembre"}
TOWNS = {"en": ["Bristol", "Bath", "Clifton", "Cardiff", "Stroud"], "de": ["Berlin", "Hamburg", "München", "Köln"],
"es": ["Madrid", "Barcelona", "Valencia", "Sevilla"], "fr": ["Paris", "Lyon", "Nantes", "Lille"]}
SPAMMERS = [("Growth Partners", "offers@growth-partners.example"), ("Crypto Ride", "team@cryptoride.example"),
("Mail Admin", "admin@mail-verify.example"), ("Prize Desk", "win@prize-claim.example")]
def fmt_date(d, lang, rng):
m = MONTHS[lang].split()[d.month - 1]
styles = {"en": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}", f"{m} {d.day}{'th' if 10 < d.day < 14 else {1: 'st', 2: 'nd', 3: 'rd'}.get(d.day % 10, 'th')}"],
"de": [f"{d.day}. {m}", f"{d.day:02d}.{d.month:02d}."], "es": [f"{d.day} de {m}", f"{d.day}/{d.month}"],
"fr": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}/{d.year}"]}[lang]
return rng.choice(styles) if rng else styles[0]
def fmt_money(gbp, lang, rng):
styles = {"en": [f"£{gbp:,}", f"£{gbp:,}.00", f"{gbp} pounds"], "de": [f"{gbp:,} £".replace(",", "."), f"{gbp} Pfund"],
"es": [f"{gbp:,} libras".replace(",", "."), f"{gbp} £"], "fr": [f"{gbp:,} £".replace(",", " "), f"{gbp} livres"]}[lang]
return rng.choice(styles) if rng else styles[0]
def typo(word, rng):
i = rng.randrange(1, len(word) - 1)
return rng.choice([word[:i] + word[i + 1] + word[i] + word[i + 2:], word[:i] + word[i + 1:], word[:i] + word[i] + word[i:]])
print(sum(len(v) for v in T.values()), "wordings,", sum("de" in t for v in T.values() for t in v), "of them in all four languages")Code walkthrough
- Block 2 of course-guide/06: every message wording. This lesson reuses its
MONTHSandLANGS.
Output
35 wordings, 18 of them in all four languagesimport html, unicodedata
POOLS = {"PRODUCT": ["bike", "ebike", "accessory"], "BIKE": ["bike", "ebike"], "EBIKE": ["ebike"],
"ACCESSORY": ["accessory"], "SERVICE": ["service"], "SERVICE2": ["service"], "KIDBIKE": ["kids"]}
EXTRA = {"greet": {"en": ["Hi,", "Hello,", "Hello Tidewell,", "Dear Sir or Madam,"], "de": ["Hallo,", "Guten Tag,"],
"es": ["Hola,", "Buenos días,"], "fr": ["Bonjour,", "Madame, Monsieur,"]},
"sign": {"en": ["Thanks,", "Many thanks,", "Cheers,"], "de": ["Danke,", "Viele Grüße,"],
"es": ["Gracias,", "Un saludo,"], "fr": ["Merci,", "Cordialement,"]},
"phone": {"en": "You can call me on ", "de": "Sie erreichen mich unter ", "es": "Me podéis llamar al ",
"fr": "Vous pouvez m'appeler au "},
"quote": {"en": ("On ", ", Tidewell Cycles <help@tidewell.example> wrote:"), "de": ("Am ", " schrieb Tidewell Cycles <help@tidewell.example>:"),
"es": ("El ", ", Tidewell Cycles <help@tidewell.example> escribió:"), "fr": ("Le ", ", Tidewell Cycles <help@tidewell.example> a écrit :")},
"again": {"en": "Sorry, sending this again as I have not heard back. ", "de": "Entschuldigung, ich schicke das noch einmal. ",
"es": "Perdón, lo envío otra vez porque no he recibido respuesta. ", "fr": "Désolé, je renvoie ce message car je n'ai pas eu de réponse. "}}
LABEL_OF = {"PRODUCT": "PRODUCT", "BIKE": "PRODUCT", "EBIKE": "PRODUCT", "ACCESSORY": "PRODUCT", "SERVICE": "PRODUCT",
"SERVICE2": "PRODUCT", "KIDBIKE": "PRODUCT", "ORDER_ID": "ORDER_ID", "DATE": "DATE", "MONEY": "MONEY",
"LOC": "LOC", "PERSON": "PERSON"}
def fill(template, values, lang, rng):
"""Template text -> list of (text, label or None) pieces, formatting dates and money for `lang`."""
pieces = []
for part in re.split(r"(\{\w+\})", template):
if not (part.startswith("{") and part.endswith("}")):
pieces.append((part, None)) if part else None
continue
slot = part[1:-1]
v = values[slot]
if slot == "DATE":
v = fmt_date(v, lang, rng)
elif slot == "MONEY":
v = fmt_money(v, lang, rng)
elif slot in ("fault", "wear", "damage"):
v = PHRASES[slot][v][LANGS.index(lang)]
pieces.append((v, LABEL_OF.get(slot)))
return pieces
def email_of(first, last):
"""'Mehmet', 'Yılmaz' -> 'mehmet.yilmaz@...'; one fixed address per customer, on a reserved .example domain."""
plain = unicodedata.normalize("NFKD", f"{first}.{last}".lower().replace("ı", "i").replace(" ", "").replace("'", ""))
plain = "".join(ch for ch in plain if ch.isascii())
return plain + "@" + ["post.example", "mailbox.example", "inbox.example"][len(first + last) % 3]
def make_messages(n=1800, seed=SEED):
rng = random.Random(seed)
days = [WINDOW[0] + timedelta(i) for i in range((WINDOW[1] - WINDOW[0]).days + 1)]
base = {"order_status": 26, "returns_refund": 16, "repair_booking": 17, "product_question": 18,
"warranty_fault": 11, "praise": 6, "spam": 6}
def weights(d):
w = dict(base)
w["spam"] *= 8 if EVENTS["spam_wave"][0] <= d <= EVENTS["spam_wave"][1] else 1
w["warranty_fault"] *= 3.5 if EVENTS["battery_notice"] <= d <= date(2026, 7, 31) else 2 if d.month == 8 else 1
w["order_status"] *= (2.5 if EVENTS["summer_sale"][0] <= d <= EVENTS["summer_sale"][1] else 1) * (1.6 if d >= EVENTS["carrier_change"] else 1)
w["returns_refund"] *= 1.3 if date(2026, 7, 29) <= d <= date(2026, 8, 15) else 1
vol = (0.6 if d.weekday() >= 5 else 1.0) * (1.5 if d >= EVENTS["eu_shipping"] else 1.0)
return [w[i] * vol for i in INTENTS]
W = [weights(d) for d in days]
n_dup = round(n * 0.03)
picks = sorted(rng.choices(range(len(days)), weights=[sum(w) for w in W], k=n - n_dup))
by_kind = {k: [p for p in PRODUCTS if p[3] == k] for k in ["bike", "ebike", "accessory", "service"]}
by_kind["kids"] = [p for p in PRODUCTS if p[1] == "Harbour Kids 20"]
out = []
for di in picks:
d = days[di]
intent = rng.choices(INTENTS, weights=W[di])[0]
lang = "en" if intent == "spam" else rng.choices(LANGS, weights=[95, 2, 1.5, 1.5] if d < EVENTS["eu_shipping"] else [58, 16, 14, 12])[0]
options = [t for t in T[intent] if lang in t and date.fromisoformat(t.get("from", "2026-01-01")) <= d]
tmpl = rng.choices(options, weights=[t.get("w", 1) for t in options])[0]
if intent == "spam":
cust, email = rng.choice(SPAMMERS)
first, last, gender, town = cust, "", None, None
else:
first, last, gender, _, town = rng.choice([c for c in CUSTOMERS if c[3] == lang])
email = email_of(first, last)
v = {}
for slot, kinds in POOLS.items():
pool = [p for k in kinds for p in by_kind[k]]
pick = rng.choice([p for p in pool if p[1] != v.get("SERVICE")])
v[slot] = pick[1]
v[slot + "_sku"], v[slot + "_price"] = pick[0], pick[2]
main = next((s for s in POOLS if "{" + s + "}" in tmpl["en"]), None)
future = intent == "repair_booking" or "before {DATE}" in tmpl["en"]
v["DATE"] = d + timedelta(rng.randint(2, 14)) if future else d - timedelta(rng.randint(3, 20))
v["MONEY"] = v[main + "_price"] if main else rng.choice([50, 100, 250, 500])
v["ORDER_ID"] = f"TW-{100000 + di * 120 + rng.randrange(120)}"
lives_there = "live in {LOC}" in tmpl["en"] or "shop in {LOC}" in tmpl["en"]
v["LOC"] = town if lives_there and town else rng.choice(TOWNS[lang])
v["PERSON"] = rng.choice(["Dev", "Dev", "Ines", "Callum"]) if intent == "praise" else first
v.update(carrier="SwiftDrop" if d >= EVENTS["carrier_change"] else "ParcelNorth", email=email, batch=BATCH,
ticket=f"RJ-2026-{300 + di * 2 + rng.randrange(3):04d}", fault=rng.randrange(4), wear=rng.randrange(4),
damage=rng.randrange(3))
core = fill(tmpl[lang], v, lang, rng)
en_ref = "".join(t for t, _ in fill(tmpl["en"], v, "en", None)) if lang != "en" else None
channel = rng.choices(["email", "web form", "review"], weights=CHANNELS[intent])[0]
channel = "web form" if channel == "review" and tmpl["s"] == "neu" else channel # reviews praise or complain
pieces, g = list(core), 0
if channel == "email":
if rng.random() < 0.7:
pieces, g = [(rng.choice(EXTRA["greet"][lang]) + "\n\n", None)] + pieces, 1
if rng.random() < 0.15:
pieces += [("\n" + EXTRA["phone"][lang] + rng.choice(["07700 900", "07700900", "+44 7700 900"]) + f"{rng.randrange(1000):03d}.", None)]
if lang == "en" and intent in ("repair_booking", "order_status") and rng.random() < 0.3:
pieces += [(f"\nMy postcode is BS{rng.randint(1, 16)} {rng.randint(1, 9)}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}.", None)]
if rng.random() < 0.7 and intent != "spam":
pieces += [("\n\n" + rng.choice(EXTRA["sign"][lang]) + "\n", None), (first if rng.random() < 0.6 else f"{first} {last}", "PERSON")]
if rng.random() < 0.15:
pieces += [("\n" + {"en": "Sent from my phone", "de": "Von meinem Handy gesendet", "es": "Enviado desde mi móvil", "fr": "Envoyé de mon téléphone"}[lang], None)]
if rng.random() < 0.25 and intent != "spam":
q = EXTRA["quote"][lang]
prev = fmt_date(d - timedelta(rng.randint(1, 6)), lang, None) + (" de 2026" if lang == "es" else " 2026")
pieces += [("\n\n> " + q[0], None), (prev, "DATE"), (q[1] + "\n> Thanks for your message. We will reply within one working day.", None)]
r = rng.random
if r() < 0.15: # typos
for _ in range(rng.randint(1, 2)):
i = rng.randrange(len(pieces))
if pieces[i][1] is None:
words = pieces[i][0].split(" ")
j = rng.randrange(len(words))
if len(words[j]) >= 5 and words[j].isalpha():
words[j] = typo(words[j], rng)
pieces[i] = (" ".join(words), None)
case = r()
if case < 0.03:
pieces = [(t.upper(), l) for t, l in pieces]
elif case < 0.09:
pieces = [(t.lower(), l) for t, l in pieces]
if r() < 0.08: # emoji after the main text
pieces.insert(g + len(core), (" " + rng.choice(["😡", "😤", "🚲"] if tmpl["s"] == "neg" else ["🙂", "👍", "🚲"]), None))
if r() < 0.05: # odd whitespace
i = next((i for i, (t, l) in enumerate(pieces) if l is None and " " in t), None)
if i is not None:
t = pieces[i][0]
pieces[i] = (t.replace(" ", " ", 1) if r() < 0.5 else t.replace(" ", " ", 1), None)
if channel == "web form" and r() < 0.6: # HTML from the form
pieces = [("<p>", None)] + [(html.escape(t).replace("\n", "<br>\n"), None) if l is None else (t, l) for t, l in pieces] + [("</p>", None)]
if r() < 0.04: # mojibake: UTF-8 bytes read as cp1252
try:
pieces = [(t.encode("utf-8").decode("cp1252"), l) for t, l in pieces]
except UnicodeDecodeError:
pass
body, spans, pos = "", [], 0
for t, l in pieces:
if l:
spans.append((pos, pos + len(t), l))
body, pos = body + t, pos + len(t)
sent = tmpl["s"]
stars = None
if channel == "review":
stars = {"pos": rng.choice([4, 5]), "neu": 3, "neg": rng.choice([1, 2])}[sent]
if r() < 0.08:
stars = {"pos": rng.choice([1, 2]), "neu": rng.choice([1, 5]), "neg": 5}[sent]
subject = {"email": SUBJECTS[intent][LANGS.index(lang)], "web form": "Web form: " + FORM_TOPIC[intent],
"review": "Review: " + (v[main] if main else "Tidewell Cycles")}[channel]
out.append({"date": d.isoformat(), "channel": channel, "language": lang, "customer": f"{first} {last}".strip(),
"gender": gender, "email": email, "subject": subject, "body": body, "intent": intent, "sentiment": sent,
"stars": stars, "entities": spans, "sku": v[main + "_sku"] if main else None,
"order_id": v["ORDER_ID"] if "{ORDER_ID}" in tmpl["en"] else None, "duplicate_of": None,
"core": "".join(t for t, _ in core), "en_ref": en_ref, "label_flipped": False})
originals = [m for m in out if m["intent"] != "spam"]
for m in rng.sample(originals, n_dup): # near-duplicates: the same customer writes again
again, cut = EXTRA["again"][m["language"]], 3 if m["body"].startswith("<p>") else 0 # inside the form's HTML
dup = dict(m, body=m["body"][:cut] + again + m["body"][cut:], entities=[(a + len(again), b + len(again), l) for a, b, l in m["entities"]],
date=min(date.fromisoformat(m["date"]) + timedelta(rng.randint(0, 2)), WINDOW[1]).isoformat(), duplicate_of=m)
out.append(dup)
out.sort(key=lambda m: (m["date"], m["duplicate_of"] is not None))
for i, m in enumerate(out, 1):
m["id"] = f"TW-M{i:04d}"
for m in out:
if m["duplicate_of"] is not None:
m["duplicate_of"] = m["duplicate_of"]["id"]
for m in rng.sample(out, round(n * 0.02)): # deliberate wrong labels, for error analysis
m["intent"], m["label_flipped"] = rng.choice([i for i in INTENTS if i != m["intent"]]), True
split_of = {}
for intent in INTENTS: # stratified 70 / 15 / 15, duplicates follow their original
ids = [m["id"] for m in out if m["intent"] == intent and m["duplicate_of"] is None]
rng.shuffle(ids)
a, b = round(len(ids) * 0.70), round(len(ids) * 0.85)
split_of.update({x: "train" if k < a else "dev" if k < b else "test" for k, x in enumerate(ids)})
for m in out:
m["split"] = split_of[m["duplicate_of"] or m["id"]]
m["period"] = "before_july" if m["date"] < "2026-07-01" else "from_july"
return out
MESSAGES = make_messages()
print(len(MESSAGES), "messages;", MESSAGES[0]["date"], "to", MESSAGES[-1]["date"])
print(MESSAGES[0]["id"], "|", MESSAGES[0]["subject"], "|", MESSAGES[0]["body"][:80].replace("\n", " "))Code walkthrough
- Block 3 of course-guide/06: the generator. Each message carries gold
entities, the exact character spans of its order numbers, dates and prices.
Output
1800 messages; 2026-03-02 to 2026-08-30
TW-M0001 | Web form: Workshop | <p>Mon vélo (Causeway) a besoin d'une réparation : les plaquettes de freiimport hashlib, json
def fingerprint(*datasets):
"""The first 12 hex digits of a SHA-256 hash of the data: equal fingerprints mean identical data."""
blob = json.dumps(datasets, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
return hashlib.sha256(blob).hexdigest()[:12]
print("messages ", fingerprint(MESSAGES))Code walkthrough
- The fingerprint function from block 12 of course-guide/06. It must print
294611361162.
Output
messages 294611361162A literal is a character that matches itself: TW- finds those three characters in that order. A character class matches any one character from a set: \d any digit, [A-Z] any capital letter. The picture lines TW-\d{6} up under Kofi's order number.
A quantifier says how many times the piece before it may repeat: {6} exactly six, + one or more, * zero or more, ? zero or one, {2,3} two or three. Without quantifiers you would write \d\d\d\d\d\d.
import re
kofi = next(m for m in MESSAGES if m["id"] == "TW-M0435")
text = kofi["body"]
print(text)
print("---")
print(re.search("order", text)) # a literal: these five letters, in this order
print(re.search(r"TW-\d", text)) # \d: any one digit
print(re.search(r"TW-\d{6}", text)) # {6}: exactly six of the piece before
print(re.search(r"BS\d+ \d[A-Z]{2}", text)) # + one or more; [A-Z] one capital letterCode walkthrough
re.search(pattern, text)finds the first place the pattern fits. It returns a match object, showing the position (span) and the matched text, orNone.- The
rbefore a pattern makes a raw string, so Python passes\dtoreunchanged. Write every pattern this way.
Output
Hello,
Could you send me the tracking link for order TW-106489? The confirmation email never came through to kofi.asante@mailbox.example.
You can call me on 07700 900070.
My postcode is BS12 2FR.
Cheers,
Kofi
> On 24 April 2026, Tidewell Cycles <help@tidewell.example> wrote:
> Thanks for your message. We will reply within one working day.
---
<re.Match object; span=(48, 53), match='order'>
<re.Match object; span=(54, 58), match='TW-1'>
<re.Match object; span=(54, 63), match='TW-106489'>
<re.Match object; span=(187, 195), match='BS12 2FR'>phone = "07700 900070"
for pattern in [r"\d", r"\d?", r"\d*", r"\d+", r"\d{2,3}", r"\d{5}\s\d{6}", r"\w+", r".+"]:
print(f"{pattern:14} -> {re.search(pattern, phone).group()!r}")Code walkthrough
.group()returns the matched text.\sis any whitespace,\wany letter, digit or underscore,.any character except a line break.- Each quantified pattern starts at the first character and takes as much as it may:
\d+stops at the space.
Output
\d -> '0'
\d? -> '0'
\d* -> '07700'
\d+ -> '07700'
\d{2,3} -> '077'
\d{5}\s\d{6} -> '07700 900070'
\w+ -> '07700'
.+ -> '07700 900070'2. Anchors, word boundaries and flags (Essential)
A pattern that fits too much is as bad as one that fits too little. Boundaries and anchors pin a match to a place, and flags change how the whole pattern behaves.
A word boundary, \b, matches no character: it matches the gap between a word character and a non-word character. Customers mistype: TW-1064890 has seven digits. TW-\d{6} matches its first six; \bTW-\d{6}\b refuses, because no gap follows the sixth digit.
An anchor pins a match to a position: ^ the start and $ the end, of the whole text by default. Kofi's postcode line sits mid-message, so ^My postcode is fails until the re.MULTILINE flag makes ^ and $ work at every line.
samples = ["order TW-106489?", "order TW-1064890", "code XTW-106489", "pack TW-BATT-500"]
for s in samples:
plain = re.search(r"TW-\d{6}", s)
bounded = re.search(r"\bTW-\d{6}\b", s)
print(f"{s:18} plain: {plain.group() if plain else None!s:10} bounded: {bounded.group() if bounded else None}")
print(re.search(r"^My postcode is (.+)$", text)) # ^ and $: start and end of the whole text
print(re.search(r"^My postcode is (.+)$", text, re.MULTILINE)) # with MULTILINE: of each lineCode walkthrough
- The samples: a correct order number, one digit too many, one glued to a letter, and a product code (SKU).
- The third argument of
re.searchis a flag.
Output
order TW-106489? plain: TW-106489 bounded: TW-106489
order TW-1064890 plain: TW-106489 bounded: None
code XTW-106489 plain: TW-106489 bounded: None
pack TW-BATT-500 plain: None bounded: None
None
<re.Match object; span=(172, 196), match='My postcode is BS12 2FR.'>The flag that matters most here is re.IGNORECASE. The generator writes 6% of messages in lower case, as people do on phones, so some order numbers arrive as tw-100474.
ORDER_CS = re.compile(r"\bTW-\d{6}\b")
ORDER = re.compile(r"\bTW-\d{6}\b", re.IGNORECASE)
gold = [m["body"][a:b] for m in MESSAGES for a, b, label in m["entities"] if label == "ORDER_ID"]
print(len(gold), "gold order numbers")
print("case-sensitive finds", sum(bool(ORDER_CS.search(g)) for g in gold))
print("IGNORECASE finds ", sum(bool(ORDER.search(g)) for g in gold))
print([g for g in gold if not ORDER_CS.search(g)][:3])
POSTCODE = re.compile(r"""
\b [A-Z]{1,2} \d{1,2} [A-Z]? # outward part: area letters, district number, maybe a letter
\s? \d [A-Z]{2} \b # inward part: one digit and two letters
""", re.VERBOSE | re.IGNORECASE)
print(POSTCODE.search(text).group())Code walkthrough
re.compile(pattern, flags)builds a reusable pattern object once;ORDER.search(...)then needs no pattern text.goldholds the text of everyORDER_IDspan:body[a:b]is exactly the order number.re.VERBOSEallows comments and ignores spaces in the pattern, so the real space is written\s?. Flags combine with|.
Output
751 gold order numbers
case-sensitive finds 709
IGNORECASE finds 751
['tw-100474', 'tw-100474', 'tw-101077']
BS12 2FR3. Groups, alternation and named groups (Essential)
Dev's booking form wants the postcode area on its own and every phone in one style. Groups let a pattern match and also hand back its parts.
Alternation, |, means "this or that". Inbox phones start 07700 900, 07700900 or +44 7700 900, and (?:\+44\s?|0)7700\s?900\d{3} covers all three. (?:...) groups the choices without keeping them; \+ is an escaped plus, since a bare + is a quantifier.
A capturing group, plain brackets, keeps what its part matched. A named group, (?P<area>...), keeps it under a name, easier to read later than group(1).
PHONE = re.compile(r"(?:\+44\s?|0)7700\s?900\d{3}\b")
for s in ["call me on 07700 900070.", "call me on 07700900158.", "call me on +44 7700 900762."]:
print(PHONE.search(s).group())
PC = re.compile(r"\b(?P<area>[A-Z]{1,2}\d{1,2}[A-Z]?)\s?(?P<unit>\d[A-Z]{2})\b", re.IGNORECASE)
m = PC.search(text)
print(m.group(0), "|", m.group(1), "|", m.group("unit"))
print(m.groupdict())
print(re.search(r"(order|parcel) (TW-\d{6})", text).groups())Code walkthrough
- The three strings are the inbox's phone styles, from Kofi's, Fatima Hussain's and Raj Gupta's messages.
group(0)is the whole match,group(1)the first group,group("unit")a named one;groupdict()returns all named groups.groups()returns every capturing group as a tuple.
Output
07700 900070
07700900158
+44 7700 900762
BS12 2FR | BS12 | 2FR
{'area': 'BS12', 'unit': '2FR'}
('order', 'TW-106489')4. Python's re module: search, findall, finditer and sub (Essential)
Python runs a pattern in several ways; picking the wrong one causes quiet bugs.
search returns the first match anywhere; match checks only the start. findall returns every match as a string; finditer returns every match object, with positions. sub replaces every match, with a string or with what a function returns.
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
print(EMAIL.search(text), EMAIL.match(text)) # first match anywhere; only at the start
print(EMAIL.findall(text)) # every match, as strings
print(re.findall(r"@([\w-]+)", text)) # with a group, findall returns the group only
for m in ORDER.finditer(text): # every match, with its span
print("found", m.span(), "gold", [(a, b) for a, b, l in kofi["entities"] if l == "ORDER_ID"])
def one_style(m):
return "07700 900" + m.group()[-3:] # the last three digits, in one fixed style
calls = "07700900158, +44 7700 900762 and 07700 900070"
print(PHONE.sub(one_style, calls))
print(PHONE.subn("[PHONE]", calls))Code walkthrough
[\w.+-]+is the part before the@; inside a class most symbols lose their special meaning.findallreturns two addresses: Kofi's and Tidewell's own, in the quoted reply. Given a capturing group it returns only the group: a common trap.m.span()is(start, end), and it equals the gold span the generator recorded.subcallsone_stylefor each match and inserts its return value;subnalso counts replacements. Masking personal data properly is Lesson 4's job.
Output
<re.Match object; span=(110, 137), match='kofi.asante@mailbox.example'> None
['kofi.asante@mailbox.example', 'help@tidewell.example']
['mailbox', 'tidewell']
found (54, 63) gold [(54, 63)]
07700 900158, 07700 900762 and 07700 900070
('[PHONE], [PHONE] and [PHONE]', 3)That span check is the basis of all measurement in section 8. Not every message has an order number, though. Run ORDER.search("refund??").group() and you will see:
AttributeError: 'NoneType' object has no attribute 'group'search returned None. Test the result (if m:) before reading it.
5. Greedy versus lazy matching (Essential)
Quantifiers take as much as they can. That is right for digits and wrong for anything with a closing mark, such as an HTML tag, and 459 inbox messages arrived from the web form wrapped in HTML.
Quantifiers are greedy by default: .* runs to the end of the text, then gives characters back only until the rest fits. In <.*> the last > closes </p>, so the match is the whole message. Add ? and the quantifier is lazy: <.*?> stops at the first >.
Break it: a greedy tag remover
form = next(m["body"] for m in MESSAGES if m["id"] == "TW-M0271")
print(repr(form))
print(repr(re.sub(r"<.*>", "", form))) # greedy: from the first < to the last >
print(repr(re.sub(r"<.*?>", "", form))) # lazy: each tag on its own
print(repr(re.sub(r"<[^>]*>", "", form))) # a class that cannot cross a >
print(ORDER.findall(re.sub(r"<.*>", "", form)), ORDER.findall(re.sub(r"<.*?>", "", form)))Code walkthrough
TW-M0271is one web-form line inside<p>and</p>.- The greedy pattern deletes everything, order number included, without an error.
[^>]is "any character except>", so<[^>]*>cannot run past a tag: often clearer than a lazy quantifier.
Output
'<p>any update on TW-104542? need the Causeway before 21/04 if poss</p>'
''
'any update on TW-104542? need the Causeway before 21/04 if poss'
'any update on TW-104542? need the Causeway before 21/04 if poss'
[] ['TW-104542']A greedy mistake fails silently: the message becomes empty and every later step works on nothing. That is why patterns are measured on the whole inbox, not one example.
6. Order numbers, emails, phone numbers and postcodes (Essential)
These four fields have fixed shapes, so they are where rules shine. Give each pattern a name and run them together.
Order numbers are always TW- and six digits, phones use the drama range 07700 900, and postcodes follow the UK shape. Real email addresses allow far more than this pattern does; it covers what customers write. A postcode with the right shape may still not exist.
FIELDS = {"order": ORDER, "email": EMAIL, "phone": PHONE, "postcode": PC}
def extract_fields(body):
"""Every field found in a message, as {name: [matched text, ...]}."""
return {name: [m.group() for m in rx.finditer(body)] for name, rx in FIELDS.items()}
print(extract_fields(text))
for name, rx in FIELDS.items():
hits = [m.group() for msg in MESSAGES for m in rx.finditer(msg["body"])]
print(f"{name:9} {len(hits):4} matches in {sum(bool(rx.search(msg['body'])) for msg in MESSAGES):4} messages")Code walkthrough
FIELDSnames each pattern, so adding a field is one line.extract_fieldsusesfinditerand.group(), notfindall, becausePChas groups andfindallwould return tuples.
Output
{'order': ['TW-106489'], 'email': ['kofi.asante@mailbox.example', 'help@tidewell.example'], 'phone': ['07700 900070'], 'postcode': ['BS12 2FR']}
order 751 matches in 751 messages
email 366 matches in 344 messages
phone 165 matches in 165 messages
postcode 81 matches in 81 messagesTwo surprises. Of the 366 addresses found, 213 are Tidewell's own, quoted back: a pattern finds shapes, not owners. And TW-M1222 says "My psotcode is BS10 2SS": a rule keyed on "postcode is" would miss it; the shape pattern does not.
7. Dates and prices written many ways (Essential)
Dev needs a booking date as a date, not as "August 14th", and £2,199.00, 2199 pounds and 2.199 £ are one price. Extraction has two steps: find the text, then normalise it into one standard form.
Inbox dates come in many shapes across four languages: 15/03, 24 April 2026, August 14th, 12. Juni, 20 de julio. One pattern with three branches covers them: numbers with a slash or dot, a day before a month name, and a month name before a day.
MONTH = "|".join(w for lang in LANGS for w in MONTHS[lang].split()) # 48 month names, from block 2
DATE = re.compile(rf"""
\b (?:[0-2]?\d|3[01]) [/.] (?:0?[1-9]|1[0-2]) (?:[/.](?:\d{{4}})?)? # 15/03 12.06. 12/06/2026
| \b \d{{1,2}} \.? \s (?:de\s)? (?:{MONTH}) \b (?:\s(?:de\s)?\d{{4}})? # 1 April 12. Juni 20 de julio
| \b (?:{MONTH}) \s \d{{1,2}} (?:st|nd|rd|th) \b # August 14th
""", re.IGNORECASE | re.VERBOSE)
print(DATE.findall(text))
for lang in LANGS:
found = [m.group() for msg in MESSAGES if msg["language"] == lang for m in DATE.finditer(msg["body"])]
print(lang, len(found), list(dict.fromkeys(found))[:4])Code walkthrough
MONTHjoins every month name in the four languages with|.rf"..."is a raw f-string:{MONTH}is filled in, and{{4}}is doubled sorereceives{4}.- The first branch allows days 0 to 31 and months 1 to 12, so
£50.00is not a date.dict.fromkeysdrops repeats, keeping order.
Output
['24 April 2026']
en 628 ['March 12th', '10 March', '05/03', '25 February 2026']
de 85 ['11. März', '27. März 2026', '27.04.', '19. April 2026']
es 49 ['3/3', '17 de abril', '24 de abril de 2026', '16/4']
fr 47 ['1 mars 2026', '15 mars', '22 février', '10 mars 2026']to_iso turns each found date into ISO form, 2026-07-03, assuming 2026 when no year is written. In the UK 3/7 is 3 July; in the United States it is 7 March. A pattern cannot know which. Tidewell writes day first, so the code does too.
MONTH_NUMBER = {w.lower(): i % 12 + 1 for i, w in enumerate(MONTH.split("|"))}
def to_iso(found, year=2026):
"""'August 14th' -> '2026-08-14'; '12/06/2026' -> '2026-06-12'. Day before month, as in the UK."""
nums = [int(n) for n in re.findall(r"\d+", found)]
word = re.search(r"[^\W\d_]{3,}", found) # three or more letters: a month name
if word and word.group().lower() in MONTH_NUMBER:
day, month = nums[0], MONTH_NUMBER[word.group().lower()]
year = nums[1] if len(nums) > 1 else year
else:
day, month = nums[0], nums[1]
year = nums[2] if len(nums) > 2 else year
return f"{year}-{month:02d}-{day:02d}"
for s in ["15/03", "August 14th", "24 April 2026", "12. Juni", "20 de julio", "12/06/2026", "3/7"]:
print(f"{s:14} -> {to_iso(s)}")Code walkthrough
- Month names come in twelves, so position
i % 12 + 1is the month number. [^\W\d_]means a word character that is not a digit or underscore: a letter in any language,äincluded.- With a month name, the numbers are day then year; without one, day, month, year.
Output
15/03 -> 2026-03-15
August 14th -> 2026-08-14
24 April 2026 -> 2026-04-24
12. Juni -> 2026-06-12
20 de julio -> 2026-07-20
12/06/2026 -> 2026-06-12
3/7 -> 2026-07-03Prices work the same way: a pound sign before the number, or a currency word or sign after it, with thousands split by a comma, dot or space.
MONEY = re.compile(r"""
£ \d+ (?:,\d{3})* (?:\.\d{2})? # £649 £2,199 £2,199.00
| \b \d+ (?:[.,\s]\d{3})* \s? (?:£|pounds|Pfund|libras|livres) # 549 pounds 2.199 £ 2 199 £
""", re.IGNORECASE | re.VERBOSE)
def to_gbp(found):
"""'£2,199.00', '2.199 £' and '2199 pounds' -> 2199."""
return int(re.sub(r"\D", "", re.sub(r"[.,]00\b", "", found)))
for s in ["£649", "£2,199.00", "2199 pounds", "2.199 £", "2 199 £", "45 Pfund"]:
print(f"{s:12} -> {MONEY.fullmatch(s) is not None} {to_gbp(s)}")Code walkthrough
fullmatchsucceeds only if the pattern covers the whole string: a quick test on sample values.to_gbpremoves pence written.00(the inbox has no other pence), then every non-digit,\D.
Output
£649 -> True 649
£2,199.00 -> True 2199
2199 pounds -> True 2199
2.199 £ -> True 2199
2 199 £ -> True 2199
45 Pfund -> True 45These patterns were written for the inbox's own templates. Real mail is less tidy: eight sentences copied from HARD_TEST, the hand-written test set in course-guide/06, show the gap.
HAND = [ # (text, gold date or price), copied from HARD_TEST in course-guide/06
("yo the gears on my Causeway are proper knackered, can u lot sort it sat?", "sat"),
("Can I bring my Harbour Kids 20 in for a Basic Service next week? Any day but Wednesday.", "Wednesday"),
("Is the Full Service still £129? If so I'll book it for the 2nd of September.", "2nd of September"),
("Ordered on 3/7, still waiting on TW-119880. SwiftDrop says 'exception'. What does that mean?", "3/7"),
("Der Akku von meinem Volt 2 lädt nicht mehr. Das Rad habe ich erst seit Juli.", "Juli"),
("Est-ce que je peux prendre rendez-vous pour un Full Service le 3 septembre ?", "3 septembre"),
("Super, die Erstattung von 549 £ dauert jetzt schon drei Wochen. Ganz toll.", "549 £"),
("Gana 5.000 euros al mes desde casa. Haz clic aquí.", "5.000 euros"),
]
for sentence, want in HAND:
got = [m.group() for m in DATE.finditer(sentence)] + [m.group() for m in MONEY.finditer(sentence)]
print(f"{want:17} {'found' if want in got else 'MISSED':7} {got}")Code walkthrough
- Each pair is a sentence and the value a person labelled in it. Every miss is a shape no pattern was written for: "sat", a weekday, "2nd of", a month alone, euros.
Output
sat MISSED []
Wednesday MISSED []
2nd of September MISSED ['£129']
3/7 found ['3/7']
Juli MISSED []
3 septembre found ['3 septembre']
549 £ found ['549 £']
5.000 euros MISSED []8. A pattern's precision and recall (Essential)
"It works on Kofi's message" is not a measurement. Rosa asked how often the code is wrong, and two numbers answer it.
Compare what a pattern found with the gold spans. Precision is the share of found spans that are right: of what it flagged, how much was correct. Recall is the share of gold spans it found: of what was there, how much it caught.
def score(rx, label):
"""Exact-span precision and recall of a compiled pattern against the gold entities in MESSAGES."""
gold = {(m["id"], a, b) for m in MESSAGES for a, b, l in m["entities"] if l == label}
found = {(m["id"], x.start(), x.end()) for m in MESSAGES for x in rx.finditer(m["body"])}
right = len(gold & found)
missed = [by_id[i]["body"][a:b] for i, a, b in sorted(gold - found)]
return {"found": len(found), "right": right, "false alarms": len(found) - right,
"precision": round(right / len(found), 3), "recall": round(right / len(gold), 3), "missed": missed[:5]}
by_id = {m["id"]: m for m in MESSAGES}
LOOSE = re.compile(r"[A-Z]+-\d+", re.IGNORECASE)
for name, rx, label in [("TW-\\d{6}", ORDER_CS, "ORDER_ID"), ("[A-Z]+-\\d+ (i)", LOOSE, "ORDER_ID"),
("\\bTW-\\d{6}\\b (i)", ORDER, "ORDER_ID"), ("DATE", DATE, "DATE"), ("MONEY", MONEY, "MONEY")]:
print(f"{name:17}", score(rx, label))Code walkthrough
- Spans are stored as
(message id, start, end), so the set intersectiongold & foundis the right answers andgold - foundthe misses. A span counts only if both ends match exactly. LOOSE, "letters, a hyphen, digits", is a tempting shortcut.(i)marks theIGNORECASEpatterns.
Output
TW-\d{6} {'found': 709, 'right': 709, 'false alarms': 0, 'precision': 1.0, 'recall': 0.944, 'missed': ['tw-100474', 'tw-100474', 'tw-101077', 'tw-102156', 'tw-102121']}
[A-Z]+-\d+ (i) {'found': 968, 'right': 751, 'false alarms': 217, 'precision': 0.776, 'recall': 1.0, 'missed': []}
\bTW-\d{6}\b (i) {'found': 751, 'right': 751, 'false alarms': 0, 'precision': 1.0, 'recall': 1.0, 'missed': []}
DATE {'found': 809, 'right': 807, 'false alarms': 2, 'precision': 0.998, 'recall': 0.998, 'missed': ['9 août 2026', '17 août 2026']}
MONEY {'found': 276, 'right': 271, 'false alarms': 5, 'precision': 0.982, 'recall': 0.982, 'missed': ['£500.00', '£100', '£50', '£250.00', '£39']}The order patterns show the trade. The case-sensitive one is never wrong but misses 42 numbers: high precision, lower recall. The loose one finds every order number plus 217 things that are not orders, mostly the battery batch VB-0426 and repair tickets. Only the bounded, case-insensitive pattern gets both.
Read the errors before trusting the scores. Every miss is mojibake, text broken by a wrong encoding (Lesson 2): août arrived as août, £ as £. Clean first, then match. The two date false alarms are German prices: in 1.149 £ the date branch sees 1.1. And remember the eight hand-written sentences: near-perfect scores on template messages say the patterns fit the templates, not real mail.
In production
Teams that run extraction rules for years keep every pattern in one file, named and commented, and clean text before matching. Python's documentation notes that re caches recently used patterns, so compiling is mostly for readability. They report precision and recall per field on a labelled set after every change, because a fix for one shape often breaks another, and they read the misses. The regex package (pinned in course-guide/02) helps where re falls short on Unicode. Patterns with nested quantifiers can take exponential time on some inputs: Lesson 4 starts there.
Save the patterns as the first part of inbox/rules.py. %%writefile does not create folders in Colab, so make inbox/ first.
import os
os.makedirs("inbox", exist_ok=True)
if not os.path.exists("inbox/__init__.py"): # an earlier lesson may have made it already
open("inbox/__init__.py", "w").close()
print(sorted(os.listdir("inbox")))Code walkthrough
exist_ok=Truemakes the call safe if the folder exists; an empty__init__.pymakesinboxan importable package, and an existing one is left alone.
Output
['__init__.py']%%writefile inbox/rules.py
"""Field extraction with regular expressions (Module 1, Lesson 3). Lesson 4 adds more."""
import re
ORDER = re.compile(r"\bTW-\d{6}\b", re.IGNORECASE)
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
PHONE = re.compile(r"(?:\+44\s?|0)7700\s?900\d{3}\b")
POSTCODE = re.compile(r"\b(?P<area>[A-Z]{1,2}\d{1,2}[A-Z]?)\s?(?P<unit>\d[A-Z]{2})\b", re.IGNORECASE)
MONTHS = ("January February March April May June July August September October November December "
"Januar Februar März April Mai Juni Juli August September Oktober November Dezember "
"enero febrero marzo abril mayo junio julio agosto septiembre octubre noviembre diciembre "
"janvier février mars avril mai juin juillet août septembre octobre novembre décembre").split()
MONTH = "|".join(MONTHS)
MONTH_NUMBER = {w.lower(): i % 12 + 1 for i, w in enumerate(MONTHS)}
DATE = re.compile(rf"""
\b (?:[0-2]?\d|3[01]) [/.] (?:0?[1-9]|1[0-2]) (?:[/.](?:\d{{4}})?)?
| \b \d{{1,2}} \.? \s (?:de\s)? (?:{MONTH}) \b (?:\s(?:de\s)?\d{{4}})?
| \b (?:{MONTH}) \s \d{{1,2}} (?:st|nd|rd|th) \b
""", re.IGNORECASE | re.VERBOSE)
MONEY = re.compile(r"""
£ \d+ (?:,\d{3})* (?:\.\d{2})?
| \b \d+ (?:[.,\s]\d{3})* \s? (?:£|pounds|Pfund|libras|livres)
""", re.IGNORECASE | re.VERBOSE)
FIELDS = {"order": ORDER, "email": EMAIL, "phone": PHONE, "postcode": POSTCODE, "date": DATE, "money": MONEY}Code walkthrough
- The patterns from this lesson, with
MONTHScopied from block 2 so the file stands alone.
%%writefile -a inbox/rules.py
def extract_fields(body):
"""Every field found in a message, as {name: [matched text, ...]}."""
return {name: [m.group() for m in rx.finditer(body)] for name, rx in FIELDS.items()}
def to_iso(found, year=2026):
"""'August 14th' -> '2026-08-14'. Day before month, as in the UK; 2026 when no year is written."""
nums = [int(n) for n in re.findall(r"\d+", found)]
word = re.search(r"[^\W\d_]{3,}", found)
if word and word.group().lower() in MONTH_NUMBER:
day, month, year = nums[0], MONTH_NUMBER[word.group().lower()], nums[1] if len(nums) > 1 else year
else:
day, month, year = nums[0], nums[1], nums[2] if len(nums) > 2 else year
return f"{year}-{month:02d}-{day:02d}"
def to_gbp(found):
"""'£2,199.00', '2.199 £' and '2199 pounds' -> 2199."""
return int(re.sub(r"\D", "", re.sub(r"[.,]00\b", "", found)))Code walkthrough
-aappends to the file. These are the lesson's three functions;extract_fieldsnow covers six fields.
from inbox.rules import extract_fields, to_iso, to_gbp
fields = extract_fields(text)
print(fields)
print([to_iso(d) for d in fields["date"]], to_gbp("£2,199.00"))Code walkthrough
- Importing from
inbox.rulesproves the file works on its own.
Output
{'order': ['TW-106489'], 'email': ['kofi.asante@mailbox.example', 'help@tidewell.example'], 'phone': ['07700 900070'], 'postcode': ['BS12 2FR'], 'date': ['24 April 2026'], 'money': []}
['2026-04-24'] 2199What you added to the Tidewell inbox
inbox/rules.py: named patterns for six fields, withextract_fields,to_isoandto_gbp.- A
scorefunction for any pattern's precision and recall against gold spans. - Rosa's first numbers: order numbers at 1.0 precision and recall, dates at 0.998, prices at 0.982, every miss caused by mojibake.
- A warning with evidence: on eight hand-written sentences the patterns found three values.
Check yourself
- Why does
\bTW-\d{6}\brejectTW-1064890?
A seventh digit follows the sixth, and \b needs a gap between a word character and a non-word character. Two digits have no gap between them.
- What does
re.findall(r"(\d{2})/(\d{2})", "15/03 and 21/04")return?
[('15', '03'), ('21', '04')]: with groups, findall returns the groups as tuples. Use finditer and .group() for whole matches.
- A pattern finds 968 spans, 751 right, and there were 751 gold spans. Precision and recall?
Precision 751 / 968 = 0.776, recall 751 / 751 = 1.0: the loose order pattern.
- Which date score should Rosa hear, 0.998 or "3 of 8"?
Both, with the reason: the inbox comes from the templates the patterns were written for; the hand-written sentences are closer to real mail.
Exercise
Repair tickets look like RJ-2026-0412. Write a pattern TICKET that finds them in any case and nothing else, then run the check.
def check_ticket(rx):
hits = [m.group() for msg in MESSAGES for m in rx.finditer(msg["body"])]
expected = sum(msg["body"].lower().count("rj-2026-") for msg in MESSAGES)
assert rx.search("moving my booking (RJ-2026-0412) please") and rx.search("rj-2026-0301")
assert not rx.search("RJ-2026-04123") and not rx.search("TW-104233") and not rx.search("RJ-26-0412")
assert len(hits) == expected, (len(hits), expected)
print(len(hits), "tickets found; all checks passed")Code walkthrough
expectedcounts ticket prefixes with plain string methods, an independent check. The asserts try two strings that must match and three that must not.
TICKET = re.compile(r"\bRJ-2026-\d{4}\b", re.IGNORECASE)
check_ticket(TICKET)Code walkthrough
- A literal prefix, exactly four digits, boundaries at both ends, and
IGNORECASEfor lower-case messages.
Output
59 tickets found; all checks passedKey terms
| Term | Meaning |
|---|---|
| Regular expression | A pattern describing a family of strings by shape |
| Character class | Any one character from a set: \d, [A-Z] |
| Quantifier | How many times the piece before repeats: ?, *, +, {6} |
| Anchor | A position a match must sit at: ^, $ |
| Word boundary | \b, the gap between a word and a non-word character |
| Flag | A switch for the whole pattern, such as IGNORECASE |
| Named group | (?P<name>...), part of a match kept under a name |
| Greedy, lazy | Taking as much as possible, or (with ?) as little |
| Precision | Share of found spans that are right |
| Recall | Share of gold spans that were found |
Sources
- Ken Thompson, "Regular Expression Search Algorithm", Communications of the ACM 11(6), 1968. https://doi.org/10.1145/363347.363387
- Python Software Foundation, "re: Regular expression operations", Python 3 documentation, 2026. https://docs.python.org/3/library/re.html
- A.M. Kuchling, "Regular Expression HOWTO", Python 3 documentation, 2026. https://docs.python.org/3/howto/regex.html