CourseNatural Language Processing: The Complete Course · Module 1: Text as Data and Rules · part 9 of 73
Part 9 · Module 1: Text as Data and Rules

Lesson 3: Regular expressions that find order numbers and dates

42 min read·14 Oct 2026
Module 1 · Lesson 3 · About 45 minutes · Needs: basic Python (strings, lists, loops, functions); no NLP

The situation

It is Wednesday 9 September 2026. Ines opens message TW-M0435 from Kofi Asante, who wants a tracking link. Before she can answer she copies five things by hand: his order number, email, phone, postcode and the date of Tidewell's last reply. She does that about thirty times a day, and Dev wants the same fields from every repair booking.

Rosa asks two questions: how much of that copying can code do, and how often will the code be wrong? This lesson writes the patterns and then measures them against the gold labels in the shared data.

TW-M0435, KOFI ASANTE, FIVE FIELDS ORDER ...link for order TW-106489 ? EMAIL ...through to kofi.asante@mailbox.example PHONE You can call me on 07700 900070 . POSTCODE My postcode is BS12 2FR . DATE > On 24 April 2026 , Tidewell...
Ines copies five fields out of Kofi's message by hand; amber marks the text a pattern has to find.

A real case, first

In 1968 Ken Thompson published "Regular Expression Search Algorithm" in Communications of the ACM. It described a way to search text with regular expressions: short patterns that describe a whole family of strings, such as "TW, a hyphen, then six digits". Thompson built the method into the text editors he wrote.

One editor command, g/re/p (globally, find a regular expression, print the matching lines), became a separate Unix tool named after it: grep. Regular expressions have been the standard way to find patterns in text ever since, and Python's re module carries the same idea.

Thompson's patterns find text by its shape, not its meaning. An order number has a shape, and so does a postcode. But a pattern finds only the shapes you thought of, and this lesson ends by measuring exactly that.

g globally /re/ regular expression p print grep an editor command became a Unix tool
The editor command g/re/p (globally, regular expression, print) gave the Unix tool grep its name.

What you will be able to do

  • Read and write patterns built from literals, classes, quantifiers, anchors, boundaries, groups and flags.
  • Choose between search, findall, finditer and sub.
  • Fix a greedy pattern that swallows a message.
  • Extract order numbers, emails, phones, postcodes, dates and prices, and normalise dates and prices.
  • Measure a pattern's precision and recall against gold labels.

1. Literals, character classes and quantifiers (Essential)

Three kinds of piece build every pattern in this lesson; learn them and you can read most patterns at work.

First, paste blocks 1, 2 and 3 from course-guide/06 unchanged, then check the fingerprint. Nothing needs installing: re comes with Python.

python
from datetime import date, timedelta

COURSE_DATE = date(2026, 10, 14)   # facts, versions and model names checked on this day
DATA_AS_OF = date(2026, 8, 31)     # the inbox snapshot: count days to this, never to the clock
WINDOW = (date(2026, 3, 2), date(2026, 8, 30))   # 26 weeks of messages, Monday to Sunday
SEED = 20260831

EVENTS = {   # fixed dates the data and the lessons share
    "spam_wave": (date(2026, 4, 6), date(2026, 4, 19)),
    "battery_notice": date(2026, 6, 9),      # email to Volt 2 owners about pack batch VB-0426
    "eu_shipping": date(2026, 7, 1),          # Germany, Spain and France
    "summer_sale": (date(2026, 7, 15), date(2026, 7, 28)),
    "carrier_change": date(2026, 8, 10),      # ParcelNorth -> SwiftDrop
}
BATCH = "VB-0426"
STAFF = ["Rosa", "Ines", "Dev", "Callum", "Amara"]

PRODUCTS = [   # (sku, name, price in GBP, kind)
    ("TW-VOLT2-BLK", "Volt 2", 2199, "ebike"),
    ("TW-VOLT2-STP", "Volt 2 Step-Through", 2249, "ebike"),
    ("TW-HARB-GRN", "Harbour", 649, "bike"),
    ("TW-HKID-20R", "Harbour Kids 20", 329, "bike"),
    ("TW-CAUS-GRY", "Causeway", 1149, "bike"),
    ("TW-PANN-SET", "Commuter Pannier Set", 89, "accessory"),
    ("TW-BATT-500", "Volt 2 battery pack (500 Wh)", 549, "accessory"),
    ("TW-LOCK-DLK", "LockSure D-lock", 45, "accessory"),
    ("TW-LITE-BCN", "Beacon Light Set", 39, "accessory"),
    ("TW-SERV-BAS", "Basic Service", 65, "service"),
    ("TW-SERV-FUL", "Full Service", 129, "service"),
    ("TW-REPR-PUN", "Puncture Repair", 18, "service"),
    ("TW-DIAG-EBK", "E-bike Diagnostic", 45, "service"),
    ("TW-GIFT-CRD", "Gift Card", 50, "gift"),          # sold from 25 to 200 GBP; 50 is the usual
]
INTENTS = ["order_status", "returns_refund", "repair_booking", "product_question",
           "warranty_fault", "praise", "spam"]
LABELS = {   # Ines's written definitions, used for every label in the course
    "order_status": "Where an order is, delivery problems, missing items, invoices and order documents.",
    "returns_refund": "Sending something back, exchanges without a fault, cancellations, refunds and wrong charges.",
    "repair_booking": "Booking, moving or asking the price of a service or repair for wear and tear.",
    "product_question": "Questions before or after buying: sizes, stock, range, compatibility, delivery areas.",
    "warranty_fault": "A product that is faulty or arrived damaged, including the battery batch VB-0426.",
    "praise": "Thanks, compliments and good reviews, with no request that needs action.",
    "spam": "Unsolicited selling, phishing and scams.",
}

CUSTOMERS = [   # (first, last, gender, language, home town); all fictional
    ("Joan", "Hartley", "f", "en", "Clifton"), ("Oliver", "Bennett", "m", "en", "Bristol"),
    ("Amelia", "Clarke", "f", "en", "Bath"), ("Mohammed", "Khan", "m", "en", "Bristol"),
    ("Aisha", "Begum", "f", "en", "Bedminster"), ("Daniel", "O'Connor", "m", "en", "Bristol"),
    ("Priya", "Patel", "f", "en", "Swindon"), ("Arjun", "Sharma", "m", "en", "Bristol"),
    ("Grace", "Okafor", "f", "en", "Cardiff"), ("Emeka", "Eze", "m", "en", "Bristol"),
    ("Hannah", "Lewis", "f", "en", "Stroud"), ("Tom", "Hughes", "m", "en", "Gloucester"),
    ("Siobhan", "Murphy", "f", "en", "Bristol"), ("Liam", "Walsh", "m", "en", "Bath"),
    ("Mei", "Chen", "f", "en", "Bristol"), ("Wei", "Zhang", "m", "en", "Clifton"),
    ("Fatima", "Hussain", "f", "en", "Bristol"), ("Yusuf", "Ali", "m", "en", "Bedminster"),
    ("Zara", "Ahmed", "f", "en", "Taunton"), ("Jack", "Robinson", "m", "en", "Bristol"),
    ("Ella", "Morgan", "f", "en", "Cardiff"), ("Rhys", "Evans", "m", "en", "Cardiff"),
    ("Megan", "Price", "f", "en", "Weston-super-Mare"), ("Ben", "Carter", "m", "en", "Bristol"),
    ("Lucy", "Ward", "f", "en", "Bath"), ("Kofi", "Asante", "m", "en", "Bristol"),
    ("Ama", "Boateng", "f", "en", "Bristol"), ("Olusegun", "Adeyemi", "m", "en", "Swindon"),
    ("Funmi", "Adebayo", "f", "en", "Bristol"), ("Raj", "Gupta", "m", "en", "Gloucester"),
    ("Neha", "Kapoor", "f", "en", "Bristol"), ("Tomasz", "Nowak", "m", "en", "Bristol"),
    ("Agnieszka", "Kowalska", "f", "en", "Stroud"), ("Stefan", "Ionescu", "m", "en", "Bristol"),
    ("Ioana", "Popescu", "f", "en", "Bath"), ("Hiroshi", "Tanaka", "m", "en", "Bristol"),
    ("Lukas", "Becker", "m", "de", "Berlin"), ("Anna", "Schmidt", "f", "de", "Hamburg"),
    ("Jonas", "Weber", "m", "de", "München"), ("Lea", "Fischer", "f", "de", "Köln"),
    ("Mehmet", "Yılmaz", "m", "de", "Berlin"), ("Elif", "Demir", "f", "de", "Leipzig"),
    ("Felix", "Wagner", "m", "de", "Hamburg"), ("Sophie", "Müller", "f", "de", "München"),
    ("Javier", "García", "m", "es", "Madrid"), ("Lucía", "Fernández", "f", "es", "Sevilla"),
    ("Alejandro", "Martín", "m", "es", "Valencia"), ("Carmen", "Ruiz", "f", "es", "Madrid"),
    ("Mateo", "López", "m", "es", "Barcelona"), ("Valentina", "Torres", "f", "es", "Bilbao"),
    ("Youssef", "El Amrani", "m", "es", "Barcelona"), ("Nuria", "Puig", "f", "es", "Barcelona"),
    ("Hugo", "Martin", "m", "fr", "Paris"), ("Camille", "Dubois", "f", "fr", "Lyon"),
    ("Lucas", "Bernard", "m", "fr", "Nantes"), ("Chloé", "Lefèvre", "f", "fr", "Bordeaux"),
    ("Karim", "Benali", "m", "fr", "Lille"), ("Amina", "Diallo", "f", "fr", "Paris"),
    ("Théo", "Moreau", "m", "fr", "Lyon"), ("Léa", "Laurent", "f", "fr", "Nantes"),
]

print(len(PRODUCTS), "products,", len(CUSTOMERS), "customers,", len(INTENTS), "intents")
print("women:", sum(c[2] == "f" for c in CUSTOMERS), "men:", sum(c[2] == "m" for c in CUSTOMERS))
print({lang: sum(c[3] == lang for c in CUSTOMERS) for lang in ["en", "de", "es", "fr"]})
print((DATA_AS_OF - WINDOW[0]).days, "days from the first message day to DATA_AS_OF")

Code walkthrough

  • Block 1 of course-guide/06: dates, products, customers and Ines's labels.

Output

text
14 products, 60 customers, 7 intents
women: 30 men: 30
{'en': 36, 'de': 8, 'es': 8, 'fr': 8}
182 days from the first message day to DATA_AS_OF
python
import random, re

# Templates: one dict per wording. Keys en, de, es, fr hold the same message in each language
# (English-only wordings have just "en"). {UPPER} slots become gold entities; {lower} slots do not.
T = {
 "order_status": [
  {"s": "neu", "en": "Hi, I placed an order on {DATE} (order {ORDER_ID}, item: {PRODUCT}) and it still says processing. When will it ship?",
   "de": "Hallo, ich habe am {DATE} bestellt (Bestellung {ORDER_ID}, Artikel: {PRODUCT}) und der Status steht immer noch auf in Bearbeitung. Wann wird sie verschickt?",
   "es": "Hola, hice un pedido el {DATE} (pedido {ORDER_ID}, artículo: {PRODUCT}) y sigue en preparación. ¿Cuándo lo enviaréis?",
   "fr": "Bonjour, j'ai passé commande le {DATE} (commande {ORDER_ID}, article : {PRODUCT}) et elle est toujours en préparation. Quand sera-t-elle expédiée ?"},
  {"s": "neg", "en": "Order {ORDER_ID} was due on {DATE} and nothing has arrived. The {carrier} tracking has not changed for three days. Where is my parcel ({PRODUCT})?",
   "de": "Bestellung {ORDER_ID} sollte am {DATE} ankommen, aber es ist nichts da. Die Sendungsverfolgung von {carrier} hat sich seit drei Tagen nicht geändert. Wo ist mein Paket ({PRODUCT})?",
   "es": "El pedido {ORDER_ID} tenía que llegar el {DATE} y no ha llegado nada. El seguimiento de {carrier} no cambia desde hace tres días. ¿Dónde está mi paquete ({PRODUCT})?",
   "fr": "La commande {ORDER_ID} devait arriver le {DATE} et rien n'est arrivé. Le suivi {carrier} n'a pas bougé depuis trois jours. Où est mon colis ({PRODUCT}) ?"},
  {"s": "neu", "en": "Could you send me the tracking link for order {ORDER_ID}? The confirmation email never came through to {email}.",
   "de": "Könnten Sie mir den Tracking-Link für Bestellung {ORDER_ID} schicken? Die Bestätigung ist nie bei {email} angekommen.",
   "es": "¿Me podéis enviar el enlace de seguimiento del pedido {ORDER_ID}? El correo de confirmación nunca llegó a {email}.",
   "fr": "Pouvez-vous m'envoyer le lien de suivi de la commande {ORDER_ID} ? L'e-mail de confirmation n'est jamais arrivé sur {email}."},
  {"s": "neg", "from": "2026-08-10", "w": 3, "en": "Third time {carrier} has said 'attempted delivery' when I was in all day. Order {ORDER_ID}. This never happened with your old courier."},
  {"s": "neu", "en": "any update on {ORDER_ID}? need the {PRODUCT} before {DATE} if poss"},
 ],
 "returns_refund": [
  {"s": "neu", "en": "I'd like to return the {PRODUCT} from order {ORDER_ID}. It's unused and still in the box. How do I send it back?",
   "de": "Ich möchte {PRODUCT} aus Bestellung {ORDER_ID} zurückgeben. Unbenutzt und noch originalverpackt. Wie schicke ich es zurück?",
   "es": "Quiero devolver {PRODUCT} del pedido {ORDER_ID}. Está sin usar y en su caja. ¿Cómo lo envío?",
   "fr": "Je voudrais retourner {PRODUCT} de la commande {ORDER_ID}. Il n'a pas servi et il est encore dans sa boîte. Comment le renvoyer ?"},
  {"s": "neg", "en": "I sent the return for {ORDER_ID} back on {DATE} and still no refund of {MONEY}. How long does this take?",
   "de": "Ich habe die Rücksendung für {ORDER_ID} am {DATE} abgeschickt und noch keine Erstattung über {MONEY} bekommen. Wie lange dauert das?",
   "es": "Envié la devolución del pedido {ORDER_ID} el {DATE} y todavía no me habéis devuelto {MONEY}. ¿Cuánto tarda?",
   "fr": "J'ai renvoyé la commande {ORDER_ID} le {DATE} et je n'ai toujours pas reçu le remboursement de {MONEY}. Combien de temps faut-il ?"},
  {"s": "neu", "en": "The {KIDBIKE} is the wrong size for my daughter. Can I swap it or get a refund? Order {ORDER_ID}.",
   "de": "{KIDBIKE} ist für meine Tochter die falsche Größe. Kann ich umtauschen oder bekomme ich das Geld zurück? Bestellung {ORDER_ID}.",
   "es": "{KIDBIKE} es de una talla equivocada para mi hija. ¿Puedo cambiarlo o me devolvéis el dinero? Pedido {ORDER_ID}.",
   "fr": "{KIDBIKE} n'est pas à la bonne taille pour ma fille. Puis-je l'échanger ou être remboursé ? Commande {ORDER_ID}."},
  {"s": "neg", "en": "You charged me {MONEY} twice for order {ORDER_ID}. Please refund the duplicate payment."},
  {"s": "neu", "en": "want to cancel order {ORDER_ID} before it ships, changed my mind about the {PRODUCT}. refund to my card pls"},
 ],
 "repair_booking": [
  {"s": "neu", "en": "Can I book a {SERVICE} for {DATE}? It's for my bike ({BIKE}). I can drop it at the workshop in the morning.",
   "de": "Kann ich für den {DATE} einen Termin für {SERVICE} buchen? Es geht um mein Rad ({BIKE}). Ich kann es morgens in die Werkstatt bringen.",
   "es": "¿Puedo reservar {SERVICE} para el {DATE}? Es para mi bici ({BIKE}). Puedo dejarla en el taller por la mañana.",
   "fr": "Puis-je réserver {SERVICE} pour le {DATE} ? C'est pour mon vélo ({BIKE}). Je peux le déposer à l'atelier le matin."},
  {"s": "neu", "en": "My bike ({BIKE}) needs a repair: {wear}. Can I bring it in or send it to you? I live in {LOC}.",
   "de": "Mein Rad ({BIKE}) muss repariert werden: {wear}. Kann ich es vorbeibringen oder einschicken? Ich wohne in {LOC}.",
   "es": "Mi bici ({BIKE}) necesita una reparación: {wear}. ¿Puedo llevarla o enviárosla? Vivo en {LOC}.",
   "fr": "Mon vélo ({BIKE}) a besoin d'une réparation : {wear}. Puis-je l'apporter ou vous l'envoyer ? J'habite à {LOC}."},
  {"s": "neu", "en": "Please can I move my {SERVICE} booking ({ticket}) from {DATE} to the following week? Something came up.",
   "de": "Kann ich meinen Termin für {SERVICE} ({ticket}) vom {DATE} auf die Woche danach verschieben? Mir ist etwas dazwischengekommen.",
   "es": "¿Puedo cambiar mi cita de {SERVICE} ({ticket}) del {DATE} a la semana siguiente? Me ha surgido algo.",
   "fr": "Puis-je déplacer mon rendez-vous {SERVICE} ({ticket}) du {DATE} à la semaine suivante ? J'ai un empêchement."},
  {"s": "neu", "en": "hiya, how much is a {SERVICE} and can u fit me in {DATE}? bike is a {BIKE}, cheers"},
  {"s": "neu", "en": "I'd like to book the {SERVICE} for my {BIKE}. Is {MONEY} still the price? Any day after {DATE} works."},
 ],
 "product_question": [
  {"s": "neu", "en": "Does the {BIKE} come in a smaller frame? I'm 158 cm tall.",
   "de": "Gibt es {BIKE} auch mit einem kleineren Rahmen? Ich bin 1,58 m groß.",
   "es": "¿{BIKE} viene en una talla de cuadro más pequeña? Mido 1,58 m.",
   "fr": "{BIKE} existe-t-il avec un cadre plus petit ? Je mesure 1,58 m."},
  {"s": "neu", "en": "What range does the {EBIKE} get on one charge, and is the {ACCESSORY} compatible with it?",
   "de": "Welche Reichweite hat {EBIKE} mit einer Akkuladung, und passt {ACCESSORY} dazu?",
   "es": "¿Qué autonomía tiene {EBIKE} con una carga, y es compatible con {ACCESSORY}?",
   "fr": "Quelle autonomie a {EBIKE} avec une charge, et {ACCESSORY} est-il compatible ?"},
  {"s": "neu", "en": "Is the {PRODUCT} in stock? The website says {MONEY} but I can't add it to the basket.",
   "de": "Ist {PRODUCT} auf Lager? Auf der Website steht {MONEY}, aber ich kann es nicht in den Warenkorb legen.",
   "es": "¿Tenéis {PRODUCT} en stock? En la web pone {MONEY} pero no puedo añadirlo a la cesta.",
   "fr": "Est-ce que {PRODUCT} est en stock ? Le site indique {MONEY} mais je n'arrive pas à l'ajouter au panier."},
  {"s": "neu", "en": "Do you deliver to {LOC}? And do bikes arrive fully built or do I need to assemble them?"},
  {"s": "neu", "en": "quick q: whats the difference between the {SERVICE} and the {SERVICE2}?"},
 ],
 "warranty_fault": [
  {"s": "neg", "en": "My e-bike ({EBIKE}) is only four months old and {fault}. Is this covered by the warranty?",
   "de": "Mein E-Bike ({EBIKE}) ist erst vier Monate alt und {fault}. Ist das von der Garantie abgedeckt?",
   "es": "Mi bici eléctrica ({EBIKE}) solo tiene cuatro meses y {fault}. ¿Lo cubre la garantía?",
   "fr": "Mon vélo électrique ({EBIKE}) n'a que quatre mois et {fault}. Est-ce couvert par la garantie ?"},
  {"s": "neg", "from": "2026-06-09", "w": 4, "en": "I got your email about battery batch {batch}. My pack has that code and {fault}. What happens now?",
   "de": "Ich habe Ihre E-Mail zur Akku-Charge {batch} bekommen. Mein Akku hat diesen Code und {fault}. Wie geht es jetzt weiter?",
   "es": "Recibí vuestro correo sobre el lote de baterías {batch}. Mi batería tiene ese código y {fault}. ¿Qué hago ahora?",
   "fr": "J'ai reçu votre e-mail sur le lot de batteries {batch}. Ma batterie porte ce code et {fault}. Que se passe-t-il maintenant ?"},
  {"s": "neg", "en": "The {PRODUCT} arrived with {damage}. I've attached photos. Can you send a replacement?",
   "de": "{PRODUCT} kam mit {damage} an. Fotos sind im Anhang. Können Sie Ersatz schicken?",
   "es": "{PRODUCT} llegó con {damage}. Adjunto fotos. ¿Podéis enviarme uno nuevo?",
   "fr": "{PRODUCT} est arrivé avec {damage}. Je joins des photos. Pouvez-vous m'en envoyer un autre ?"},
  {"s": "neg", "en": "Spoke snapped on my {BIKE} after two rides. That's not normal wear, surely? Bought {DATE}, order {ORDER_ID}."},
  {"s": "neg", "en": "{EBIKE} battery dead again. Second pack in a month. I want this sorted properly this time."},
 ],
 "praise": [
  {"s": "pos", "en": "Just wanted to say thank you to {PERSON} in the workshop. My bike ({BIKE}) rides like new after the {SERVICE}.",
   "de": "Ich wollte mich nur bei {PERSON} in der Werkstatt bedanken. Mein Rad ({BIKE}) fährt sich nach {SERVICE} wie neu.",
   "es": "Solo quería dar las gracias a {PERSON} del taller. Mi bici ({BIKE}) va como nueva después de {SERVICE}.",
   "fr": "Je voulais juste remercier {PERSON} de l'atelier. Mon vélo ({BIKE}) roule comme neuf après {SERVICE}."},
  {"s": "pos", "en": "Love my new bike ({BIKE}). Fast delivery to {LOC} and it was easy to set up. Five stars.",
   "de": "Ich liebe mein neues Rad ({BIKE}). Schnelle Lieferung nach {LOC} und leicht aufzubauen. Fünf Sterne.",
   "es": "Me encanta mi nueva bici ({BIKE}). Entrega rápida a {LOC} y fácil de montar. Cinco estrellas.",
   "fr": "J'adore mon nouveau vélo ({BIKE}). Livraison rapide à {LOC} et facile à monter. Cinq étoiles."},
  {"s": "pos", "en": "Great service from {PERSON}, who sorted my problem in one email. Thank you!",
   "de": "Super Service von {PERSON}, das Problem war mit einer E-Mail gelöst. Vielen Dank!",
   "es": "Un servicio genial de {PERSON}, que me lo resolvió en un solo correo. ¡Gracias!",
   "fr": "Super service de la part de {PERSON}, qui a réglé mon problème en un seul e-mail. Merci !"},
  {"s": "pos", "en": "the {ACCESSORY} is brilliant, fits perfectly. cheers"},
  {"s": "pos", "en": "Honestly the best bike shop in {LOC}. Friendly, quick and fair prices."},
 ],
 "spam": [
  {"s": "neu", "en": "Dear partner, we offer guaranteed first-page search ranking for your online store. Reply today for a {MONEY} discount."},
  {"s": "neu", "en": "Exclusive partnership opportunity: invest in our crypto bike-sharing token and earn 300% returns. Contact {email} now."},
  {"s": "neu", "en": "Your mailbox storage is full. Verify your account within 24 hours at http://mail-verify.example/login to avoid suspension."},
  {"s": "neu", "en": "Hello, I am an influencer with 50k followers. Send me a free {BIKE} and I will post about your brand."},
  {"s": "neu", "en": "Congratulations! You have been selected for a {MONEY} gift voucher. Click http://prize-claim.example to claim."},
 ],
}
PHRASES = {   # paired fillers for the {lower} slots: (en, de, es, fr)
 "fault": [("the battery will not charge", "der Akku lädt nicht mehr", "la batería no carga", "la batterie ne se charge plus"),
           ("the motor cuts out on hills", "der Motor setzt am Berg aus", "el motor se para en las cuestas", "le moteur se coupe dans les montées"),
           ("the display shows error E07", "das Display zeigt Fehler E07", "la pantalla muestra el error E07", "l'écran affiche l'erreur E07"),
           ("the battery runs flat very quickly", "der Akku ist sehr schnell leer", "la batería se agota muy rápido", "la batterie se vide très vite")],
 "wear": [("the back tyre has a puncture", "der Hinterreifen hat einen Platten", "la rueda trasera está pinchada", "le pneu arrière est crevé"),
          ("the gears keep slipping", "die Gangschaltung springt ständig", "los cambios saltan todo el rato", "les vitesses sautent sans arrêt"),
          ("the brake pads are worn out", "die Bremsbeläge sind abgenutzt", "las pastillas de freno están gastadas", "les plaquettes de frein sont usées"),
          ("the chain squeaks", "die Kette quietscht", "la cadena chirría", "la chaîne grince")],
 "damage": [("a cracked mudguard", "einem gebrochenen Schutzblech", "un guardabarros roto", "un garde-boue fendu"),
            ("a bent front wheel", "einem verbogenen Vorderrad", "la rueda delantera doblada", "une roue avant voilée"),
            ("a broken clip", "einem kaputten Clip", "un clip roto", "un clip cassé")],
}
SUBJECTS = {"order_status": ["Where is my order?", "Wo bleibt meine Bestellung?", "¿Dónde está mi pedido?", "Où est ma commande ?"],
            "returns_refund": ["Return and refund", "Rücksendung", "Devolución", "Retour"],
            "repair_booking": ["Booking a service", "Werkstatttermin", "Cita en el taller", "Rendez-vous atelier"],
            "product_question": ["Product question", "Frage zum Produkt", "Pregunta sobre un producto", "Question sur un produit"],
            "warranty_fault": ["Fault with my bike", "Defekt", "Avería", "Panne"],
            "praise": ["Thank you!", "Danke!", "¡Gracias!", "Merci !"],
            "spam": ["Partnership opportunity"] * 4}
FORM_TOPIC = {"order_status": "Order enquiry", "returns_refund": "Returns", "repair_booking": "Workshop",
              "product_question": "Product question", "warranty_fault": "Faulty item", "praise": "Feedback", "spam": "Other"}
CHANNELS = {"order_status": (5, 4, 1), "returns_refund": (6, 4, 0), "repair_booking": (4, 6, 0),   # email, web form, review
            "product_question": (4, 6, 0), "warranty_fault": (5, 3, 2), "praise": (3, 1, 6), "spam": (1, 0, 0)}
LANGS = ["en", "de", "es", "fr"]
MONTHS = {"en": "January February March April May June July August September October November December",
          "de": "Januar Februar März April Mai Juni Juli August September Oktober November Dezember",
          "es": "enero febrero marzo abril mayo junio julio agosto septiembre octubre noviembre diciembre",
          "fr": "janvier février mars avril mai juin juillet août septembre octobre novembre décembre"}
TOWNS = {"en": ["Bristol", "Bath", "Clifton", "Cardiff", "Stroud"], "de": ["Berlin", "Hamburg", "München", "Köln"],
         "es": ["Madrid", "Barcelona", "Valencia", "Sevilla"], "fr": ["Paris", "Lyon", "Nantes", "Lille"]}
SPAMMERS = [("Growth Partners", "offers@growth-partners.example"), ("Crypto Ride", "team@cryptoride.example"),
            ("Mail Admin", "admin@mail-verify.example"), ("Prize Desk", "win@prize-claim.example")]


def fmt_date(d, lang, rng):
    m = MONTHS[lang].split()[d.month - 1]
    styles = {"en": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}", f"{m} {d.day}{'th' if 10 < d.day < 14 else {1: 'st', 2: 'nd', 3: 'rd'}.get(d.day % 10, 'th')}"],
              "de": [f"{d.day}. {m}", f"{d.day:02d}.{d.month:02d}."], "es": [f"{d.day} de {m}", f"{d.day}/{d.month}"],
              "fr": [f"{d.day} {m}", f"{d.day:02d}/{d.month:02d}/{d.year}"]}[lang]
    return rng.choice(styles) if rng else styles[0]


def fmt_money(gbp, lang, rng):
    styles = {"en": [f"£{gbp:,}", f"£{gbp:,}.00", f"{gbp} pounds"], "de": [f"{gbp:,} £".replace(",", "."), f"{gbp} Pfund"],
              "es": [f"{gbp:,} libras".replace(",", "."), f"{gbp} £"], "fr": [f"{gbp:,} £".replace(",", " "), f"{gbp} livres"]}[lang]
    return rng.choice(styles) if rng else styles[0]


def typo(word, rng):
    i = rng.randrange(1, len(word) - 1)
    return rng.choice([word[:i] + word[i + 1] + word[i] + word[i + 2:], word[:i] + word[i + 1:], word[:i] + word[i] + word[i:]])


print(sum(len(v) for v in T.values()), "wordings,", sum("de" in t for v in T.values() for t in v), "of them in all four languages")

Code walkthrough

  • Block 2 of course-guide/06: every message wording. This lesson reuses its MONTHS and LANGS.

Output

text
35 wordings, 18 of them in all four languages
python
import html, unicodedata

POOLS = {"PRODUCT": ["bike", "ebike", "accessory"], "BIKE": ["bike", "ebike"], "EBIKE": ["ebike"],
         "ACCESSORY": ["accessory"], "SERVICE": ["service"], "SERVICE2": ["service"], "KIDBIKE": ["kids"]}
EXTRA = {"greet": {"en": ["Hi,", "Hello,", "Hello Tidewell,", "Dear Sir or Madam,"], "de": ["Hallo,", "Guten Tag,"],
                   "es": ["Hola,", "Buenos días,"], "fr": ["Bonjour,", "Madame, Monsieur,"]},
         "sign": {"en": ["Thanks,", "Many thanks,", "Cheers,"], "de": ["Danke,", "Viele Grüße,"],
                  "es": ["Gracias,", "Un saludo,"], "fr": ["Merci,", "Cordialement,"]},
         "phone": {"en": "You can call me on ", "de": "Sie erreichen mich unter ", "es": "Me podéis llamar al ",
                   "fr": "Vous pouvez m'appeler au "},
         "quote": {"en": ("On ", ", Tidewell Cycles <help@tidewell.example> wrote:"), "de": ("Am ", " schrieb Tidewell Cycles <help@tidewell.example>:"),
                   "es": ("El ", ", Tidewell Cycles <help@tidewell.example> escribió:"), "fr": ("Le ", ", Tidewell Cycles <help@tidewell.example> a écrit :")},
         "again": {"en": "Sorry, sending this again as I have not heard back. ", "de": "Entschuldigung, ich schicke das noch einmal. ",
                   "es": "Perdón, lo envío otra vez porque no he recibido respuesta. ", "fr": "Désolé, je renvoie ce message car je n'ai pas eu de réponse. "}}
LABEL_OF = {"PRODUCT": "PRODUCT", "BIKE": "PRODUCT", "EBIKE": "PRODUCT", "ACCESSORY": "PRODUCT", "SERVICE": "PRODUCT",
            "SERVICE2": "PRODUCT", "KIDBIKE": "PRODUCT", "ORDER_ID": "ORDER_ID", "DATE": "DATE", "MONEY": "MONEY",
            "LOC": "LOC", "PERSON": "PERSON"}


def fill(template, values, lang, rng):
    """Template text -> list of (text, label or None) pieces, formatting dates and money for `lang`."""
    pieces = []
    for part in re.split(r"(\{\w+\})", template):
        if not (part.startswith("{") and part.endswith("}")):
            pieces.append((part, None)) if part else None
            continue
        slot = part[1:-1]
        v = values[slot]
        if slot == "DATE":
            v = fmt_date(v, lang, rng)
        elif slot == "MONEY":
            v = fmt_money(v, lang, rng)
        elif slot in ("fault", "wear", "damage"):
            v = PHRASES[slot][v][LANGS.index(lang)]
        pieces.append((v, LABEL_OF.get(slot)))
    return pieces


def email_of(first, last):
    """'Mehmet', 'Yılmaz' -> 'mehmet.yilmaz@...'; one fixed address per customer, on a reserved .example domain."""
    plain = unicodedata.normalize("NFKD", f"{first}.{last}".lower().replace("ı", "i").replace(" ", "").replace("'", ""))
    plain = "".join(ch for ch in plain if ch.isascii())
    return plain + "@" + ["post.example", "mailbox.example", "inbox.example"][len(first + last) % 3]


def make_messages(n=1800, seed=SEED):
    rng = random.Random(seed)
    days = [WINDOW[0] + timedelta(i) for i in range((WINDOW[1] - WINDOW[0]).days + 1)]
    base = {"order_status": 26, "returns_refund": 16, "repair_booking": 17, "product_question": 18,
            "warranty_fault": 11, "praise": 6, "spam": 6}

    def weights(d):
        w = dict(base)
        w["spam"] *= 8 if EVENTS["spam_wave"][0] <= d <= EVENTS["spam_wave"][1] else 1
        w["warranty_fault"] *= 3.5 if EVENTS["battery_notice"] <= d <= date(2026, 7, 31) else 2 if d.month == 8 else 1
        w["order_status"] *= (2.5 if EVENTS["summer_sale"][0] <= d <= EVENTS["summer_sale"][1] else 1) * (1.6 if d >= EVENTS["carrier_change"] else 1)
        w["returns_refund"] *= 1.3 if date(2026, 7, 29) <= d <= date(2026, 8, 15) else 1
        vol = (0.6 if d.weekday() >= 5 else 1.0) * (1.5 if d >= EVENTS["eu_shipping"] else 1.0)
        return [w[i] * vol for i in INTENTS]

    W = [weights(d) for d in days]
    n_dup = round(n * 0.03)
    picks = sorted(rng.choices(range(len(days)), weights=[sum(w) for w in W], k=n - n_dup))
    by_kind = {k: [p for p in PRODUCTS if p[3] == k] for k in ["bike", "ebike", "accessory", "service"]}
    by_kind["kids"] = [p for p in PRODUCTS if p[1] == "Harbour Kids 20"]
    out = []
    for di in picks:
        d = days[di]
        intent = rng.choices(INTENTS, weights=W[di])[0]
        lang = "en" if intent == "spam" else rng.choices(LANGS, weights=[95, 2, 1.5, 1.5] if d < EVENTS["eu_shipping"] else [58, 16, 14, 12])[0]
        options = [t for t in T[intent] if lang in t and date.fromisoformat(t.get("from", "2026-01-01")) <= d]
        tmpl = rng.choices(options, weights=[t.get("w", 1) for t in options])[0]
        if intent == "spam":
            cust, email = rng.choice(SPAMMERS)
            first, last, gender, town = cust, "", None, None
        else:
            first, last, gender, _, town = rng.choice([c for c in CUSTOMERS if c[3] == lang])
            email = email_of(first, last)
        v = {}
        for slot, kinds in POOLS.items():
            pool = [p for k in kinds for p in by_kind[k]]
            pick = rng.choice([p for p in pool if p[1] != v.get("SERVICE")])
            v[slot] = pick[1]
            v[slot + "_sku"], v[slot + "_price"] = pick[0], pick[2]
        main = next((s for s in POOLS if "{" + s + "}" in tmpl["en"]), None)
        future = intent == "repair_booking" or "before {DATE}" in tmpl["en"]
        v["DATE"] = d + timedelta(rng.randint(2, 14)) if future else d - timedelta(rng.randint(3, 20))
        v["MONEY"] = v[main + "_price"] if main else rng.choice([50, 100, 250, 500])
        v["ORDER_ID"] = f"TW-{100000 + di * 120 + rng.randrange(120)}"
        lives_there = "live in {LOC}" in tmpl["en"] or "shop in {LOC}" in tmpl["en"]
        v["LOC"] = town if lives_there and town else rng.choice(TOWNS[lang])
        v["PERSON"] = rng.choice(["Dev", "Dev", "Ines", "Callum"]) if intent == "praise" else first
        v.update(carrier="SwiftDrop" if d >= EVENTS["carrier_change"] else "ParcelNorth", email=email, batch=BATCH,
                 ticket=f"RJ-2026-{300 + di * 2 + rng.randrange(3):04d}", fault=rng.randrange(4), wear=rng.randrange(4),
                 damage=rng.randrange(3))
        core = fill(tmpl[lang], v, lang, rng)
        en_ref = "".join(t for t, _ in fill(tmpl["en"], v, "en", None)) if lang != "en" else None
        channel = rng.choices(["email", "web form", "review"], weights=CHANNELS[intent])[0]
        channel = "web form" if channel == "review" and tmpl["s"] == "neu" else channel   # reviews praise or complain
        pieces, g = list(core), 0
        if channel == "email":
            if rng.random() < 0.7:
                pieces, g = [(rng.choice(EXTRA["greet"][lang]) + "\n\n", None)] + pieces, 1
            if rng.random() < 0.15:
                pieces += [("\n" + EXTRA["phone"][lang] + rng.choice(["07700 900", "07700900", "+44 7700 900"]) + f"{rng.randrange(1000):03d}.", None)]
            if lang == "en" and intent in ("repair_booking", "order_status") and rng.random() < 0.3:
                pieces += [(f"\nMy postcode is BS{rng.randint(1, 16)} {rng.randint(1, 9)}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}{rng.choice('ABDEFGHJLNPQRSTUWXYZ')}.", None)]
            if rng.random() < 0.7 and intent != "spam":
                pieces += [("\n\n" + rng.choice(EXTRA["sign"][lang]) + "\n", None), (first if rng.random() < 0.6 else f"{first} {last}", "PERSON")]
                if rng.random() < 0.15:
                    pieces += [("\n" + {"en": "Sent from my phone", "de": "Von meinem Handy gesendet", "es": "Enviado desde mi móvil", "fr": "Envoyé de mon téléphone"}[lang], None)]
            if rng.random() < 0.25 and intent != "spam":
                q = EXTRA["quote"][lang]
                prev = fmt_date(d - timedelta(rng.randint(1, 6)), lang, None) + (" de 2026" if lang == "es" else " 2026")
                pieces += [("\n\n> " + q[0], None), (prev, "DATE"), (q[1] + "\n> Thanks for your message. We will reply within one working day.", None)]
        r = rng.random
        if r() < 0.15:                                          # typos
            for _ in range(rng.randint(1, 2)):
                i = rng.randrange(len(pieces))
                if pieces[i][1] is None:
                    words = pieces[i][0].split(" ")
                    j = rng.randrange(len(words))
                    if len(words[j]) >= 5 and words[j].isalpha():
                        words[j] = typo(words[j], rng)
                        pieces[i] = (" ".join(words), None)
        case = r()
        if case < 0.03:
            pieces = [(t.upper(), l) for t, l in pieces]
        elif case < 0.09:
            pieces = [(t.lower(), l) for t, l in pieces]
        if r() < 0.08:                                          # emoji after the main text
            pieces.insert(g + len(core), (" " + rng.choice(["😡", "😤", "🚲"] if tmpl["s"] == "neg" else ["🙂", "👍", "🚲"]), None))
        if r() < 0.05:                                          # odd whitespace
            i = next((i for i, (t, l) in enumerate(pieces) if l is None and " " in t), None)
            if i is not None:
                t = pieces[i][0]
                pieces[i] = (t.replace(" ", " ", 1) if r() < 0.5 else t.replace(" ", " ​", 1), None)
        if channel == "web form" and r() < 0.6:                 # HTML from the form
            pieces = [("<p>", None)] + [(html.escape(t).replace("\n", "<br>\n"), None) if l is None else (t, l) for t, l in pieces] + [("</p>", None)]
        if r() < 0.04:                                          # mojibake: UTF-8 bytes read as cp1252
            try:
                pieces = [(t.encode("utf-8").decode("cp1252"), l) for t, l in pieces]
            except UnicodeDecodeError:
                pass
        body, spans, pos = "", [], 0
        for t, l in pieces:
            if l:
                spans.append((pos, pos + len(t), l))
            body, pos = body + t, pos + len(t)
        sent = tmpl["s"]
        stars = None
        if channel == "review":
            stars = {"pos": rng.choice([4, 5]), "neu": 3, "neg": rng.choice([1, 2])}[sent]
            if r() < 0.08:
                stars = {"pos": rng.choice([1, 2]), "neu": rng.choice([1, 5]), "neg": 5}[sent]
        subject = {"email": SUBJECTS[intent][LANGS.index(lang)], "web form": "Web form: " + FORM_TOPIC[intent],
                   "review": "Review: " + (v[main] if main else "Tidewell Cycles")}[channel]
        out.append({"date": d.isoformat(), "channel": channel, "language": lang, "customer": f"{first} {last}".strip(),
                    "gender": gender, "email": email, "subject": subject, "body": body, "intent": intent, "sentiment": sent,
                    "stars": stars, "entities": spans, "sku": v[main + "_sku"] if main else None,
                    "order_id": v["ORDER_ID"] if "{ORDER_ID}" in tmpl["en"] else None, "duplicate_of": None,
                    "core": "".join(t for t, _ in core), "en_ref": en_ref, "label_flipped": False})
    originals = [m for m in out if m["intent"] != "spam"]
    for m in rng.sample(originals, n_dup):                      # near-duplicates: the same customer writes again
        again, cut = EXTRA["again"][m["language"]], 3 if m["body"].startswith("<p>") else 0   # inside the form's HTML
        dup = dict(m, body=m["body"][:cut] + again + m["body"][cut:], entities=[(a + len(again), b + len(again), l) for a, b, l in m["entities"]],
                   date=min(date.fromisoformat(m["date"]) + timedelta(rng.randint(0, 2)), WINDOW[1]).isoformat(), duplicate_of=m)
        out.append(dup)
    out.sort(key=lambda m: (m["date"], m["duplicate_of"] is not None))
    for i, m in enumerate(out, 1):
        m["id"] = f"TW-M{i:04d}"
    for m in out:
        if m["duplicate_of"] is not None:
            m["duplicate_of"] = m["duplicate_of"]["id"]
    for m in rng.sample(out, round(n * 0.02)):                  # deliberate wrong labels, for error analysis
        m["intent"], m["label_flipped"] = rng.choice([i for i in INTENTS if i != m["intent"]]), True
    split_of = {}
    for intent in INTENTS:                                      # stratified 70 / 15 / 15, duplicates follow their original
        ids = [m["id"] for m in out if m["intent"] == intent and m["duplicate_of"] is None]
        rng.shuffle(ids)
        a, b = round(len(ids) * 0.70), round(len(ids) * 0.85)
        split_of.update({x: "train" if k < a else "dev" if k < b else "test" for k, x in enumerate(ids)})
    for m in out:
        m["split"] = split_of[m["duplicate_of"] or m["id"]]
        m["period"] = "before_july" if m["date"] < "2026-07-01" else "from_july"
    return out


MESSAGES = make_messages()
print(len(MESSAGES), "messages;", MESSAGES[0]["date"], "to", MESSAGES[-1]["date"])
print(MESSAGES[0]["id"], "|", MESSAGES[0]["subject"], "|", MESSAGES[0]["body"][:80].replace("\n", " "))

Code walkthrough

  • Block 3 of course-guide/06: the generator. Each message carries gold entities, the exact character spans of its order numbers, dates and prices.

Output

text
1800 messages; 2026-03-02 to 2026-08-30
TW-M0001 | Web form: Workshop | <p>Mon vélo (Causeway) a besoin d&#x27;une réparation : les plaquettes de frei
python
import hashlib, json

def fingerprint(*datasets):
    """The first 12 hex digits of a SHA-256 hash of the data: equal fingerprints mean identical data."""
    blob = json.dumps(datasets, ensure_ascii=False, sort_keys=True, default=str).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()[:12]


print("messages ", fingerprint(MESSAGES))

Code walkthrough

  • The fingerprint function from block 12 of course-guide/06. It must print 294611361162.

Output

text
messages  294611361162
PATTERN TW-\D{6} UNDER KOFI'S ORDER NUMBER T W - \d{6} T W - 1 0 6 4 8 9 literals: match themselves class \d with quantifier {6}
TW-\d{6} is three literals (blue) and one character class repeated six times (amber); every character of TW-106489 is covered by exactly one piece.

A literal is a character that matches itself: TW- finds those three characters in that order. A character class matches any one character from a set: \d any digit, [A-Z] any capital letter. The picture lines TW-\d{6} up under Kofi's order number.

\d any digit 0 7 7 0 0 [A-Z] one capital letter B S F R \s space, tab, line break the gap in 07700 900070 \w letter, digit or _ k o f i 1 2 [^>] anything except > every letter inside a tag . any character but \n all of one line
A character class matches any one character from a set; these six cover most patterns in this course.

A quantifier says how many times the piece before it may repeat: {6} exactly six, + one or more, * zero or more, ? zero or one, {2,3} two or three. Without quantifiers you would write \d\d\d\d\d\d.

python
import re

kofi = next(m for m in MESSAGES if m["id"] == "TW-M0435")
text = kofi["body"]
print(text)
print("---")
print(re.search("order", text))               # a literal: these five letters, in this order
print(re.search(r"TW-\d", text))              # \d: any one digit
print(re.search(r"TW-\d{6}", text))           # {6}: exactly six of the piece before
print(re.search(r"BS\d+ \d[A-Z]{2}", text))   # + one or more; [A-Z] one capital letter

Code walkthrough

  • re.search(pattern, text) finds the first place the pattern fits. It returns a match object, showing the position (span) and the matched text, or None.
  • The r before a pattern makes a raw string, so Python passes \d to re unchanged. Write every pattern this way.

Output

text
Hello,

Could you send me the tracking link for order TW-106489? The confirmation email never came through to kofi.asante@mailbox.example.
You can call me on 07700 900070.
My postcode is BS12 2FR.

Cheers,
Kofi

> On 24 April 2026, Tidewell Cycles <help@tidewell.example> wrote:
> Thanks for your message. We will reply within one working day.
---
<re.Match object; span=(48, 53), match='order'>
<re.Match object; span=(54, 58), match='TW-1'>
<re.Match object; span=(54, 63), match='TW-106489'>
<re.Match object; span=(187, 195), match='BS12 2FR'>
WHAT EACH PATTERN TAKES FROM 07700 900070 \d? 0 7 7 0 0 ␣ 9 0 0 0 7 0 \d{2,3} 0 7 7 0 0 ␣ 9 0 0 0 7 0 \d+ 0 7 7 0 0 ␣ 9 0 0 0 7 0 \d{5}\s\d{6} 0 7 7 0 0 ␣ 9 0 0 0 7 0 .+ 0 7 7 0 0 ␣ 9 0 0 0 7 0
Quantifiers set how many: ? takes one digit, {2,3} three, + all five before the space; each starts at the first character it can.
python
phone = "07700 900070"
for pattern in [r"\d", r"\d?", r"\d*", r"\d+", r"\d{2,3}", r"\d{5}\s\d{6}", r"\w+", r".+"]:
    print(f"{pattern:14} -> {re.search(pattern, phone).group()!r}")

Code walkthrough

  • .group() returns the matched text. \s is any whitespace, \w any letter, digit or underscore, . any character except a line break.
  • Each quantified pattern starts at the first character and takes as much as it may: \d+ stops at the space.

Output

text
\d             -> '0'
\d?            -> '0'
\d*            -> '07700'
\d+            -> '07700'
\d{2,3}        -> '077'
\d{5}\s\d{6}   -> '07700 900070'
\w+            -> '07700'
.+             -> '07700 900070'

2. Anchors, word boundaries and flags (Essential)

A pattern that fits too much is as bad as one that fits too little. Boundaries and anchors pin a match to a place, and flags change how the whole pattern behaves.

TW-\d{6} \bTW-\d{6}\b T W - 1 0 6 4 8 9 0 T W - 1 0 6 4 8 9 0 matches TW-106489: wrong no match: no gap after 6 digits
Red is a wrong match, green a correct refusal: \b needs a gap after the sixth digit, so a seven-digit typo is not read as an order number.

A word boundary, \b, matches no character: it matches the gap between a word character and a non-word character. Customers mistype: TW-1064890 has seven digits. TW-\d{6} matches its first six; \bTW-\d{6}\b refuses, because no gap follows the sixth digit.

^ without a flag ^ with re.MULTILINE Hello, ^ Could you send me the tracking... You can call me on... My postcode is BS12 2FR. Cheers, Hello, ^ Could you send me the tracking... ^ You can call me on... ^ My postcode is BS12 2FR. ^ Cheers, ^ only the first line can start a match every line can; the postcode line fits
^ marks the start of the whole text unless re.MULTILINE is set; then it marks every line, and the postcode line (green) matches.

An anchor pins a match to a position: ^ the start and $ the end, of the whole text by default. Kofi's postcode line sits mid-message, so ^My postcode is fails until the re.MULTILINE flag makes ^ and $ work at every line.

python
samples = ["order TW-106489?", "order TW-1064890", "code XTW-106489", "pack TW-BATT-500"]
for s in samples:
    plain = re.search(r"TW-\d{6}", s)
    bounded = re.search(r"\bTW-\d{6}\b", s)
    print(f"{s:18} plain: {plain.group() if plain else None!s:10} bounded: {bounded.group() if bounded else None}")

print(re.search(r"^My postcode is (.+)$", text))                 # ^ and $: start and end of the whole text
print(re.search(r"^My postcode is (.+)$", text, re.MULTILINE))   # with MULTILINE: of each line

Code walkthrough

  • The samples: a correct order number, one digit too many, one glued to a letter, and a product code (SKU).
  • The third argument of re.search is a flag.

Output

text
order TW-106489?   plain: TW-106489  bounded: TW-106489
order TW-1064890   plain: TW-106489  bounded: None
code XTW-106489    plain: TW-106489  bounded: None
pack TW-BATT-500   plain: None       bounded: None
None
<re.Match object; span=(172, 196), match='My postcode is BS12 2FR.'>
751 GOLD ORDER NUMBERS 709 written TW- 42 written tw- case-sensitive: 709 re.IGNORECASE: 751
42 of the 751 order numbers arrive in lower case (red); the IGNORECASE flag is the difference between 709 found and all 751.

The flag that matters most here is re.IGNORECASE. The generator writes 6% of messages in lower case, as people do on phones, so some order numbers arrive as tw-100474.

python
ORDER_CS = re.compile(r"\bTW-\d{6}\b")
ORDER = re.compile(r"\bTW-\d{6}\b", re.IGNORECASE)
gold = [m["body"][a:b] for m in MESSAGES for a, b, label in m["entities"] if label == "ORDER_ID"]
print(len(gold), "gold order numbers")
print("case-sensitive finds", sum(bool(ORDER_CS.search(g)) for g in gold))
print("IGNORECASE finds    ", sum(bool(ORDER.search(g)) for g in gold))
print([g for g in gold if not ORDER_CS.search(g)][:3])

POSTCODE = re.compile(r"""
    \b [A-Z]{1,2} \d{1,2} [A-Z]?   # outward part: area letters, district number, maybe a letter
    \s? \d [A-Z]{2} \b             # inward part: one digit and two letters
""", re.VERBOSE | re.IGNORECASE)
print(POSTCODE.search(text).group())

Code walkthrough

  • re.compile(pattern, flags) builds a reusable pattern object once; ORDER.search(...) then needs no pattern text.
  • gold holds the text of every ORDER_ID span: body[a:b] is exactly the order number.
  • re.VERBOSE allows comments and ignores spaces in the pattern, so the real space is written \s?. Flags combine with |.

Output

text
751 gold order numbers
case-sensitive finds 709
IGNORECASE finds     751
['tw-100474', 'tw-100474', 'tw-101077']
BS12 2FR

3. Groups, alternation and named groups (Essential)

Dev's booking form wants the postcode area on its own and every phone in one style. Groups let a pattern match and also hand back its parts.

07700 900070 07700900158 +44 7700 900762 (?:\+44\s?|0) 7700\s?900\d{3} 3 of 3 | means: +44 and maybe a space, or 0
Alternation (|) lets one pattern accept every phone style in the inbox: +44 7700, 07700 with a space, and 07700 without one.

Alternation, |, means "this or that". Inbox phones start 07700 900, 07700900 or +44 7700 900, and (?:\+44\s?|0)7700\s?900\d{3} covers all three. (?:...) groups the choices without keeping them; \+ is an escaped plus, since a bare + is a quantifier.

B S 1 2 ␣ 2 F R area group("area") = BS12 unit group("unit") = 2FR (?P<area>[A-Z]{1,2}\d{1,2}[A-Z]?)\s?(?P<unit>\d[A-Z]{2})
Named groups hand back the parts of one match by name: the postcode BS12 2FR gives area BS12 and unit 2FR.

A capturing group, plain brackets, keeps what its part matched. A named group, (?P<area>...), keeps it under a name, easier to read later than group(1).

python
PHONE = re.compile(r"(?:\+44\s?|0)7700\s?900\d{3}\b")
for s in ["call me on 07700 900070.", "call me on 07700900158.", "call me on +44 7700 900762."]:
    print(PHONE.search(s).group())

PC = re.compile(r"\b(?P<area>[A-Z]{1,2}\d{1,2}[A-Z]?)\s?(?P<unit>\d[A-Z]{2})\b", re.IGNORECASE)
m = PC.search(text)
print(m.group(0), "|", m.group(1), "|", m.group("unit"))
print(m.groupdict())
print(re.search(r"(order|parcel) (TW-\d{6})", text).groups())

Code walkthrough

  • The three strings are the inbox's phone styles, from Kofi's, Fatima Hussain's and Raj Gupta's messages.
  • group(0) is the whole match, group(1) the first group, group("unit") a named one; groupdict() returns all named groups.
  • groups() returns every capturing group as a tuple.

Output

text
07700 900070
07700900158
+44 7700 900762
BS12 2FR | BS12 | 2FR
{'area': 'BS12', 'unit': '2FR'}
('order', 'TW-106489')

4. Python's re module: search, findall, finditer and sub (Essential)

Python runs a pattern in several ways; picking the wrong one causes quiet bugs.

EMAIL PATTERN ON KOFI'S MESSAGE search first match anywhere Match (110, 137) match only at the start None findall every match, as text 2 strings finditer every match, with spans 2 Match objects sub replace every match new text
The same pattern, five functions: search and match return one match or None, findall and finditer return all of them, sub rewrites the text.

search returns the first match anywhere; match checks only the start. findall returns every match as a string; finditer returns every match object, with positions. sub replaces every match, with a string or with what a function returns.

python
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
print(EMAIL.search(text), EMAIL.match(text))   # first match anywhere; only at the start
print(EMAIL.findall(text))                      # every match, as strings
print(re.findall(r"@([\w-]+)", text))           # with a group, findall returns the group only
for m in ORDER.finditer(text):                  # every match, with its span
    print("found", m.span(), "gold", [(a, b) for a, b, l in kofi["entities"] if l == "ORDER_ID"])

def one_style(m):
    return "07700 900" + m.group()[-3:]         # the last three digits, in one fixed style

calls = "07700900158, +44 7700 900762 and 07700 900070"
print(PHONE.sub(one_style, calls))
print(PHONE.subn("[PHONE]", calls))

Code walkthrough

  • [\w.+-]+ is the part before the @; inside a class most symbols lose their special meaning.
  • findall returns two addresses: Kofi's and Tidewell's own, in the quoted reply. Given a capturing group it returns only the group: a common trap.
  • m.span() is (start, end), and it equals the gold span the generator recorded.
  • sub calls one_style for each match and inserts its return value; subn also counts replacements. Masking personal data properly is Lesson 4's job.

Output

text
<re.Match object; span=(110, 137), match='kofi.asante@mailbox.example'> None
['kofi.asante@mailbox.example', 'help@tidewell.example']
['mailbox', 'tidewell']
found (54, 63) gold [(54, 63)]
07700 900158, 07700 900762 and 07700 900070
('[PHONE], [PHONE] and [PHONE]', 3)
CHARACTERS 48 TO 63 OF KOFI'S MESSAGE o r d e r ␣ T W - 1 0 6 4 8 9 54 63 found gold ORDER_ID
finditer gives the span (54, 63); amber is what the pattern found, green the gold label, and they sit on exactly the same characters.

That span check is the basis of all measurement in section 8. Not every message has an order number, though. Run ORDER.search("refund??").group() and you will see:

text
AttributeError: 'NoneType' object has no attribute 'group'

search returned None. Test the result (if m:) before reading it.

5. Greedy versus lazy matching (Essential)

Quantifiers take as much as they can. That is right for digits and wrong for anything with a closing mark, such as an HTML tag, and 459 inbox messages arrived from the web form wrapped in HTML.

<.*> greedy < p > a n y ␣ u p d a t e ␣ o n ␣ T W - 1 0 4 5 4 2 ? ␣ . . . < / p > <.*?> lazy < p > a n y ␣ u p d a t e ␣ o n ␣ T W - 1 0 4 5 4 2 ? ␣ . . . < / p > greedy runs to the last >; lazy stops at the first
Greedy <.*> (red) swallows the whole web-form message from the first < to the last >; lazy <.*?> (green) takes only the two tags.

Quantifiers are greedy by default: .* runs to the end of the text, then gives characters back only until the rest fits. In <.*> the last > closes </p>, so the match is the whole message. Add ? and the quantifier is lazy: <.*?> stops at the first >.

Break it: a greedy tag remover

python
form = next(m["body"] for m in MESSAGES if m["id"] == "TW-M0271")
print(repr(form))
print(repr(re.sub(r"<.*>", "", form)))      # greedy: from the first < to the last >
print(repr(re.sub(r"<.*?>", "", form)))     # lazy: each tag on its own
print(repr(re.sub(r"<[^>]*>", "", form)))   # a class that cannot cross a >
print(ORDER.findall(re.sub(r"<.*>", "", form)), ORDER.findall(re.sub(r"<.*?>", "", form)))

Code walkthrough

  • TW-M0271 is one web-form line inside <p> and </p>.
  • The greedy pattern deletes everything, order number included, without an error.
  • [^>] is "any character except >", so <[^>]*> cannot run past a tag: often clearer than a lazy quantifier.

Output

text
'<p>any update on TW-104542? need the Causeway before 21/04 if poss</p>'
''
'any update on TW-104542? need the Causeway before 21/04 if poss'
'any update on TW-104542? need the Causeway before 21/04 if poss'
[] ['TW-104542']
TW-M0271 sub <.*> empty text orders: [] TW-M0271 sub <.*?> any update on... ['TW-104542'] no error is raised in either case
A greedy mistake raises no error: the message silently becomes empty and the order finder downstream returns nothing.

A greedy mistake fails silently: the message becomes empty and every later step works on nothing. That is why patterns are measured on the whole inbox, not one example.

6. Order numbers, emails, phone numbers and postcodes (Essential)

These four fields have fixed shapes, so they are where rules shine. Give each pattern a name and run them together.

MATCHES IN THE 1,800 MESSAGES order numbers 751 emails 366 phones 165 postcodes 81
Four short patterns, one pass over the inbox: 751 order numbers, 366 email addresses, 165 phone numbers and 81 postcodes.

Order numbers are always TW- and six digits, phones use the drama range 07700 900, and postcodes follow the UK shape. Real email addresses allow far more than this pattern does; it covers what customers write. A postcode with the right shape may still not exist.

python
FIELDS = {"order": ORDER, "email": EMAIL, "phone": PHONE, "postcode": PC}

def extract_fields(body):
    """Every field found in a message, as {name: [matched text, ...]}."""
    return {name: [m.group() for m in rx.finditer(body)] for name, rx in FIELDS.items()}

print(extract_fields(text))
for name, rx in FIELDS.items():
    hits = [m.group() for msg in MESSAGES for m in rx.finditer(msg["body"])]
    print(f"{name:9} {len(hits):4} matches in {sum(bool(rx.search(msg['body'])) for msg in MESSAGES):4} messages")

Code walkthrough

  • FIELDS names each pattern, so adding a field is one line.
  • extract_fields uses finditer and .group(), not findall, because PC has groups and findall would return tuples.

Output

text
{'order': ['TW-106489'], 'email': ['kofi.asante@mailbox.example', 'help@tidewell.example'], 'phone': ['07700 900070'], 'postcode': ['BS12 2FR']}
order      751 matches in  751 messages
email      366 matches in  344 messages
phone      165 matches in  165 messages
postcode    81 matches in   81 messages
366 EMAIL ADDRESSES FOUND 213 Tidewell's own 153 others help@tidewell.example, in quoted replies
A pattern finds shapes, not owners: 213 of the 366 addresses (grey) are Tidewell's own, quoted back in replies.

Two surprises. Of the 366 addresses found, 213 are Tidewell's own, quoted back: a pattern finds shapes, not owners. And TW-M1222 says "My psotcode is BS10 2SS": a rule keyed on "postcode is" would miss it; the shape pattern does not.

7. Dates and prices written many ways (Essential)

Dev needs a booking date as a date, not as "August 14th", and £2,199.00, 2199 pounds and 2.199 £ are one price. Extraction has two steps: find the text, then normalise it into one standard form.

DATES FOUND IN THE INBOX 05/03 branch 1 27.04. branch 1 10 March branch 2 11. März branch 2 17 de abril branch 2 March 12th branch 3 1: numbers, / or . 2: day, month name 3: month name, 14th
Dates in four languages fall into three shapes, and the date pattern has one branch for each shape.

Inbox dates come in many shapes across four languages: 15/03, 24 April 2026, August 14th, 12. Juni, 20 de julio. One pattern with three branches covers them: numbers with a slash or dot, a day before a month name, and a month name before a day.

python
MONTH = "|".join(w for lang in LANGS for w in MONTHS[lang].split())   # 48 month names, from block 2
DATE = re.compile(rf"""
    \b (?:[0-2]?\d|3[01]) [/.] (?:0?[1-9]|1[0-2]) (?:[/.](?:\d{{4}})?)?   # 15/03  12.06.  12/06/2026
  | \b \d{{1,2}} \.? \s (?:de\s)? (?:{MONTH}) \b (?:\s(?:de\s)?\d{{4}})?   # 1 April  12. Juni  20 de julio
  | \b (?:{MONTH}) \s \d{{1,2}} (?:st|nd|rd|th) \b                          # August 14th
""", re.IGNORECASE | re.VERBOSE)

print(DATE.findall(text))
for lang in LANGS:
    found = [m.group() for msg in MESSAGES if msg["language"] == lang for m in DATE.finditer(msg["body"])]
    print(lang, len(found), list(dict.fromkeys(found))[:4])

Code walkthrough

  • MONTH joins every month name in the four languages with |.
  • rf"..." is a raw f-string: {MONTH} is filled in, and {{4}} is doubled so re receives {4}.
  • The first branch allows days 0 to 31 and months 1 to 12, so £50.00 is not a date. dict.fromkeys drops repeats, keeping order.

Output

text
['24 April 2026']
en 628 ['March 12th', '10 March', '05/03', '25 February 2026']
de 85 ['11. März', '27. März 2026', '27.04.', '19. April 2026']
es 49 ['3/3', '17 de abril', '24 de abril de 2026', '16/4']
fr 47 ['1 mars 2026', '15 mars', '22 février', '10 mars 2026']
15/03 2026-03-15 August 14th 2026-08-14 12. Juni 2026-06-12 20 de julio 2026-07-20 3/7 2026-07-03 to_iso UK: 3 July US: 7 March
to_iso turns every found date into one form; 3/7 is amber because the text alone cannot say whether it means 3 July or 7 March.

to_iso turns each found date into ISO form, 2026-07-03, assuming 2026 when no year is written. In the UK 3/7 is 3 July; in the United States it is 7 March. A pattern cannot know which. Tidewell writes day first, so the code does too.

python
MONTH_NUMBER = {w.lower(): i % 12 + 1 for i, w in enumerate(MONTH.split("|"))}

def to_iso(found, year=2026):
    """'August 14th' -> '2026-08-14'; '12/06/2026' -> '2026-06-12'. Day before month, as in the UK."""
    nums = [int(n) for n in re.findall(r"\d+", found)]
    word = re.search(r"[^\W\d_]{3,}", found)                  # three or more letters: a month name
    if word and word.group().lower() in MONTH_NUMBER:
        day, month = nums[0], MONTH_NUMBER[word.group().lower()]
        year = nums[1] if len(nums) > 1 else year
    else:
        day, month = nums[0], nums[1]
        year = nums[2] if len(nums) > 2 else year
    return f"{year}-{month:02d}-{day:02d}"

for s in ["15/03", "August 14th", "24 April 2026", "12. Juni", "20 de julio", "12/06/2026", "3/7"]:
    print(f"{s:14} -> {to_iso(s)}")

Code walkthrough

  • Month names come in twelves, so position i % 12 + 1 is the month number.
  • [^\W\d_] means a word character that is not a digit or underscore: a letter in any language, ä included.
  • With a month name, the numbers are day then year; without one, day, month, year.

Output

text
15/03          -> 2026-03-15
August 14th    -> 2026-08-14
24 April 2026  -> 2026-04-24
12. Juni       -> 2026-06-12
20 de julio    -> 2026-07-20
12/06/2026     -> 2026-06-12
3/7            -> 2026-07-03
£2,199.00 2199 pounds 2.199 £ 2 199 £ 2199 to_gbp: drop .00, keep the digits
Four ways the inbox writes the Volt 2's price, in English, German and French styles, become one number in pounds: 2199.

Prices work the same way: a pound sign before the number, or a currency word or sign after it, with thousands split by a comma, dot or space.

python
MONEY = re.compile(r"""
    £ \d+ (?:,\d{3})* (?:\.\d{2})?                                   # £649  £2,199  £2,199.00
  | \b \d+ (?:[.,\s]\d{3})* \s? (?:£|pounds|Pfund|libras|livres)     # 549 pounds  2.199 £  2 199 £
""", re.IGNORECASE | re.VERBOSE)

def to_gbp(found):
    """'£2,199.00', '2.199 £' and '2199 pounds' -> 2199."""
    return int(re.sub(r"\D", "", re.sub(r"[.,]00\b", "", found)))

for s in ["£649", "£2,199.00", "2199 pounds", "2.199 £", "2 199 £", "45 Pfund"]:
    print(f"{s:12} -> {MONEY.fullmatch(s) is not None}  {to_gbp(s)}")

Code walkthrough

  • fullmatch succeeds only if the pattern covers the whole string: a quick test on sample values.
  • to_gbp removes pence written .00 (the inbox has no other pence), then every non-digit, \D.

Output

text
£649         -> True  649
£2,199.00    -> True  2199
2199 pounds  -> True  2199
2.199 £      -> True  2199
2 199 £      -> True  2199
45 Pfund     -> True  45
8 HAND-WRITTEN SENTENCES FROM HARD_TEST missed: sat missed: Wednesday missed: 2nd of September found: 3/7 missed: Juli found: 3 septembre found: 549 £ missed: 5.000 euros inbox recall 0.998; here 3 of 8
Green found, red missed: patterns that score 0.998 recall on the template inbox find 3 of 8 values in hand-written messages.

These patterns were written for the inbox's own templates. Real mail is less tidy: eight sentences copied from HARD_TEST, the hand-written test set in course-guide/06, show the gap.

python
HAND = [   # (text, gold date or price), copied from HARD_TEST in course-guide/06
    ("yo the gears on my Causeway are proper knackered, can u lot sort it sat?", "sat"),
    ("Can I bring my Harbour Kids 20 in for a Basic Service next week? Any day but Wednesday.", "Wednesday"),
    ("Is the Full Service still £129? If so I'll book it for the 2nd of September.", "2nd of September"),
    ("Ordered on 3/7, still waiting on TW-119880. SwiftDrop says 'exception'. What does that mean?", "3/7"),
    ("Der Akku von meinem Volt 2 lädt nicht mehr. Das Rad habe ich erst seit Juli.", "Juli"),
    ("Est-ce que je peux prendre rendez-vous pour un Full Service le 3 septembre ?", "3 septembre"),
    ("Super, die Erstattung von 549 £ dauert jetzt schon drei Wochen. Ganz toll.", "549 £"),
    ("Gana 5.000 euros al mes desde casa. Haz clic aquí.", "5.000 euros"),
]
for sentence, want in HAND:
    got = [m.group() for m in DATE.finditer(sentence)] + [m.group() for m in MONEY.finditer(sentence)]
    print(f"{want:17} {'found' if want in got else 'MISSED':7} {got}")

Code walkthrough

  • Each pair is a sentence and the value a person labelled in it. Every miss is a shape no pattern was written for: "sat", a weekday, "2nd of", a month alone, euros.

Output

text
sat               MISSED  []
Wednesday         MISSED  []
2nd of September  MISSED  ['£129']
3/7               found   ['3/7']
Juli              MISSED  []
3 septembre       found   ['3 septembre']
549 £             found   ['549 £']
5.000 euros       MISSED  []

8. A pattern's precision and recall (Essential)

"It works on Kofi's message" is not a measurement. Rosa asked how often the code is wrong, and two numbers answer it.

false alarms found, not gold right misses gold, not found found gold precision = right / found recall = right / gold
Blue is what the pattern found, green outline the gold spans; precision asks how much of blue is right, recall how much of the gold was found.

Compare what a pattern found with the gold spans. Precision is the share of found spans that are right: of what it flagged, how much was correct. Recall is the share of gold spans it found: of what was there, how much it caught.

python
def score(rx, label):
    """Exact-span precision and recall of a compiled pattern against the gold entities in MESSAGES."""
    gold = {(m["id"], a, b) for m in MESSAGES for a, b, l in m["entities"] if l == label}
    found = {(m["id"], x.start(), x.end()) for m in MESSAGES for x in rx.finditer(m["body"])}
    right = len(gold & found)
    missed = [by_id[i]["body"][a:b] for i, a, b in sorted(gold - found)]
    return {"found": len(found), "right": right, "false alarms": len(found) - right,
            "precision": round(right / len(found), 3), "recall": round(right / len(gold), 3), "missed": missed[:5]}

by_id = {m["id"]: m for m in MESSAGES}
LOOSE = re.compile(r"[A-Z]+-\d+", re.IGNORECASE)
for name, rx, label in [("TW-\\d{6}", ORDER_CS, "ORDER_ID"), ("[A-Z]+-\\d+ (i)", LOOSE, "ORDER_ID"),
                        ("\\bTW-\\d{6}\\b (i)", ORDER, "ORDER_ID"), ("DATE", DATE, "DATE"), ("MONEY", MONEY, "MONEY")]:
    print(f"{name:17}", score(rx, label))

Code walkthrough

  • Spans are stored as (message id, start, end), so the set intersection gold & found is the right answers and gold - found the misses. A span counts only if both ends match exactly.
  • LOOSE, "letters, a hyphen, digits", is a tempting shortcut. (i) marks the IGNORECASE patterns.

Output

text
TW-\d{6}          {'found': 709, 'right': 709, 'false alarms': 0, 'precision': 1.0, 'recall': 0.944, 'missed': ['tw-100474', 'tw-100474', 'tw-101077', 'tw-102156', 'tw-102121']}
[A-Z]+-\d+ (i)    {'found': 968, 'right': 751, 'false alarms': 217, 'precision': 0.776, 'recall': 1.0, 'missed': []}
\bTW-\d{6}\b (i)  {'found': 751, 'right': 751, 'false alarms': 0, 'precision': 1.0, 'recall': 1.0, 'missed': []}
DATE              {'found': 809, 'right': 807, 'false alarms': 2, 'precision': 0.998, 'recall': 0.998, 'missed': ['9 août 2026', '17 août 2026']}
MONEY             {'found': 276, 'right': 271, 'false alarms': 5, 'precision': 0.982, 'recall': 0.982, 'missed': ['£500.00', '£100', '£50', '£250.00', '£39']}
precision recall TW-\d{6} 1.000 0.944 [A-Z]+-\d+ (i) 0.776 1.000 \bTW-\d{6}\b (i) 1.000 1.000
On one axis from 0 to 1: the strict pattern misses lower-case numbers, the loose one raises 217 false alarms, the bounded case-insensitive one gets both right.

The order patterns show the trade. The case-sensitive one is never wrong but misses 42 numbers: high precision, lower recall. The loose one finds every order number plus 217 things that are not orders, mostly the battery batch VB-0426 and repair tickets. Only the bounded, case-insensitive pattern gets both.

THE ERRORS BEHIND 0.998 AND 0.982 missed 17 août 2026 mojibake broke the month missed £500.00 match starts one character late false alarm 1.149 £ date branch reads 1.1
Every error has a reason you can read: mojibake causes the misses, and a German price written 1.149 £ looks like a date.

Read the errors before trusting the scores. Every miss is mojibake, text broken by a wrong encoding (Lesson 2): août arrived as août, £ as £. Clean first, then match. The two date false alarms are German prices: in 1.149 £ the date branch sees 1.1. And remember the eight hand-written sentences: near-perfect scores on template messages say the patterns fit the templates, not real mail.

In production

clean Lesson 2 rules.py patterns extract fields score P and R tests Lesson 4 clean first, then match, then measure
This lesson builds the blue steps and the green measurement; cleaning comes before them and Lesson 4's tests after (grey).

Teams that run extraction rules for years keep every pattern in one file, named and commented, and clean text before matching. Python's documentation notes that re caches recently used patterns, so compiling is mostly for readability. They report precision and recall per field on a labelled set after every change, because a fix for one shape often breaks another, and they read the misses. The regex package (pinned in course-guide/02) helps where re falls short on Unicode. Patterns with nested quantifiers can take exponential time on some inputs: Lesson 4 starts there.

Save the patterns as the first part of inbox/rules.py. %%writefile does not create folders in Colab, so make inbox/ first.

python
import os
os.makedirs("inbox", exist_ok=True)
if not os.path.exists("inbox/__init__.py"):     # an earlier lesson may have made it already
    open("inbox/__init__.py", "w").close()
print(sorted(os.listdir("inbox")))

Code walkthrough

  • exist_ok=True makes the call safe if the folder exists; an empty __init__.py makes inbox an importable package, and an existing one is left alone.

Output

text
['__init__.py']
python
%%writefile inbox/rules.py
"""Field extraction with regular expressions (Module 1, Lesson 3). Lesson 4 adds more."""
import re

ORDER = re.compile(r"\bTW-\d{6}\b", re.IGNORECASE)
EMAIL = re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")
PHONE = re.compile(r"(?:\+44\s?|0)7700\s?900\d{3}\b")
POSTCODE = re.compile(r"\b(?P<area>[A-Z]{1,2}\d{1,2}[A-Z]?)\s?(?P<unit>\d[A-Z]{2})\b", re.IGNORECASE)
MONTHS = ("January February March April May June July August September October November December "
          "Januar Februar März April Mai Juni Juli August September Oktober November Dezember "
          "enero febrero marzo abril mayo junio julio agosto septiembre octubre noviembre diciembre "
          "janvier février mars avril mai juin juillet août septembre octobre novembre décembre").split()
MONTH = "|".join(MONTHS)
MONTH_NUMBER = {w.lower(): i % 12 + 1 for i, w in enumerate(MONTHS)}
DATE = re.compile(rf"""
    \b (?:[0-2]?\d|3[01]) [/.] (?:0?[1-9]|1[0-2]) (?:[/.](?:\d{{4}})?)?
  | \b \d{{1,2}} \.? \s (?:de\s)? (?:{MONTH}) \b (?:\s(?:de\s)?\d{{4}})?
  | \b (?:{MONTH}) \s \d{{1,2}} (?:st|nd|rd|th) \b
""", re.IGNORECASE | re.VERBOSE)
MONEY = re.compile(r"""
    £ \d+ (?:,\d{3})* (?:\.\d{2})?
  | \b \d+ (?:[.,\s]\d{3})* \s? (?:£|pounds|Pfund|libras|livres)
""", re.IGNORECASE | re.VERBOSE)
FIELDS = {"order": ORDER, "email": EMAIL, "phone": PHONE, "postcode": POSTCODE, "date": DATE, "money": MONEY}

Code walkthrough

  • The patterns from this lesson, with MONTHS copied from block 2 so the file stands alone.
python
%%writefile -a inbox/rules.py


def extract_fields(body):
    """Every field found in a message, as {name: [matched text, ...]}."""
    return {name: [m.group() for m in rx.finditer(body)] for name, rx in FIELDS.items()}


def to_iso(found, year=2026):
    """'August 14th' -> '2026-08-14'. Day before month, as in the UK; 2026 when no year is written."""
    nums = [int(n) for n in re.findall(r"\d+", found)]
    word = re.search(r"[^\W\d_]{3,}", found)
    if word and word.group().lower() in MONTH_NUMBER:
        day, month, year = nums[0], MONTH_NUMBER[word.group().lower()], nums[1] if len(nums) > 1 else year
    else:
        day, month, year = nums[0], nums[1], nums[2] if len(nums) > 2 else year
    return f"{year}-{month:02d}-{day:02d}"


def to_gbp(found):
    """'£2,199.00', '2.199 £' and '2199 pounds' -> 2199."""
    return int(re.sub(r"\D", "", re.sub(r"[.,]00\b", "", found)))

Code walkthrough

  • -a appends to the file. These are the lesson's three functions; extract_fields now covers six fields.
python
from inbox.rules import extract_fields, to_iso, to_gbp
fields = extract_fields(text)
print(fields)
print([to_iso(d) for d in fields["date"]], to_gbp("£2,199.00"))

Code walkthrough

  • Importing from inbox.rules proves the file works on its own.

Output

text
{'order': ['TW-106489'], 'email': ['kofi.asante@mailbox.example', 'help@tidewell.example'], 'phone': ['07700 900070'], 'postcode': ['BS12 2FR'], 'date': ['24 April 2026'], 'money': []}
['2026-04-24'] 2199

What you added to the Tidewell inbox

  • inbox/rules.py: named patterns for six fields, with extract_fields, to_iso and to_gbp.
  • A score function for any pattern's precision and recall against gold spans.
  • Rosa's first numbers: order numbers at 1.0 precision and recall, dates at 0.998, prices at 0.982, every miss caused by mojibake.
  • A warning with evidence: on eight hand-written sentences the patterns found three values.

Check yourself

  1. Why does \bTW-\d{6}\b reject TW-1064890?

A seventh digit follows the sixth, and \b needs a gap between a word character and a non-word character. Two digits have no gap between them.

  1. What does re.findall(r"(\d{2})/(\d{2})", "15/03 and 21/04") return?

[('15', '03'), ('21', '04')]: with groups, findall returns the groups as tuples. Use finditer and .group() for whole matches.

  1. A pattern finds 968 spans, 751 right, and there were 751 gold spans. Precision and recall?

Precision 751 / 968 = 0.776, recall 751 / 751 = 1.0: the loose order pattern.

  1. Which date score should Rosa hear, 0.998 or "3 of 8"?

Both, with the reason: the inbox comes from the templates the patterns were written for; the hand-written sentences are closer to real mail.

Exercise

Repair tickets look like RJ-2026-0412. Write a pattern TICKET that finds them in any case and nothing else, then run the check.

python
def check_ticket(rx):
    hits = [m.group() for msg in MESSAGES for m in rx.finditer(msg["body"])]
    expected = sum(msg["body"].lower().count("rj-2026-") for msg in MESSAGES)
    assert rx.search("moving my booking (RJ-2026-0412) please") and rx.search("rj-2026-0301")
    assert not rx.search("RJ-2026-04123") and not rx.search("TW-104233") and not rx.search("RJ-26-0412")
    assert len(hits) == expected, (len(hits), expected)
    print(len(hits), "tickets found; all checks passed")

Code walkthrough

  • expected counts ticket prefixes with plain string methods, an independent check. The asserts try two strings that must match and three that must not.
python
TICKET = re.compile(r"\bRJ-2026-\d{4}\b", re.IGNORECASE)
check_ticket(TICKET)

Code walkthrough

  • A literal prefix, exactly four digits, boundaries at both ends, and IGNORECASE for lower-case messages.

Output

text
59 tickets found; all checks passed

Key terms

TermMeaning
Regular expressionA pattern describing a family of strings by shape
Character classAny one character from a set: \d, [A-Z]
QuantifierHow many times the piece before repeats: ?, *, +, {6}
AnchorA position a match must sit at: ^, $
Word boundary\b, the gap between a word and a non-word character
FlagA switch for the whole pattern, such as IGNORECASE
Named group(?P<name>...), part of a match kept under a name
Greedy, lazyTaking as much as possible, or (with ?) as little
PrecisionShare of found spans that are right
RecallShare of gold spans that were found

Sources

  • Ken Thompson, "Regular Expression Search Algorithm", Communications of the ACM 11(6), 1968. https://doi.org/10.1145/363347.363387
  • Python Software Foundation, "re: Regular expression operations", Python 3 documentation, 2026. https://docs.python.org/3/library/re.html
  • A.M. Kuchling, "Regular Expression HOWTO", Python 3 documentation, 2026. https://docs.python.org/3/howto/regex.html