Lesson 1: The core toolkit you bring on site

Module 1 · Lesson 1 · About 50 minutes · Needs: you have written small programs in any language

The situation

Monday 16 March 2026, your first day at Harbourline, spent shadowing Joan Petrov. That evening Mei Lin sends you a sample of this morning's TMS export, a text file nobody has documented. One line catches your eye, because Joan's spreadsheet has a note on it: "HF-88301 ... 5 days late. 10% = 42.00".

On Wednesday Joan wants a cross-system lookup. Before then you need a small repository you trust: code that reads the file, a test that pins the late rule, a shell check and a clean Git history. Nobody from Larkfield is on site to help.

A real case, first

On 2 July 2019 at 13:42 UTC, an engineer on Cloudflare's firewall team deployed "a minor change": one new rule for its web application firewall (WAF). The rule held a regular expression that exhausted the CPUs on every core handling HTTP traffic worldwide. At 14:07 the team switched the WAF off globally; by 14:09 traffic was back to normal.

Cloudflare's write-up (12 July 2019) is frank about why nothing caught it. The pull request was approved. The rule had tests, but "what it didn't do was test for runaway CPU utilization". A protection against slow regular expressions had been removed by mistake in a refactor weeks earlier. And rule changes could go out globally in one step, unlike Cloudflare's other software.

One line of text wrangling, a test that measured the wrong thing, a review that approved it: every one of those skills is in this lesson.

What you will be able to do

• Clean a messy export in Python, and read the same code in TypeScript.
• Choose a data structure by how you look things up, and predict how cost grows.
• Write a tight regular expression, and show why a loose one stalls.
• Script a shell check, and map unfamiliar code by following one question.
• Pin a bug in a failing test, fix it on a branch, and review the diff.

1. One language fluently: clean the export (Essential)

On site, nobody waits while you look up how to read a file. Being fluent in one language means turning messy text into clean records while Joan is talking to you; this course uses Python.

The running example is the line for HF-88301: eight fields separated by semicolons. Nobody wrote down what PROM_DT or STAT mean; on your first evening you can only guess that X means trouble. Paste the export, copied exactly from the course data:

TMS_EXPORT = """CONS_NO;CUST;PROM_DT;DEL_DT;FRT_CHG;DV;WGT;STAT
HF-88101;C-1042;02/03/2026;02/03/2026;180.00;;420;D
HF-88102;C-1042;03/03/2026;09/03/2026;240.00;;610;D
HF-88102;C-1042;03/03/2026;09/03/2026;240.00;;610;D
HF-88145;C-1077;2026-03-04;2026-03-04;95.00;;60;D
HF-88190; C-1130 ;05/03/2026;;130.00;;150;T
HF-88207;C-1188;06/03/2026;06/03/2026;310,00;6500.00;380;D
HF-88233;C-1203;09/03/2026;10/03/2026;75.00;;25;D
HF-88260;c-1077;10/03/2026;10/03/2026;110.00;;90;D
HF-88301;C-1188;11/03/2026;16/03/2026;420.00;;700;X
"""

import csv, io
rows = list(csv.DictReader(io.StringIO(TMS_EXPORT), delimiter=";"))
print(len(rows), "rows,", len({r["CONS_NO"] for r in rows}), "distinct consignments")
print(rows[4]["CUST"], "|", rows[5]["FRT_CHG"], "|", rows[7]["CUST"])

Code walkthrough

• csv.DictReader turns each line into a dictionary keyed by the header; io.StringIO lets it read a string as if it were a file.
• The output: 9 rows, 8 distinct consignments (HF-88102 appears twice), then three traps: ' C-1130 ' with spaces, 310,00 with a decimal comma, and c-1077 in lower case.

Every trap has one clean answer. Dates are day first: 11/03/2026 is 11 March, not 3 November.

The cleaner never removes duplicates on purpose. Storing each shipment under its consignment number means the repeated row simply overwrites itself.

from datetime import datetime

def parse_date(text):
    text = text.strip()
    if not text:
        return None                                  # no delivery scan yet
    for fmt in ("%d/%m/%Y", "%Y-%m-%d"):             # day first, or ISO
        try:
            return datetime.strptime(text, fmt).date()
        except ValueError:
            pass
    raise ValueError(f"unknown date: {text!r}")

def clean(row):
    return {"ref": row["CONS_NO"].strip(),
            "customer": row["CUST"].strip().upper(),
            "promised": parse_date(row["PROM_DT"]),
            "delivered": parse_date(row["DEL_DT"]),
            "freight_charge": float(row["FRT_CHG"].replace(",", ".")),
            "status": row["STAT"].strip()}

shipments = {}
for row in rows:
    s = clean(row)
    shipments[s["ref"]] = s                          # a repeat overwrites itself
print(len(rows), "rows in,", len(shipments), "shipments out")
print(shipments["HF-88301"])

Code walkthrough

• parse_date tries day first, then ISO; strptime raises ValueError when a format does not fit, so the loop moves on. Empty means no delivery scan (None); anything else fails loudly rather than guessing.
• clean uses .strip() for spaces, .upper() for c-1077, and .replace(",", ".") so 310,00 becomes a number.
• The output is 9 rows in, 8 shipments out, then HF-88301 with real dates (11 and 16 March), freight 420.0 and status X.

2. Python and TypeScript, a second language passably (Essential)

The customer chooses the stack, not you. Many customer integrations and most web front ends are TypeScript, so being passable in a second language means you can read it, change a line safely and run it.

The same cleaner in TypeScript has the same shape. Reading across, line for line, is the fastest way to become passable.

type Shipment = { ref: string; customer: string; freightCharge: number; status: string };

function clean(line: string): Shipment {
  const f = line.split(";").map((v) => v.trim());
  return {
    ref: f[0],
    customer: f[1].toUpperCase(),
    freightCharge: Number(f[4].replace(",", ".")),
    status: f[7],
  };
}

const lines = [
  "HF-88207;C-1188;06/03/2026;06/03/2026;310,00;6500.00;380;D",
  "HF-88301;C-1188;11/03/2026;16/03/2026;420.00;;700;X",
];
const byRef = new Map<string, Shipment>();
for (const line of lines) {
  const s = clean(line);
  byRef.set(s.ref, s);
}
console.log(byRef.get("HF-88301"));
console.log(byRef.get("HF-88207")?.freightCharge);

Code walkthrough

• type Shipment names the fields and their types; TypeScript checks them, plain JavaScript would not.
• split(";").map((v) => v.trim()) splits and strips every field; Number(...) is Python's float.
• Map<string, Shipment> is the dictionary; ?. returns undefined instead of crashing when a key is missing.

Colab runs Python, not TypeScript, so run this on your own computer, saved as parse.ts:

# not run in Colab; tested on a laptop with Node.js 24.17
node parse.ts

Code walkthrough

• Node.js strips the types and runs the file; its documentation says this is on by default from v22.18.0 and v23.6.0 (checked October 2026). It does not type-check: run tsc for that.
• The output is HF-88301 as { ref: 'HF-88301', customer: 'C-1188', freightCharge: 420, status: 'X' }, then 310.

3. Data structures and complexity intuition (Essential)

Your code will meet a year of Harbourline's data, not nine rows. How you store things decides whether the nightly check takes a blink or a coffee break.

Joan's sheet grows by about 1,400 exceptions a week, about 72,800 a year. Each night you ask, for each exception in the export, "is it already in the sheet?". A list is checked row after row. A set hashes the reference and jumps to one slot.

import random, time
random.seed(16)                                       # simulated data, same every run
sheet_list = [f"HF-{n}" for n in random.sample(range(100000, 999999), 72_800)]
sheet_set = set(sheet_list)
tonight = random.sample(sheet_list, 1_000)            # 1,000 refs to check

t = time.perf_counter(); sum(r in sheet_list for r in tonight); t_list = time.perf_counter() - t
t = time.perf_counter(); sum(r in sheet_set for r in tonight); t_set = time.perf_counter() - t

comparisons = sum(sheet_list.index(r) + 1 for r in tonight)
print(f"list: {comparisons:,} comparisons, {t_list:.3f} s")
print(f"set:  {len(tonight):,} hash lookups, {t_set:.5f} s")

Code walkthrough

• random.seed(16) fixes the simulated references, so everyone gets the same lists.
• r in sheet_list scans from the start; r in sheet_set looks in one place. sheet_list.index(r) + 1 counts the rows a scan reads.
• The run printed 36,845,875 comparisons for the list and 1,000 lookups for the set. Times depend on your machine (on the test laptop, 0.12 s against well under a millisecond).

The count is the point, not the seconds. On average a scan reads half the list, about 36,800 rows, per lookup. The set costs about one step, however big it gets.

That is complexity intuition: ask what happens when the data grows. A year is 52 weeks. Work that reads every row once grows 52 times. A loop inside a loop grows 52 times 52: 2,704 times. A one-second check on a week becomes 45 minutes on a year.

4. Regex and text wrangling (Recommended)

A regular expression (regex) is a small pattern language for text. It checks that a line has the shape you expect, or pulls every consignment number out of an email, without writing a parser.

Each part of the pattern accepts only what it should, so a line of the wrong shape is rejected: the right default for a file nobody documented.

import re

LINE = re.compile(r"^(HF-\d{5});\s*([Cc]-\d{4})\s*;([\d/-]{10});([\d/-]{10})?;(\d+[.,]\d{2});")

lines = TMS_EXPORT.splitlines()[1:]
bad = [l for l in lines if not LINE.match(l)]
print("lines that fail the shape check:", bad)
print(LINE.match(lines[-1]).groups())

email = "HF-88301 arrived five days late, and HF-88102 was late too."
print(re.findall(r"HF-\d{5}", email))

Code walkthrough

• \d{5} is exactly five digits, [Cc] either case, \s* stray spaces, (...)? an optional date, [.,] either decimal mark. ^ anchors at the start; brackets capture parts.
• The run printed an empty list (all nine lines fit), the five captured fields of HF-88301, and ['HF-88301', 'HF-88102'] from the email.

Break it: the pattern from the Cloudflare outage

Cloudflare's post simplifies its rule to .*.*=.*. Each .* first grabs the whole line, then gives characters back one at a time. When a line does not match, the engine tries every way to split it before giving up. That is catastrophic backtracking.

def timed(pattern, line):
    t = time.perf_counter()
    found = re.search(pattern, line)
    return found, (time.perf_counter() - t) * 1000      # milliseconds

for n in (250, 500, 1000, 2000):
    line = "x=" + "x" * n                                # no semicolon, so no match
    _, slow = timed(r".*.*=.*;", line)
    _, tight = timed(r"^[^=;]*=[^;]*;", line)
    print(f"{n:5} chars   loose {slow:8.2f} ms   tight {tight:6.3f} ms")

Code walkthrough

• timed runs one search and returns the result and the milliseconds.
• The loose pattern adds ; to Cloudflare's, so a line without one never matches. The tight pattern says "not = or ;" ([^=;]) and is anchored, so there is one way to try.

Each doubling multiplied the loose pattern's time by about seven, close to cubic growth. Python's re backtracks like this. Cloudflare's fixes included engines with run-time guarantees, re2 or Rust's regex. Your habit: never .* when you can say what the characters are.

5. Command line and shell scripting (Essential)

On a customer's server there may be no notebook, only a terminal. A few commands chained together answer most "what is in this file?" questions faster than code.

A pipe sends one command's output into the next: skip the header, cut field 1, sort, keep repeats. Colab runs shell scripts through Python, which also hands you the exit code.

import subprocess
from pathlib import Path

Path("tms_export.csv").write_text(TMS_EXPORT)
script = r"""
set -euo pipefail
f=tms_export.csv
echo "data rows: $(tail -n +2 "$f" | wc -l | tr -d ' ')"
echo "STAT=X: $(grep ';X$' "$f" | cut -d';' -f1)"
dupes=$(tail -n +2 "$f" | cut -d';' -f1 | sort | uniq -d)
if [ -n "$dupes" ]; then echo "repeated: $dupes"; exit 1; fi
echo "no repeats"
"""
r = subprocess.run(["bash", "-c", script], capture_output=True, text=True)
print(r.stdout, "exit code:", r.returncode)

Code walkthrough

• set -euo pipefail stops the script on any failed command, unset variable or broken pipe.
• tail -n +2 skips the header, grep ';X$' keeps lines ending ;X, cut -d';' -f1 keeps field 1.
• sort | uniq -d prints values that appear twice; uniq only compares neighbours, so sort first.
• The run printed 9 data rows, HF-88301 for X, repeated: HF-88102, and exit code 1.

The exit code is how programs talk: 0 is success, anything else failure. The CI pipeline in Lesson 4 runs this script and stops the build on 1, which is why it exits rather than printing a warning.

6. Reading unfamiliar codebases fast (Essential)

On site you inherit code: Mei's export scripts, ClaimBot, a CRM client library. You never read it all. You follow one question.

Our question: csv.DictReader gives us column names, but where from? Map what exists, find the entry point, search for the name, run it to confirm. Python's own csv module is real code you have never read.

import ast, inspect

path = inspect.getsourcefile(csv)
source = Path(path).read_text()
print(path.split("/")[-1], len(source.splitlines()), "lines")

for node in ast.parse(source).body:                      # step 1: the map
    if isinstance(node, (ast.ClassDef, ast.FunctionDef)):
        print("  ", type(node).__name__.replace("Def", "").lower(), node.name)

for n, line in enumerate(source.splitlines(), 1):        # step 3: grep the name
    if "_fieldnames =" in line:
        print(n, line.strip())

print(csv.DictReader(io.StringIO("A;B\n1;2"), delimiter=";").fieldnames)   # step 4: run it

Code walkthrough

• inspect.getsourcefile(csv) finds the module's file. ast.parse turns it into a tree; walking .body lists classes and functions without running anything.
• The numbered-line loop is a grep for _fieldnames =. The last line runs a tiny example.

The map shows seven classes. The grep finds three lines; the one that matters is self._fieldnames = next(self.reader): the header is the first row, read the first time you ask. The run prints ['A', 'B']. Line numbers vary between Python versions; the method does not.

7. Git and branching (Essential)

Without Git you cannot say what changed, when, or undo it. Mei will inherit this repository, and its history tells her why each line exists.

A branch is a separate line of commits. main holds what everyone trusts; fixes join it only after review. The first commit holds a quick days_late that counts calendar days, which is how Joan's note got "5 days late".

import shutil, sys

REPO = Path("harbourline-claims")
shutil.rmtree(REPO, ignore_errors=True)                     # start clean on a rerun
(REPO / "claims").mkdir(parents=True)
(REPO / "tests").mkdir()
(REPO / "claims" / "__init__.py").write_text("")
(REPO / "claims" / "lateness.py").write_text('''from datetime import date

def days_late(promised: date, delivered: date) -> int:
    return (delivered - promised).days
''')

def git(*args):
    r = subprocess.run(["git", "-c", "user.name=You", "-c", "user.email=you@larkfield.example", *args],
                       cwd=REPO, capture_output=True, text=True)
    return (r.stdout + r.stderr).strip()

git("init", "-b", "main"); git("add", ".")
print(git("commit", "-m", "Add days_late for the late-credit rule").splitlines()[0])
print(git("switch", "-c", "fix/working-days"))

Code walkthrough

• The first lines build a tiny repository: a claims package with lateness.py, and an empty tests folder.
• git(...) runs one Git command in the repository; -c sets a name and email for that command only.
• init -b main, add . and commit record the files; switch -c fix/working-days creates the fix branch and moves onto it.

8. Testing and debugging under time pressure (Essential)

When something is wrong in front of the customer, the fastest safe move is not to fix it. It is to pin it: a test that fails for the same reason.

The policy counts working days. HF-88301 was promised Wednesday 11 March and delivered Monday 16 March: 5 calendar days, 3 working days. The credit is the same here, but a Friday promise delivered on Tuesday is 4 calendar days and 2 working days: no credit due.

The failing test is the reproduction, written down. It runs in a second and, once fixed, stops the bug returning. Colab already has pytest, the standard Python test runner.

(REPO / "tests" / "test_lateness.py").write_text('''from datetime import date
from claims.lateness import days_late

def test_hf_88301_is_three_working_days_late():
    # promised Wed 11 March, delivered Mon 16 March 2026
    assert days_late(date(2026, 3, 11), date(2026, 3, 16)) == 3

def test_on_time_is_zero():
    assert days_late(date(2026, 3, 2), date(2026, 3, 2)) == 0
''')

def run_tests():
    r = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider"],
                       cwd=REPO, capture_output=True, text=True)
    return [l for l in r.stdout.splitlines() if l.startswith("E ") or " passed" in l or " failed" in l]

print("\n".join(run_tests()))

Code walkthrough

• pytest runs every function whose name starts with test_; assert fails the test and shows both values.
• run_tests runs pytest in the repository and keeps the error lines and the summary. -p no:cacheprovider stops it writing a cache folder.
• The run printed E assert 5 == 3 and 1 failed, 1 passed: the bug, reproduced.

The fix is the course's shared working-day function, moved into the module with the bank holidays of England and Wales.

(REPO / "claims" / "lateness.py").write_text('''from datetime import date, timedelta

BANK_HOLIDAYS_2026 = [date(2026, 1, 1), date(2026, 4, 3), date(2026, 4, 6), date(2026, 5, 4),
                      date(2026, 5, 25), date(2026, 8, 31), date(2026, 12, 25), date(2026, 12, 28)]

def days_late(promised: date, delivered: date) -> int:
    """Working days after `promised`, up to and including `delivered`."""
    days, d = 0, promised
    while d < delivered:
        d += timedelta(days=1)
        if d.weekday() < 5 and d not in BANK_HOLIDAYS_2026:
            days += 1
    return days
''')
print("\n".join(run_tests()))

Code walkthrough

• BANK_HOLIDAYS_2026 lists the eight 2026 bank holidays for England and Wales, from gov.uk.
• The loop steps a day at a time and counts Monday to Friday (weekday() < 5) that are not holidays. The run printed 2 passed.

Break it: a test that measures the wrong thing

Cloudflare's rule passed its tests because they checked the answer, not the time. Check the two patterns from section 4 both ways.

line = "x=" + "x" * 2000
for name, pattern in (("loose", r".*.*=.*;"), ("tight", r"^[^=;]*=[^;]*;")):
    found, ms = timed(pattern, line)
    right_answer = found is None                      # the line has no ';', so no match
    fast_enough = ms < 50                             # a budget of 50 ms per line
    print(f"{name:5}  right answer: {right_answer}   within budget: {fast_enough}   ({ms:.1f} ms)")

Code walkthrough

• right_answer is the ordinary test: both patterns correctly find nothing.
• fast_enough adds a 50 ms budget. The loose pattern took about half a second and fails; the tight one passes. Cloudflare added performance profiling of every rule to its test suite.

9. Code review before merging (Essential)

Code review is a second reading of a change before it reaches main. Alone on site, review your own diff as if a stranger wrote it, then ask Mei to look.

git("add", ".")
git("commit", "-m", "Count working days in days_late, not calendar days")
print(git("diff", "--stat", "main...fix/working-days"))

Code walkthrough

• git add . stages everything in the folder, the habit behind the problem below.
• diff --stat main...fix/working-days lists each file the branch changed since it left main.

Read the list before the code. Two files are expected. The other three are compiled .pyc files Python wrote when the tests ran: noise that differs between machines. A reviewer reading only the code would miss them.

(REPO / ".gitignore").write_text("__pycache__/\n*.pyc\n")
git("rm", "-r", "--cached", "--quiet", "claims/__pycache__", "tests/__pycache__")
git("add", ".")
git("commit", "-m", "Ignore compiled Python files")
print(git("diff", "--stat", "main...fix/working-days"))

Code walkthrough

• .gitignore lists patterns Git must never stage. git rm -r --cached drops the compiled files from Git without deleting them.
• The new summary shows three files: .gitignore, claims/lateness.py, tests/test_lateness.py. Small enough to review properly.

In production

Cloudflare's change was reviewed and still took the network down, so the write-up changed the system around the review. Rules gained performance profiling in tests, the regex protection came back, and rule changes moved to staged rollouts "in the same manner used for other software at Cloudflare", with emergency global pushes kept for active attacks.

Good teams keep diffs small enough to read in one sitting, let machines check what people miss (tests, time budgets, stray files), and release in stages. At Harbourline the first stage will be three clerks, Joan, Dev and Priti, before all nine.

What you added to the engagement

• harbourline-claims, a Git repository with main and a reviewed fix branch.
• claims/lateness.py, counting working days late, pinned by two tests.
• A cleaner and a regex shape check for the TMS export.
• A shell check that exits with 1 when the export repeats a consignment.

Check yourself

1. Why does a dictionary keyed by consignment number remove the repeated HF-88102 row?
2. A check compares every exception with every other. The data grows from a week to a year. Roughly how much slower is it?
3. Why can .*.*=.* take seconds on one line, and what habit avoids it?
4. Why does the shell script exit 1 instead of printing a warning?
5. Why write the failing test before fixing days_late?

1. A dictionary holds one value per key, so the second row overwrites the first.
2. About 2,704 times (52 squared): a loop inside a loop grows with the square of the data.
3. The wildcards can split a non-matching line in very many ways, and the engine tries them all. Say exactly which characters are allowed, and anchor the pattern.
4. CI reads the exit code, not the words. 1 stops the pipeline.
5. It proves you reproduced the bug, shows when the fix works, and stops it returning.

Exercise

Add two tests to tests/test_lateness.py: HF-88102, promised Tuesday 3 March and delivered Monday 9 March 2026, is 4 working days late; a shipment promised Thursday 2 April and delivered Tuesday 7 April 2026 is 1 working day late, because of Good Friday and Easter Monday. Then run the check.

result = "\n".join(run_tests())
print("done" if "4 passed" in result else "not yet: " + result)

Code walkthrough

• run_tests is the helper from section 8. The check prints done when four tests pass, and not yet with pytest's summary before you add yours.

Key terms

Sources

• Cloudflare, "Details of the Cloudflare outage on July 2, 2019", 12 July 2019. https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/
• Node.js documentation, "Modules: TypeScript" (type stripping history), checked October 2026. https://nodejs.org/api/typescript.html