CourseForward Deployed Engineer: The Complete Course · Module 1: Engineering Floor · part 10 of 85
Part 10 · Module 1: Engineering Floor

Lesson 4: Infrastructure: getting the claims tool off your laptop

16 min read·13 Oct 2026
Module 1 · Lesson 4 · About 50 minutes · Needs: Lesson 2 (a small web service); Docker on your own computer only to try section 1

The situation

The week before kickoff, Nadia Okafor forwards a note from Harbourline: the claims tool may run on one Linux virtual machine (VM) inside their network, and Viktor Hale, Head of IT and Security, wants three answers first. How is it packaged? Where do the credentials live? What does it need to reach? Mei Lin, who will own it after you leave, adds a fourth: when it breaks, where does she look?

This lesson follows one request. In a few weeks Joan Petrov will type HF-88301 into the tool, and that lookup will cross five pieces of infrastructure.

ONE LOOKUP: HF-88301 Joan's PC types HF-88301 DNS name to address firewall port 443 open? TLS certificate ok? container claims-api on VM proxy outbound gate CRM API vendor cloud
One lookup for HF-88301 crosses five pieces of infrastructure (blue) on its way to the CRM, and any one of them can stop it.

A real case, first

On Monday 4 January 2021, the first working day of the year, Slack went down. Laura Nolan's post-mortem on the Slack Engineering blog (1 February 2021) explains why. An AWS Transit Gateway, a managed hub linking Slack's private networks, "did not scale fast enough" when holiday traffic returned. Packet loss, on top of the usual hourly peak, saturated the web tier. At 6:57am PST, 99% of messages were being sent, against a usual rate above 99.999%.

SLACK, 4 JANUARY 2021 (PST) 6:57 99% sent usual 99.999% 7:00 web tier saturated 7:01-7:15 1,200 servers requested ~8:15 provisioning working again 9:15 enough hosts serving 10:40 network normal
Slack's morning on 4 January 2021: red marks failure, green recovery. Message success fell to 99% at 6:57 and the network was normal again at 10:40.

Autoscaling did its job: "We attempted to add 1,200 servers to our web tier between 7:01am PST and 7:15am PST." Most never served. The service that prepares new servers "hit two separate resource bottlenecks": "the Linux open files limit" and "an AWS quota limit". The dashboards went dark too, because they reached their database across the same hub. AWS raised the hub's capacity and the network was normal by 10:40am.

network hub overloaded web tier saturated autoscaling asks for 1,200 provision-service prepares servers open-files limit AWS quota new servers mostly not serving
An overloaded network hub started it; an open-files limit and an AWS quota stopped the servers meant to fix it. Blue is the system working as designed, red is failure.

Nothing exotic failed. You will set up the same pieces for Harbourline, at a smaller size.

What you will be able to do

  • Package the service as an image and run it with its database using Compose.
  • Load configuration that refuses to start without its secrets.
  • Answer four questions on a Linux server.
  • Find which network layer stopped a request.
  • Write a least-privilege request and a CI pipeline that cannot lie.

1. Docker and Compose (Essential)

Viktor's VM has a different Python from your laptop and none of your libraries. Unless you ship the environment with the code, "it works on my machine" is all you can promise.

Dockerfile the recipe build image main.py libraries python:3.14-slim run container running copy as appuser
A Dockerfile is built once into an image (your code on top, its base layers below); a container is a running copy of that image.

A container image is a package holding your code plus everything it needs: system files, Python, libraries. You describe it in a Dockerfile and build it once. A container is a running copy of an image, isolated from the rest of the machine. The same image runs on your laptop, in CI and on the VM, so they cannot drift apart.

text
# not run here, verified against https://docs.docker.com/reference/dockerfile/ and https://fastapi.tiangolo.com/deployment/docker/
FROM python:3.14-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY main.py .
RUN useradd --create-home appuser
USER appuser
EXPOSE 8000
CMD ["fastapi", "run", "main.py", "--port", "8000"]

Code walkthrough

  • FROM picks a small Linux with Python 3.14. Installing requirements before copying main.py lets Docker reuse that slow step when only code changes.
  • Copy named files: COPY . . next to a .env file bakes the secret into the image.
  • USER appuser avoids running as root. Viktor will look for this.
  • EXPOSE "doesn't actually publish the port" (Docker's reference). CMD starts FastAPI's server.
VM: COMPOSE.YAML api container db postgres:17 db:5432 volume db_data healthy first .env not in Git 127.0.0.1:8000
Compose runs the API and its database together, starts the API only once the database is healthy, and keeps the data in a volume (grey means supporting detail).

The service needs a database, so you run two containers. Compose describes containers that belong together in one file, compose.yaml, and gives them a private network where each is reachable by its service name.

yaml
# not run here, verified against https://docs.docker.com/reference/compose-file/services/
services:
  api:
    build: .
    ports: ["127.0.0.1:8000:8000"]
    env_file: .env
    depends_on:
      db: {condition: service_healthy}
  db:
    image: postgres:17
    environment:
      POSTGRES_USER: claims
      POSTGRES_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env}
    volumes: ["db_data:/var/lib/postgresql/data"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U claims"]
      interval: 5s
volumes:
  db_data:

Code walkthrough

  • build: . uses the Dockerfile above; db uses the official PostgreSQL image.
  • ports publishes 8000 on the VM's own address only. Without an address, Docker's docs warn, it binds to all interfaces, "bypassing host firewall rules".
  • env_file passes your settings in (section 2). ${DB_PASSWORD:?...} stops with that message if the value is missing.
  • depends_on waits for the database's healthcheck. The volume db_data keeps the data when the container is replaced.
bash
# not run here, verified against https://docs.docker.com/reference/cli/docker/compose/
docker compose up -d --build     # build if needed, start both in the background
docker compose ps                # what is running, and is the database healthy?
docker compose logs -f api       # follow the API's output
docker compose down              # stop and remove containers; the volume stays

Code walkthrough: -d gives your terminal back. down keeps db_data unless you add -v, so never add it out of habit.

Break it: point the API at localhost

localhost:5432 db:5432 api container db not reached connection refused api container db container reached by name
Inside a container, localhost is that container (red, refused); the database is reached by its service name, db (green).

Set the database address to localhost:5432 and the API reports "connection refused" although the database is running. Inside a container, localhost is that container. The fix is the service name, db:5432. The error never mentions the cause.

2. Secrets and configuration (Essential)

The same image must run on your laptop, the test VM and production, with different addresses and credentials. Build three images and you no longer know which one you tested.

one image laptop CRM sandbox .env file test VM CRM test API .env, chmod 600 production CRM live API secret store
One image runs in three places; only the configuration around it changes: which CRM it calls and where the token comes from.

Configuration is everything that differs between those places. A secret is configuration that grants access, such as the CRM token. Neither belongs in the code or the image. The usual home is environment variables, set by whatever starts the service.

.env in .gitignore Compose env_file container environment load_config at start never: Git never: the image never: a log line
The token travels from an ignored .env file into the container's environment (blue), and never into Git, the image or a log (red).

The token lives in a .env file listed in .gitignore; you commit a .env.example with names and no values. Compose loads the file into the container, and the code reads it once, at start-up.

python
import os

REQUIRED = ["CRM_BASE_URL", "CRM_TOKEN", "DATABASE_URL"]
SECRETS = {"CRM_TOKEN", "DATABASE_URL"}

class ConfigError(Exception):
    pass

def load_config(env=os.environ):
    missing = [name for name in REQUIRED if not env.get(name)]
    if missing:
        raise ConfigError("missing settings: " + ", ".join(missing))
    config = {name: env[name] for name in REQUIRED}
    config["LOG_LEVEL"] = env.get("LOG_LEVEL", "INFO")   # harmless, so a default is fine
    return config

def safe_to_log(config):
    return {k: ("****" + v[-4:] if k in SECRETS else v) for k, v in config.items()}

test_vm = {"CRM_BASE_URL": "https://api.crmvendor.example/v2",
           "CRM_TOKEN": "placeholder-token-7f3a",
           "DATABASE_URL": "postgresql://claims:placeholder@db:5432/claims"}
print(safe_to_log(load_config(test_vm)))

Code walkthrough

  • REQUIRED lists what the service cannot run without; SECRETS marks what must never be printed.
  • load_config reads the real environment by default; a dictionary makes it testable. missing collects every absent name at once.
  • LOG_LEVEL may have a default; a secret never does.
  • safe_to_log prints 'CRM_TOKEN': '****7f3a': enough to tell which token is loaded, not enough to use it.

Break it: start without the token

no check fail fast starts looks healthy first CRM call hours later error clerks watching start-up check load_config() missing settings CRM_TOKEN, DATABASE_URL refuses to start fixed before anyone sees
Refusing to start and naming both missing settings beats starting, looking healthy, and failing hours later in front of the claims team.
python
laptop = {k: v for k, v in test_vm.items() if k != "CRM_TOKEN"}
laptop["DATABASE_URL"] = ""
try:
    load_config(laptop)
except ConfigError as err:
    print("refused to start:", err)

Code walkthrough: laptop is a typical half-filled .env: no token, empty database address. The output is refused to start: missing settings: CRM_TOKEN, DATABASE_URL. This is failing fast. Without it the service would start, look healthy, and fail on the first CRM call.

3. Linux basics: processes, permissions, logs, networking (Essential)

The VM has no desktop. You reach it over SSH, often with less access than you want, and when Mei calls, the answer is somewhere in a terminal.

THE VM running? process claims-api ps aux allowed? .env, owner only ls -l what did it say? service logs journalctl -u listening? port 8000 ss -ltnp
Four questions cover most server problems, each with one command: is it running, is it allowed, what did it say, is it listening.
bash
# not run here (needs a Linux server), verified against each command's man page
ps aux | grep fastapi                        # running? one line per process, PID in column 2
ls -l .env                                   # allowed? permissions, owner, group
journalctl -u claims --since "10 min ago"    # what did it say? a service's recent logs
ss -ltnp                                     # listening? ports and the process behind each
ulimit -n                                    # how many files may a process open?

Code walkthrough

  • A process is a running program with an id, its PID. grep keeps matching lines.
  • journalctl -u shows one service's logs; claims is an example name.
  • ss -ltnp lists listening ports. 127.0.0.1:8000 means only the VM itself can connect.
  • ulimit -n prints the open-files limit Slack hit.
AS COPIED: 644 - file rw- owner r-- group r-- others FIXED: 600 - file rw- owner --- group --- others r=4 w=2 x=1 chmod 600
A permission string is read, write and run for owner, group and others: 644 lets every account read the token (red), 600 lets only the owner (green).

Every file has an owner, a group and permissions: read, write and run, for owner, group and everyone else. A carelessly copied .env is often readable by every account. Colab runs on Linux, so try it.

python
import stat, tempfile
from pathlib import Path

env_file = Path(tempfile.mkdtemp(prefix="claims-")) / ".env"
env_file.write_text("CRM_TOKEN=placeholder-token-7f3a\n")

def show(path):
    mode = path.stat().st_mode
    return f"{stat.filemode(mode)}  ({oct(mode & 0o777)[2:]})"

env_file.chmod(0o644)
print("as copied:", show(env_file))
env_file.chmod(0o600)
print("fixed:    ", show(env_file))

Code walkthrough

  • tempfile.mkdtemp makes a throwaway folder; stat.filemode gives the string ls -l prints.
  • Each digit adds read (4), write (2) and run (1). The output goes from -rw-r--r-- (644) to -rw------- (600), owner only. On the VM: chmod 600 .env.

Break it: hit the open-files limit

Every open file and network connection counts against a per-process limit, the one Slack's provisioning service ran out of. A child process can hit it safely.

OPEN-FILES LIMIT SET TO 32 10 already held 22 opened by the loop file 23: errno 24 Too many open files
With the limit at 32, the child process opened 22 files (blue) on top of the 10 it already held (grey); the next one failed with "Too many open files" (red).
python
import subprocess, sys

child = r'''
import resource, tempfile
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
resource.setrlimit(resource.RLIMIT_NOFILE, (32, hard))
handles = []
try:
    while True:
        handles.append(tempfile.TemporaryFile())
except OSError as err:
    print(f"limit 32: opened {len(handles)} files, then '{err.strerror}' (errno {err.errno})")
'''
out = subprocess.run([sys.executable, "-c", child], capture_output=True, text=True)
print(out.stdout.strip() or out.stderr)

Code walkthrough

  • subprocess.run starts a separate Python, so the lowered limit dies with it.
  • setrlimit lowers the soft limit to 32; the loop opens files until refused.
  • The test run printed opened 22 files, then 'Too many open files' (errno 24). Python already held about 10; your count may differ slightly. A web server at this wall fails new work while still looking alive.

4. Networking: DNS, TLS, proxies, firewalls (Essential)

"The tool cannot reach the CRM" is a sentence you will say in your first fortnight. Viktor can fix it in a day if you name the layer that failed.

DNS name to address name not found TCP open the port refused or timeout TLS certificate verify failed HTTP the request 401, 403, 502 it fails at exactly one layer: find which
A connection is built in four layers, DNS, TCP, TLS and HTTP, and fails at exactly one of them (typical failures in red).

DNS turns a name into an address. TCP opens a connection to a port. TLS encrypts and checks the server's certificate. Then HTTP carries the request. This function walks the first three and reports where it stopped.

python
import socket, ssl

def check(host, port=443, name=None, timeout=3):
    try:
        address = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)[0][4][0]
    except socket.gaierror:
        return f"{host}: stopped at DNS (name not found)"
    try:
        sock = socket.create_connection((address, port), timeout=timeout)
    except ConnectionRefusedError:
        return f"{host}: stopped at TCP (refused on port {port}: nothing listening)"
    except OSError as err:
        return f"{host}: stopped at TCP ({err}: firewall or route)"
    try:
        with ssl.create_default_context().wrap_socket(sock, server_hostname=name or host) as tls:
            return f"{host}: DNS ok, TCP ok, TLS ok ({tls.version()})"
    except (ssl.SSLError, OSError) as err:
        sock.close()
        return f"{host}: stopped at TLS ({getattr(err, 'verify_message', None) or err})"

print(check("pypi.org"))

Code walkthrough

  • getaddrinfo asks DNS; create_connection opens TCP with a timeout, so a silent firewall cannot hang it.
  • ConnectionRefusedError means the machine answered "no". Other OSErrors, usually timeouts, point to a firewall or routing.
  • create_default_context() checks the certificate against trusted authorities, for the name in server_hostname.
  • The output is pypi.org: DNS ok, TCP ok, TLS ok (TLSv1.3).

Break it: stop the request at each layer

DNS TCP TLS pypi.org claims.harbourline.invalid 127.0.0.1, free port pypi.org, wrong name
Where the check stopped (red) tells you whom to ask; green passed, grey was never reached. These four rows are the outputs of the two cells above.
python
print(check("claims.harbourline.invalid"))

probe = socket.socket()
probe.bind(("127.0.0.1", 0))
free_port = probe.getsockname()[1]
probe.close()
print(check("127.0.0.1", port=free_port))

print(check("pypi.org", name="claims.harbourline.example"))

Code walkthrough

  • .invalid names never resolve, so the first check stops at DNS.
  • bind(("127.0.0.1", 0)) borrows a free port, then closes it: TCP refused.
  • The last expects a Harbourline certificate from PyPI and stops at TLS: "Hostname mismatch".
refused timeout check host "no", at once reached; nothing listening ask: is the service up? check host dropped, no reply probably a firewall ask Viktor: open 443
Refused means the machine answered, so look at the service; a timeout usually means a firewall dropped the packet without a word, so ask Viktor.

A refusal is good news: the machine is reachable. A firewall is a set of rules that allow or block traffic, and most drop unwanted packets silently, so you see a timeout. "DNS resolves, TCP to port 443 times out from the VM" is a change Viktor can approve.

claims-api on the VM proxy HTTPS_PROXY CRM API vendor cloud internal names NO_PROXY re-signs with company CA trust it: SSL_CERT_FILE
Outbound traffic goes through the proxy, internal names skip it, and an inspecting proxy needs its company certificate trusted (green), never verification switched off.

Companies often send outbound traffic through a proxy, a server that makes requests on your behalf. Some proxies inspect traffic and re-sign it with the company's own certificate, so every site fails with "certificate verify failed". The fix is to trust that certificate, never to switch verification off.

python
import os, urllib.request

os.environ["HTTPS_PROXY"] = "http://proxy.harbourline.example:8080"
os.environ["NO_PROXY"] = "localhost,127.0.0.1,.harbourline.example"
print("CRM goes via:", urllib.request.getproxies_environment()["https"])
for host in ["api.crmvendor.example", "claims.harbourline.example"]:
    print(host, "skips proxy:", bool(urllib.request.proxy_bypass_environment(host)))
print("extra CA file setting:", ssl.get_default_verify_paths().openssl_cafile_env)
del os.environ["HTTPS_PROXY"], os.environ["NO_PROXY"]

Code walkthrough

  • HTTPS_PROXY names the proxy; NO_PROXY lists names that go direct. Both values are examples.
  • The CRM goes via the proxy (False); the internal name skips it (True).
  • SSL_CERT_FILE points Python's ssl at the company's certificate. The last line removes the examples.

5. One cloud well (Essential)

Harbourline's warehouse is in a cloud you did not choose. Know one cloud well enough to build in it, and translate the rest.

you need AWS Google Cloud Azure a VM EC2 Compute Engine Virtual Machines containers ECS, Fargate Cloud Run Container Apps file storage S3 Cloud Storage Blob Storage who may do what IAM IAM Entra ID, RBAC secrets Secrets Manager Secret Manager Key Vault
The three big clouds sell the same building blocks under different names; learn one well and translate the rest.

Harbourline runs on AWS (see the course guide), which is also the cloud in the opening case. The idea that transfers everywhere is least privilege: every person and service gets only the permissions its job needs. AWS writes permissions as JSON policies, and "by default, all requests are implicitly denied".

claims role GetObject tms/* only GetObject tms/2026-03-16.csv allow PutObject tms/2026-03-16.csv deny (implicit) GetObject payroll/2026-03.csv deny (implicit)
The role may read one folder of one bucket (green); everything else is denied (red) because nothing allowed it.

Mei's nightly files land in the S3 bucket harbourline-exports. The service needs to read the TMS files and nothing else. Below is a simplified model of AWS's rule, not the real evaluator.

python
from fnmatch import fnmatch

POLICY_DOC = {"Version": "2012-10-17",
              "Statement": [{"Effect": "Allow", "Action": ["s3:GetObject"],
                             "Resource": ["arn:aws:s3:::harbourline-exports/tms/*"]}]}

def allowed(policy, action, resource):
    decision = "deny (implicit)"                      # nothing has said yes yet
    for st in policy["Statement"]:
        if any(fnmatch(action, a) for a in st["Action"]) and \
           any(fnmatch(resource, r) for r in st["Resource"]):
            if st["Effect"] == "Deny":
                return "deny (explicit)"              # an explicit deny always wins
            decision = "allow"
    return decision

for action, key in [("s3:GetObject", "tms/2026-03-16.csv"),
                    ("s3:PutObject", "tms/2026-03-16.csv"),
                    ("s3:GetObject", "payroll/2026-03.csv")]:
    print(f"{action:13} {key:20} -> {allowed(POLICY_DOC, action, 'arn:aws:s3:::harbourline-exports/' + key)}")

Code walkthrough

  • POLICY_DOC has the shape of a real AWS policy: Effect, Action, Resource. The * matches any file under tms/.
  • allowed starts at deny and changes only when a statement matches action and resource; "an explicit deny overrides an explicit allow", as AWS's docs say.
  • Reading a TMS file prints allow; writing it, or reading payroll, deny (implicit). Viktor can approve that this week; "admin on the account" starts a security review.
open files per process AWS quota per account, Region autoscaling max per group ceiling reached Slack's report names all three
Slack hit three ceilings in one morning, each invisible until full (red line): a Linux limit, an AWS quota and an autoscaling group maximum.

Clouds also have quotas: maximum numbers of resources per account or Region. AWS lets you "request increases", which "take some time to review, process, and approve". Before a busy day, list the quotas you depend on and check them.

6. CI/CD (Essential)

You will leave in June. The tests you run by hand leave with you, unless a machine runs them on every change.

push or pull request checkout fresh machine install Python, pytest pytest exit 0 or not docker build tag: commit id tested image your part ends change process Harbourline's
Every push runs the same steps on a fresh machine and ends with a tested image (green), where Harbourline's own release process (grey) begins.

Continuous integration (CI) runs the tests and build on every push; continuous delivery (CD) moves a passing build towards an environment the same way every time. Together they form a pipeline. Yours ends with a tested image; Harbourline's change process takes it from there (Module 6).

yaml
# .github/workflows/ci.yml, not run here, verified against https://docs.github.com/en/actions
name: ci
on:
  push:
    branches: [main]
  pull_request:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with: {python-version: "3.14"}
      - run: pip install -r requirements.txt pytest
      - run: pytest -q
      - run: docker build -t claims-api:${{ github.sha }} .

Code walkthrough

  • on: every push to main and every pull request, each on a fresh Linux machine.
  • Version 7 of checkout and setup-python was the latest in October 2026.
  • Each run is a shell command; any exit code but 0 "indicates the action failed" (GitHub's docs).
  • The commit id as image tag traces a container back to its code.

That exit-code rule is the heart of CI. This cell writes the late-credit rule and two tests, then runs them as the pipeline would.

python
import subprocess, sys, tempfile, textwrap
from pathlib import Path

repo = Path(tempfile.mkdtemp(prefix="claims-ci-"))
(repo / "claims.py").write_text(textwrap.dedent("""
    def late_credit(freight_charge, working_days_late):
        return round(0.10 * freight_charge, 2) if working_days_late > 2 else 0.0
"""))
(repo / "test_claims.py").write_text(textwrap.dedent("""
    from claims import late_credit

    def test_hf88301_three_days_late():     # freight 420.00, 3 working days late
        assert late_credit(420.00, 3) == 42.00

    def test_hf88233_one_day_late():        # freight 75.00, 1 working day late
        assert late_credit(75.00, 1) == 0.0
"""))

def ci(command):
    run = subprocess.run(command, shell=True, cwd=repo, capture_output=True, text=True)
    return f"exit code {run.returncode}: {run.stdout.strip().splitlines()[-1]}"

print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider"))

Code walkthrough

  • claims.py holds the policy rule: 10% of freight when more than 2 working days late. The tests use HF-88301 (£42.00) and HF-88233 (nothing).
  • ci runs a command and returns its exit code and pytest's last line.
  • -B stops Python caching compiled files, so the next cell's edit is seen. The output starts exit code 0: 2 passed.

Break it: a pipeline that cannot fail

pytest pytest || true 1 failed, 1 passed exit code 1: job red 1 failed, 1 passed exit code 0: job green honest same failure, hidden
Adding "|| true" turns the same failing test run into exit code 0 and a green job: the pipeline now lies.
python
(repo / "claims.py").write_text((repo / "claims.py").read_text().replace("> 2", "> 0"))
print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider"))
print(ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider || true"))

Code walkthrough

  • The edit makes any lateness earn a credit, a typical hurried slip. The run prints exit code 1: 1 failed, 1 passed: red, correctly.
  • || true means "if that failed, succeed anyway". Now it prints exit code 0 over the same failure. A pipeline that cannot fail is worse than none, because people trust it.

7. Terraform or equivalent (Recommended)

A server built by clicking through a console is one nobody can rebuild exactly, including you in six months.

.tf files in Git plan changes nothing apply makes changes cloud VM, bucket, role state file shared, locked
Terraform reads files, shows a plan, applies it, and records what it built in a state file that must be shared and locked (green).

Infrastructure as code describes servers, networks and permissions in files kept in Git and reviewed. With Terraform, plan shows what would change and apply changes it. It records what it manages in a state file, by default local, which its docs say "risks losing workspace state"; teams keep state shared and locked. Ask Mei first. If Harbourline does not work this way, a runbook she can follow beats a tool her team does not run.

8. Kubernetes (Recommended)

You may hear "we deploy to our cluster". Know what that means, even if you never build one.

declared 2 x claims-api pod A: running pod B: crashed controller starts pod C until 2 are ready actual state pulled back to declared
Kubernetes keeps the actual state equal to the declared state, replacing a crashed pod (red) without being asked.

Kubernetes runs containers across a group of machines. You declare the state you want, say two copies of claims-api, and it keeps reality matching: its docs say it "restarts containers that fail". A pod is its smallest unit, one or more containers run together. For nine clerks, a VM with Compose is far easier for Mei's team to own. Use a cluster Harbourline already has; never leave them a new one.

In production

Slack's own follow-ups are a checklist. First, keep monitoring independent of what it watches: the dashboards failed "because they were running in a different VPC from their backend databases".

before after dashboards network A metrics DB network B hub overloaded one network dashboards metrics DB
Slack's dashboards depended on the network hub they were meant to watch (red); moving them next to their database removes that dependency (green).

Second, load-test the machinery that adds capacity: Slack would "regularly load test provision-service". Third, know your ceilings before a busy day. At Harbourline, plan for the first working day after a bank holiday, when a long weekend's exceptions arrive together. Give the VM a health check Mei's monitoring can call, and agree with Viktor where alerts go.

What you added to the engagement

CLAIMS-TOOL REPOSITORY Dockerfile non-root image compose.yaml api + db .env.example names, no values .gitignore lists .env config.py fails fast ci.yml tests, then builds
What this lesson adds to the engagement repository: six short files Viktor can review and Mei can run after you leave.
  • A non-root Dockerfile and a compose.yaml with a health-checked database.
  • A config loader that fails fast and never prints a secret.
  • A connection check that names the failing layer, and a least-privilege request.
  • A CI pipeline that has been seen to go red.

Check yourself

  1. Inside the api container, why does localhost:5432 fail?

Localhost is the container itself. Use the service name, db:5432.

  1. Why should a secret never have a default value?

The service would start with a wrong credential and fail later. A missing secret should stop it at start-up.

  1. One connection times out; another is refused. What does each suggest?

A timeout usually means a firewall dropped the traffic. A refusal means the host answered and nothing listens on that port.

  1. Which two limits did Slack's provisioning service hit, and what would have found them earlier?

The Linux open files limit and an AWS quota. Load-testing the provisioning service.

Exercise

Put the config loader under CI. Restore the rule to > 2. Write load_config and ConfigError into repo / "config.py", and a repo / "test_config.py" testing that an empty CRM_TOKEN raises ConfigError (use pytest.raises). Run the check. Then delete the if missing: lines and watch it fail.

python
result = ci(f"{sys.executable} -B -m pytest -q -p no:cacheprovider")
checks = {
    "config.py written": (repo / "config.py").exists(),
    "test_config.py written": (repo / "test_config.py").exists(),
    "pipeline green with 3 tests": result.startswith("exit code 0: 3 passed"),
}
for name, passed in checks.items():
    print("PASS" if passed else "FAIL", name)
print(result)

Code walkthrough: it runs the pipeline's command and reads the exit code and summary. Before you start, all three print FAIL.

Key terms

TermMeaning
Container imageCode plus everything it needs to run
ComposeOne file and command for several containers
SecretConfiguration that grants access
Failing fastRefusing to start when something required is missing
PermissionsWho may read, write or run a file
DNS / TLSNames to addresses / encryption plus a certificate check
Proxy / firewallMakes requests for you / allows or drops traffic
Least privilegeOnly the permissions a job needs
QuotaA cloud's maximum for a resource
CI / CDTests on every change / repeatable delivery

Sources

  • Laura Nolan, Slack Engineering, "Slack's Outage on January 4th 2021", 2021. https://slack.engineering/slacks-outage-on-january-4th-2021/
  • Docker, Dockerfile and Compose references, checked October 2026. https://docs.docker.com/reference/
  • FastAPI, "FastAPI in Containers", checked October 2026. https://fastapi.tiangolo.com/deployment/docker/
  • AWS, IAM policy evaluation logic, 2026. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic_policy-eval-denyallow.html
  • AWS, What is Service Quotas, 2026. https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html
  • GitHub, Setting exit codes for actions, 2026. https://docs.github.com/en/actions/how-tos/create-and-publish-actions/set-exit-codes
  • HashiCorp, Terraform state, 2026. https://developer.hashicorp.com/terraform/language/state
  • Kubernetes, Overview, 2026. https://kubernetes.io/docs/concepts/overview/