Incident: Right numbers, wrong units, one lost spacecraft

Module 1 · Incident file · About 30 minutes

What happened

On 23 September 1999, NASA's Mars Climate Orbiter fired its main engine to go into orbit around Mars. Its signal was lost at 09:04:52 UTC, 49 seconds earlier than predicted, and never came back. NASA's Mishap Investigation Board published its Phase I report on 10 November 1999 (NASA, "Mars Climate Orbiter Mishap Investigation Board Phase I Report"). This section uses only what that report says.

The orbiter was launched on 11 December 1998. On the way, small thruster firings, called angular momentum desaturation (AMD) events, kept its reaction wheels in range. Ground software named SM_FORCES computed their effect and wrote it to a file, the AMD file, which the navigation team used to work out where the spacecraft was. The project's Software Interface Specification required that data in newton-seconds. The ground software wrote it in pound-force seconds.

Nothing crashed. The navigation software read numbers of the right shape and "underestimated the effect on the spacecraft trajectory by a factor of 4.45". The plan was a closest approach of 226 km. Estimates slid to 150 to 170 km in the last week, and to 110 km an hour before the burn; the report gives 80 km as the lowest survivable. After the loss, a reconstruction with corrected data gave 57 km. The spacecraft "either was destroyed in the atmosphere or re-entered heliocentric space". The cause was found on 29 September. The orbiter had also been meant to relay data for the Mars Polar Lander, which now had to rely on Mars Global Surveyor instead.

Why it happened

Two teams met at one file. One side wrote a number; the other read it. The file had a written specification that named the unit, and the specification "was not followed". The format was checked; after four months of format errors the files became usable in April 1999. The meaning of each number was never checked against the specification.

Each error was small. The report says AMD events happened 10 to 14 times more often than the navigation team expected, because the orbiter's single solar array was asymmetric. Every event added a little error, and over nine months it added up to a trajectory about 170 km lower than planned. No single number was alarming; the sum was fatal.

People did see something. "Within a week" of the files being used, they showed "anomalous data". Through the spring and summer, concerns about navigation discrepancies "existed at the working level" but were "only informally reported", by email instead of the project's formal Incident, Surprise, Anomaly process. The report also found that end-to-end testing of the ground software against the specification "did not appear to be accomplished". A test, or an alarm that someone had to close, would have turned a feeling into a fix.

The same failure at the returns desk

The returns desk has interfaces like that file. Here is one. It is week 10, limited live, and Dev ships a new payments endpoint whose specification says amounts are whole cents. The agent's code still holds refunds in dollars, as AGENT_RULES and the shared orders do. The replay below is simulated, and PaymentsV2 is added in this incident file. First, the shared data it needs, copied exactly.

from datetime import date

COURSE_DATE = date(2026, 10, 9)    # facts, model names and prices checked on this day
DATA_AS_OF = date(2026, 4, 20)     # the shared data is a snapshot from this morning

print(DATA_AS_OF.strftime("%A %d %B %Y"), "|", (COURSE_DATE - DATA_AS_OF).days, "days before the course date")

CUSTOMERS = {
    "cust-88": {"name": "Aisha Rahman", "email": "aisha@example.com"},
    "cust-12": {"name": "Tom Becker", "email": "tom@example.com"},
    "cust-31": {"name": "Mei Tanaka", "email": "mei@example.com"},
}

ORDERS = {
    "5521": {"customer": "cust-88", "item": "RoboSweep X2 robot vacuum", "total": 129.00,
             "status": "delivered", "delivered_on": "2026-04-02"},
    "5730": {"customer": "cust-88", "item": "QuietBoil 1.7L kettle", "total": 39.00,
             "status": "shipped", "delivered_on": None},
    "7003": {"customer": "cust-12", "item": "CinemaView 55-inch TV", "total": 899.00,
             "status": "shipped", "delivered_on": None},
    "7120": {"customer": "cust-12", "item": "GlowLine smart bulb, 4-pack", "total": 45.00,
             "status": "delivered", "delivered_on": "2026-02-10"},
    "8204": {"customer": "cust-31", "item": "SoundPod wireless headphones", "total": 79.00,
             "status": "processing", "delivered_on": None},
}

for order_id, o in ORDERS.items():
    age = None if o["delivered_on"] is None else (DATA_AS_OF - date.fromisoformat(o["delivered_on"])).days
    print(order_id, o["customer"], f"${o['total']:.2f}", o["status"], "| days since delivery:", age)

Code walkthrough The shared dates, customers and orders, unchanged. Order 8204, Mei's headphones, is $79.00 and still processing.

AGENT_RULES = {
    "return_window_days": 30,            # calendar days from delivery
    "auto_refund_limit_usd": 100.00,     # above this, finance (Marta) approves
    "max_steps_per_request": 12,         # tool calls plus model turns
    "max_seconds_per_request": 60,
    "max_cost_per_request_usd": 0.05,    # model spend only
}

in_window = [o for o, d in ORDERS.items() if d["delivered_on"]
             and (DATA_AS_OF - date.fromisoformat(d["delivered_on"])).days <= AGENT_RULES["return_window_days"]]
needs_approval = [o for o, d in ORDERS.items() if d["total"] > AGENT_RULES["auto_refund_limit_usd"]]
print("returnable on", DATA_AS_OF, ":", in_window, "| refunds needing finance:", needs_approval)

Code walkthrough The agent's limits, unchanged. The one that matters here is auto_refund_limit_usd: refunds above $100 wait for Marta.

Now the replay. R-1004 is Mei's cancellation, with a $79.00 refund. The agent's code checks the amount against every rule it has, then sends it.

class PaymentsV2:
    """Dev's new payments endpoint, simulated (added in this incident file). Spec: amount in whole cents."""

    def __init__(self):
        self.ledger = []

    def refund(self, payload: dict) -> dict:
        cents = int(payload["amount"])                   # read as cents, as the spec says
        self.ledger.append({"order_id": payload["order_id"], "cents": cents})
        return {"status": "paid", "refund_id": f"PAY-{len(self.ledger):03d}"}

def send_refund(payments, order_id, amount_usd):         # the agent's side, still thinking in dollars
    total = ORDERS[order_id]["total"]
    checks = {"positive": amount_usd > 0, "not above the order": amount_usd <= total,
              "within the agent's limit": amount_usd <= AGENT_RULES["auto_refund_limit_usd"]}
    assert all(checks.values()), checks
    return payments.refund({"order_id": order_id, "amount": amount_usd}), checks

payments = PaymentsV2()
reply, checks = send_refund(payments, "8204", 79.00)
print(reply, "| checks:", checks)
print("Mei was paid:", f"${payments.ledger[-1]['cents'] / 100:.2f}", "of $79.00")

Code walkthrough

• PaymentsV2.refund reads amount as cents, which is what its specification says, and records the payment.
• send_refund is the agent's side. It runs three checks on the amount in dollars and sends a field called amount, with no unit in its name.
• The ledger stores cents; dividing by 100 shows what Mei received.

You should see {'status': 'paid', 'refund_id': 'PAY-001'}, all three checks True, then Mei was paid: $0.79 of $79.00.

Every check passed and the payment system said paid. The limits catch a refund that is too large, not one that is too small, so a hundredfold underpayment goes through quietly. Repeated across a day's refunds, it becomes Priya's complaints queue, not an error log.

The fix

The report's first recommendation was to "verify the consistent use of units" and audit "SIS compliance on all data transferred". In code, that means the specification is a type that both sides use, and the unit is part of the field's name.

from decimal import Decimal
from pydantic import BaseModel, ConfigDict, Field, ValidationError

class RefundRequestV2(BaseModel):
    """The payments specification as code. Both sides import this one class."""
    model_config = ConfigDict(strict=True, extra="forbid")
    order_id: str = Field(pattern=r"^\d{4}$")
    amount_cents: int = Field(gt=0)

def to_cents(amount_usd: float) -> int:
    return int((Decimal(str(amount_usd)) * 100).to_integral_value())

class PaymentsV2Checked(PaymentsV2):
    def refund(self, payload: dict) -> dict:
        req = RefundRequestV2.model_validate(payload)     # refuses anything that is not the spec
        self.ledger.append({"order_id": req.order_id, "cents": req.amount_cents})
        return {"status": "paid", "refund_id": f"PAY-{len(self.ledger):03d}"}

payments = PaymentsV2Checked()
try:
    payments.refund({"order_id": "8204", "amount": 79.00})                    # the old call
except ValidationError as e:
    print("old call refused:", [(err["loc"][0], err["type"]) for err in e.errors()])
payments.refund(RefundRequestV2(order_id="8204", amount_cents=to_cents(79.00)).model_dump())
print("Mei was paid:", f"${payments.ledger[-1]['cents'] / 100:.2f}")

Code walkthrough

• RefundRequestV2 writes the specification down as a Pydantic class. strict=True stops Pydantic quietly turning 79.0 into 79; extra="forbid" refuses fields the specification does not name.
• amount_cents carries its unit in its name, so the sender has to think about it.
• to_cents converts through Decimal, which avoids binary rounding surprises such as 0.29 * 100 coming out as 28.999999999999996.
• PaymentsV2Checked validates every payload before paying.

You should see old call refused: [('amount_cents', 'missing'), ('amount', 'extra_forbidden')], then Mei was paid: $79.00.

The old call now fails loudly at the interface, on the first refund, instead of quietly on every one. It is the same move as Lesson 1's ReturnDecision and the lab's RequestFacts: data crossing a boundary is checked against a declared shape before anyone acts on it.

How you would have caught it

A schema only checks what someone remembered to write into it. The report's second lesson is the end-to-end check: compare what was intended with what actually happened, and make a mismatch impossible to ignore. Here that is a test that sends a refund through the real client and reads the ledger back.

def reconcile(payments_cls, sender):
    """End to end: send R-1001's and R-1004's refunds, then compare the ledger with what was meant."""
    payments, intended = payments_cls(), {"8204": 79.00, "5521": 129.00}
    for order_id, usd in intended.items():
        sender(payments, order_id, usd)
    paid = {row["order_id"]: row["cents"] / 100 for row in payments.ledger}
    return {o: (usd, paid.get(o)) for o, usd in intended.items() if paid.get(o) != usd}

def old_sender(p, order_id, usd):
    p.refund({"order_id": order_id, "amount": usd})

def new_sender(p, order_id, usd):
    p.refund(RefundRequestV2(order_id=order_id, amount_cents=to_cents(usd)).model_dump())

for name, cls, sender in [("before", PaymentsV2, old_sender), ("after", PaymentsV2Checked, new_sender)]:
    mismatches = reconcile(cls, sender)
    print(name, "FAIL" if mismatches else "PASS", mismatches)

Code walkthrough

• reconcile sends both refunds in the test queue (R-1001's $129.00, once Marta approves it, and R-1004's $79.00), reads the ledger back in dollars and returns every order where the two differ.
• old_sender skips the agent's limit checks on purpose: this test is about the interface, not the policy.
• In CI this is an assertion that the result is empty, and it runs whenever either side of the interface changes.

You should see before FAIL {'8204': (79.0, 0.79), '5521': (129.0, 1.29)}, then after PASS {}.

Run the same comparison in production, each night, over the day's refunds, and route a mismatch to a named owner who must close it: the formal anomaly process the report says was skipped. An email that "something looks off" is how the orbiter's discrepancies went unresolved.

Key points

• Mars Climate Orbiter was lost because one program wrote pound-force seconds where the written specification required newton-seconds, and the program reading the file trusted it.
• Format checks passed for months. Only the meaning was wrong, and small errors added up to about 170 km.
• Write an interface specification as code that both sides import, and put the unit in the field name: amount_cents, cost_usd.
• Range limits catch values that are too large; an underpayment by a factor of 100 passes them all.
• Test end to end: compare what was intended with what happened, and make every mismatch someone's job to close.

Sources

• NASA, Mars Climate Orbiter Mishap Investigation Board, "Phase I Report", 10 November 1999 (Executive Summary; sections 2, 4 and 5, contributing causes 1, 7 and 8). https://llis.nasa.gov/llis_lib/pdf/1009464main1_0641-mr.pdf (fetched 9 October 2026)
• NASA Science, "Mars Climate Orbiter" mission page, checked October 2026. https://science.nasa.gov/mission/mars-climate-orbiter/