Library · topic
Lessons from Post-mortems
Famous, publicly-documented engineering failures — Therac-25, Knight Capital, the Ariane 5 — read for what they teach about building autonomous systems. Every lesson here cites a public post-mortem and states its takeaway for agent engineering. History doesn't repeat, but failure modes do.
- Field noteverified 2026-07-14
Reuse code only after re-validating its assumptions
Ariane 5's maiden flight was destroyed in 1996 when inertial-reference software reused from the slower Ariane 4 hit an unhandled overflow under the ne…
- Field noteverified 2026-07-14
Guardrails on destructive commands limit blast radius
The 2017 AWS S3 outage began when an engineer ran a debugging command with a mistyped parameter that removed far more capacity than intended, cascadin…
- Field noteverified 2026-07-14
Unbounded work will exhaust the system
In 2019 a single regular expression with catastrophic backtracking consumed CPU globally and took Cloudflare offline. Lesson: any operation without an…
- Field noteverified 2026-07-14
Test your recovery, not just your backups
In 2017 GitLab lost production data when a tired engineer removed a directory on the wrong host, then discovered that five separate backup/replication…
- Field noteverified 2026-07-14
Error budgets balance reliability against velocity
Google SRE popularized the error budget: 100% reliability is the wrong target; you set an allowed failure rate, and while you're within budget you shi…
- Field noteverified 2026-07-14
Avoid big-bang launches; roll out incrementally
HealthCare.gov's 2013 launch failed under load because a large, integration-heavy system was flipped on for everyone at once with inadequate end-to-en…
- Field noteverified 2026-07-14
Deploy discipline: dormant code + partial rollout can be catastrophic
Knight Capital lost ~$440M in 45 minutes in 2012 after a deploy left an old, repurposed feature flag enabling long-dormant code on some servers but no…
- Field noteverified 2026-07-14
Interface contracts: mismatched units/schemas cause silent disaster
NASA lost the Mars Climate Orbiter in 1999 because one team produced output in imperial units while another consumed it as metric; the mismatch was ne…
- Field noteverified 2026-07-14
Inject failure deliberately to build resilience
Netflix's Chaos Monkey randomly kills production instances so the system is forced to tolerate failure as a normal condition rather than a rare emerge…
- Field noteverified 2026-07-14
Concurrency bugs and removed safety interlocks are lethal
The Therac-25 radiation machine (1985–87) gave massive overdoses due to race conditions after hardware safety interlocks were replaced with software-o…