Rehearsing Failure and Expiring Old State

Lesson 4 of 4

Learn from the Fault, Then Let State Go

Estimated time: 10 minutes

Learn from the fault, then let state go

Two habits keep a system honest over time: reading a fault for what it teaches, and letting old state leave on a clock rather than lingering because nothing asked it to.

The write up is a change list

A postmortem is worth the time only if it produces changes. Treat each lesson as a tracked item, and let the write up be finished when the changes are done rather than when the document is filed.

Expiry keeps the store honest

Every record that is kept forever is a record that will one day be read by someone who assumed it was current. A time to live forces the question of how long a record is actually needed, and answers it in advance.

def due_for_review(record, now):
    return (now - record.created) > record.ttl

Removal must keep pace

A limit that nothing enforces is a preference. Sweep expired records on a schedule, watch the rate of removal against the rate of arrival, and treat a growing gap as the early sign that a store is filling.