Rehearsing Failure and Expiring Old State

Lesson 3 of 4

Let Records Expire on a Clock

Estimated time: 10 minutes

Let records expire on a clock you control

State that never expires grows until it becomes the fault. But a record that vanishes sooner than its reader expects breaks work that depended on it, so the clock matters as much as the limit.

Set a time to live per record kind

One number for every kind of record is either too short for the slow reader or too long for the fast one. Set a time to live that matches how long something actually needs the record.

Expire on the system clock

An expiry decided by a reader is an expiry that can be argued with. Let the system hold the clock, so the boundary is the same for everyone.

def expired(record, now):
    return (now - record.created) > record.ttl

Sweep on a schedule and watch the gap

Removal has to keep pace with arrival. Alert when the sweep falls behind the rate of new records, because that gap is how a store fills while nothing looks wrong.