Lesson 1 of 4
Promote a Standby by Lease
Estimated time: 10 minutes
Promote a standby by lease, not by assumption
A single scheduler or coordinating worker is easy to run until it stops and nobody notices, or two of them believe they are the leader at once. Both failures come from the same thing: leadership is treated as a fact rather than as a claim with an expiry.
Hold a lease, with a token that only moves forward
The leader holds a lease that expires unless it is renewed. Each renewal carries a token larger than the last. Any action the leader takes includes the token, and a stale leader whose token sits behind the current one has its writes refused. That is what stops a paused leader from waking up and acting on a world that has moved on.
Keep the standby warm
A standby that is already caught up makes promotion a short handoff rather than a long rebuild. Warm means it is consuming the same stream and holding the same recorded position, not merely running.
Promote on the lease, not on a timer
A standby takes over when it can acquire the lease, which is the point the holder stopped renewing it. Promoting on a timer guess invites two leaders at once, and duplicated actions are hard to merge afterwards because neither side knows the other's work exists.
def acquire(lease, candidate_token):
if lease.token is None or lease.expired():
lease.token = candidate_token
lease.renew()
return True
return False
Transfer the position
The handoff is complete when the promoted standby resumes from the recorded position, so no event is processed twice or skipped across the changeover.