Leading and Sharing a Worker Pool

Lesson 3 of 4

Fence a Stale Leader with a Token

Estimated time: 10 minutes

A token that only moves forward

The lease tells a standby when the leader stopped renewing. It does not by itself stop the old leader from acting, because a leader that pauses and then resumes still believes it holds the lease. The token is the second half of the arrangement.

Each time leadership changes hands the token increases. Every write the leader makes carries the token it holds, and the store that receives the write refuses anything below the highest token it has seen. A resumed old leader therefore has its writes rejected even though its own lease record still looks valid to it.

def write(store, token, value):
    if token < store.max_token:
        return {"ok": False, "reason": "stale_leader"}
    store.max_token = token
    store.apply(value)
    return {"ok": True}

Two things follow from that. Safety no longer depends on clocks agreeing, which is what makes it work across hosts. And a rejected write is a signal worth investigating, because it means leadership was assumed after it had already moved.