Lesson 4 of 4
Recover from Where You Stopped
Estimated time: 10 minutes
Recover from where you stopped
A run that fails part way through is the normal case, not the exception. What separates a system that shrugs it off from one that stumbles is whether it knows where it stopped and whether it can put the state back.
Know the confirmed point
A record of what was confirmed is the anchor for every recovery. The last message received is not the same thing, because a message can arrive and still not be acted on. Track the point past which work is certain, and recover to that.
Put the state back before resuming
A resume that starts before the state is restored works against a world it does not match. Restore the records first, check they read back the way the run expects, and only then pick up the thread.
def recover(session, store):
if session.confirmed_at is None:
return "start_fresh"
return "resume"
Let the duplicate stay invisible no more
A recovery that quietly repeats a step produces a second effect while the caller sees a single reply. The caller cannot report what it never saw, so the duplication has to be ruled out by the system itself.