Backfill the window you missed

When a consumer is stopped, or a step is skipped, the producer carries on. The gap does not announce itself, because the live counters only ever count what did arrive. Nothing reports the events that were never read, so the run looks healthy while a whole window of work is simply missing.

Find the two ends of the window

Before replaying anything, identify the last position that was processed and the first that was not. Those two ends define the window, and writing them down first is what keeps the backfill from becoming a guess.

Replay in the original order

Order is part of the meaning. A derived total built by replaying events out of sequence will not match one built in order, even when the same events are used. Replay the window in position order and merge the results.

Make the replay idempotent

The window usually overlaps work that was already applied, so the replay will meet events it has seen before. Applying one twice must produce the same result as applying it once, which means the applied position is recorded with the effect rather than sitting beside it.

def backfill(read_from, position, apply_event):
    for event in read_from(position):
        if event["seq"] <= position:
            continue
        apply_event(event)
        position = event["seq"]
    return position

Know when it is done

The backfill is finished when the applied position reaches the head and stays there with no hole behind it. A position that jumps forward while a gap remains behind it is the failure worth watching for.