Quarantines and Watchdogs for Stuck Work

Lesson 4 of 4

Free the Resources, Keep the Evidence

Estimated time: 10 minutes

Free the resources, keep the evidence

A confirmed stall should release the lock and the worker slot, so the work queued behind it can move. An active run that is not making progress is exactly the thing that blocked the queue.

The run is marked aborted rather than left active. Leaving it active means the next reader believes it is still working, and the stalled run keeps its claim on the slot.

Keep the partial output and the last step the run reported. The step name is what makes a safe restart possible, because it says how far the work actually got.

def abort(run, store, now):
    store.mark_aborted(run["id"], now)
    store.release_lock(run["lock_id"])
    store.release_slot(run["slot_id"])
    return {"last_step": run["last_step"], "partial": run["partial_output"]}

Then decide whether to resume from a checkpoint or start again. Both are reasonable, and neither is possible if the last step was not recorded.