A run that stalls looks busy

A run that fails loudly is easy to handle. A run that stalls is not. The process is alive, it holds its lock and its worker slot, and nothing inside it will ever report that it stopped making progress. From outside, it looks busy.

A heartbeat can tell the difference if it carries a sequence number that must keep advancing. A repeated identical beat proves the process is scheduled, not that it is moving. A sequence that stops moving is visible even while the process runs.

The check cannot live inside the run it watches. A stalled run will not notice its own stall, so a watcher outside the run compares the last sequence and its age against the expected window.

def check(run, window_seconds=30):
    age = now() - run["last_heartbeat_at"]
    if run["state"] != "active":
        return "idle"
    if age > window_seconds * 2:
        return "stalled"
    return "late" if age > window_seconds else "ok"

Set the window wider than the observed jitter on the interval, or a slow minute reads as a stall.