Move work between workers without losing a job

A worker that leaves while holding a job leaves the job owned by nobody, and a worker that joins while jobs are being reassigned can receive the same job twice. Both are quiet, because the queue looks healthy either way.

Stop intake before you hand back

A leaving worker should stop taking new work first, then return the jobs it still holds. A worker that keeps accepting while draining never finishes, because the set it must hand back keeps growing.

Hand back, then place

Returning a job is not the same as placing it. The returned jobs go back into the queue with their state, and each one is placed on a worker that has room. A job that is returned but never placed is lost without an error.

def rebalance(leaving, queue, workers):
    returned = leaving.drain()
    queue.add_all(returned)
    placed = queue.place_ready(workers)
    return {"returned": len(returned), "placed": placed}

Each job has exactly one owner

The check after a rebalance is ownership, not throughput. Every job should be running on one worker, queued once or failed once, and never two of those at the same time.

Pause placement during a join

A worker that joins mid-rebalance should wait for the move to settle before it takes work. Letting it pull while jobs are in flight is how the same job ends up on two workers.