Quarantines and Watchdogs for Stuck Work

Lesson 1 of 4

A Message That Always Fails

Estimated time: 10 minutes

A message that always fails

A message that fails because of its own content will fail at every retry. A loop that never gives up lets one bad message hold a worker slot indefinitely while the queue behind it grows, so a single message problem becomes a throughput problem.

The split that matters is whether the failure is about the message or about the world around it. A timeout or a rate limit is transient and retrying helps. A payload the handler cannot parse will fail however many times it is tried.

Count attempts on the message itself. A retry counter on the queue lets one bad message avoid the limit forever, because the queue counter resets with each pass.

def handle(message, attempts, max_attempts=5):
    if attempts >= max_attempts:
        return "quarantine"
    return "retry"

The limit is a policy. Set it low enough that a stuck message does not hold capacity, and high enough that a transient problem has a chance to clear.