Quarantines and Watchdogs for Stuck Work
Lesson 2 of 4
Set It Aside with Its Evidence
Estimated time: 10 minutes
Set it aside with its evidence
A quarantined message is a fix waiting to happen. Move it to a store of its own with the payload and the last error kept together, because the payload is what makes the failure reproducible and the error is the shortest path to the cause.
Keeping only the message id leaves the next person hunting for a body that may already have been dropped. Keeping only the error loses the exact input that triggered it.
Alert on the first message that reaches the limit rather than on a daily threshold. A quiet queue can hide the one message that needs a fix, and a threshold tuned for a busy day never fires on a slow one.
def quarantine(store, message, attempts):
store.put({
"id": message["id"],
"payload": message,
"attempts": attempts,
"last_error": message.get("last_error"),
})
The queue will not drain on its own. The alert is what turns the quarantine into work.