Leave headroom before you need it

A fixed resource sized to exactly what is used today has no room for the moment it is needed most. The busiest minute, the retry storm and the burst that arrives together all push past the line, and the shortage shows up as a failure rather than a slow response.

Size to the peak, not the average

An average hides the peak that matters. Measure the highest use over a full cycle and set the limit above it, so the resource is never the reason a request waits.

Keep a floor of free room

Decide the smallest amount of free room the resource must keep, and treat a fall below that floor as the alert. A resource at its ceiling with no free room is already failing, even before anything errors.

def headroom(in_use, limit):
    return (limit - in_use) / limit

Recheck the size when the work changes

The peak moves as the work grows. A limit chosen last season quietly becomes the ceiling that causes this season's shortage, so review the size on a fixed schedule.