Decide the rule before the numbers

A canary rollout sends a small share of live runs to a new version. It tells you more than a test case because it carries real inputs and real downstream effects, so a failure that only appears at a certain shape of request has a chance to show up.

The rule that decides the outcome has to be written before the rollout starts. If you read the numbers first and then pick a threshold the canary happens to pass, you have described the result rather than tested it.

The signal to compare is the same signal on the old version over the same window. A canary error rate means nothing alone, and a great deal next to a baseline.

def verdict(ok, total, baseline_rate, threshold, min_samples=200):
    if total < min_samples:
        return "hold", "sample too small"
    rate = 1.0 - (ok / total)
    if rate > baseline_rate + threshold:
        return "rollback", rate
    return "widen", rate

A small sample swings on one or two failures, so hold the share until the count is large enough that a real regression would move the number.