Roll back without waiting

A breached canary keeps taking its share of live runs while a person reads the alert. The cost grows with the time to decide, so an automatic rollback on a threshold that was fixed in advance is the safer default.

Keep the readings that triggered the rollback. An automatic decision is still worth reviewing, and the recorded signal is what makes the review possible.

Widening should also be stepped. Move the share in stages and hold at each one for a full window. A change that is fine at five percent can fail at fifty percent, when the second version starts to compete for the same resource as the first.

def next_share(share, verdict_str, steps=(0.05, 0.25, 0.5, 1.0)):
    if verdict_str == "rollback":
        return 0.0
    if verdict_str != "widen":
        return share
    bigger = [s for s in steps if s > share]
    return bigger[0] if bigger else share

A rollout that never widens is a version nobody uses, and a rollout that jumps to full goes straight past the point where the canary was useful.