Gate the Change on a Frozen Eval Set

Before a prompt or model change ships, it should beat the current version on a fixed set of tasks. A gate does this automatically: it runs the old and the new paths on the same inputs and blocks a change that improves the demo while regressing the set.

Fix the measure before the change

Choose the eval set, the metric, and the pass threshold before you change anything. A metric chosen after the fact can always be shown to have improved. The comparison is only meaningful if the yardstick predates the change.

Compare like with like

Run both versions on the same frozen set under the same conditions. One lucky run of the new version against the best-ever run of the old is an anecdote, not a result. A change that only looks better on different inputs has told you about the inputs.

def gate(old_score, new_score, threshold):
    return "ship" if new_score >= old_score and new_score >= threshold else "block"

Keep the fallback warm

A gate that blocks must leave the previous version available to run. If the result is worse and there is no way back, the gate has not protected anything; it has only delayed the decision.