Detecting Silent Failure and Gating Changes

Lesson 3 of 4

Assert the Outcome, Not the Exit Code

Estimated time: 10 minutes

Assert the Outcome, Not the Exit Code

An assertion after the run turns a silent non-result into a loud failure at the moment it happens, instead of days later when someone notices the report has been empty. The assertion encodes what the run promised.

Write the promise as a check

If the run says it refreshes a dashboard, assert the data timestamp is fresh. If it says it processes the queue, assert the queue drained or that a minimum number of items were handled. The check is the promise made executable.

def assert_fresh(last_update, now, max_age):
    if now - last_update > max_age:
        raise Stale(f"data older than {max_age}s")

Validate content, not just transport

A 200 response is not the same as valid content. Check the shape and the fields you depend on before treating a body as data, so an error page that arrives with a success status is caught rather than consumed.

Fail loudly and early

The cost of a false alarm is a look at a log. The cost of a missed gray failure is a pipeline that has been quietly broken for a week. Bias the assertion toward raising.