Detecting Silent Failure and Gating Changes

Lesson 1 of 4

Success That Did Nothing

Estimated time: 10 minutes

Success That Did Nothing

A run that crashes tells you so. A run that exits cleanly while quietly doing nothing tells you nothing, and that is the harder bug. The scrape returned a 200 with an error page; the API returned an empty list because a filter changed; the summariser produced a plausible paragraph from stale input. Every one of these looks like a successful run from the outside.

The two questions to ask every run

Ask what the run was supposed to produce, and ask what it actually produced. When those two answers differ and no error was raised, you have a gray failure. The exit code answers only the first question.

Why exit codes are not enough

An exit code reports whether the process finished, not whether the work happened. A pipeline whose source is empty will loop zero times, write a valid empty report, and exit zero. Nothing is wrong with the code, and nothing useful was done. Treating that exit zero as evidence of work is how silent failure enters a system.

def run_was_useful(processed, expected_min):
    return processed >= expected_min

Make the non-result visible

The fix is not more logging; it is defining what a useful run produces, and checking it. If a run is expected to process at least one item, a count of zero should raise, not pass silently.