Rehearse failure with injected faults

A system that has never failed on purpose has never shown how it recovers. Injecting one fault at a time, in a place that can be reversed, turns an assumption about recovery into something seen.

One fault at a time

Two faults at once make the result impossible to read. Inject one, watch what happens, and stop before the next is started.

Watch the recovery path

The interesting part of a drill is not the fault, it is the way back. Watch whether the system recovers on its own or needs a hand, because that is the behaviour a real incident will call on.

def drill_ok(result):
    return result.recovered and result.impact <= result.budget

Bound the impact

A rehearsal should not be able to cause the harm it was meant to reveal. Give the drill a budget and stop the fault when the budget is spent.