Replay a failing run from the transcript alone

A run fails once and produces a plausible answer the next time, so the failure cannot be studied. What changed between attempts was never the code. It was the clock, a random value, and live responses from tools that each returned something slightly different.

Record everything a step can read

To replay a run so it behaves the same way, record the inputs, the seed, the tool responses and the clock readings each step consumed. The transcript is the whole environment of the run, not only its messages.

Return the recorded response during replay

On replay, a tool call and a model call return what was recorded rather than a fresh result. That is the only way the failure reproduces instead of drifting into a new one.

def replay(transcript, step):
    recorded = transcript[step["id"]]
    if step["kind"] in ("tool", "model"):
        return recorded["response"]
    return step["input"]

Keep the transcript immutable

Editing the transcript to tidy it up means the replay is no longer the same run. If a recorded value needs correcting, record the correction as a new entry rather than editing the old one.

What a replay proves

A failure that reproduces from the transcript was caused by the recorded inputs, which is enough to write a fix and test it against the same conditions. A failure that will not reproduce points at the recording itself: something the step read was never captured.