Durable Intent and Recovery for Agent Runs
Lesson 2 of 4
Replay Old Work Without Starving Live Work
Estimated time: 10 minutes
Replay old work without starving live work
After a gap a consumer must catch up on a backlog. An unthrottled replay competes with live traffic for the same pool and can push live work behind old work, so the catch-up causes a second incident while the first is barely over.
Give replay a share, not the pool
Cap replay to a share of capacity and leave the rest for live work. A replay that takes every slot turns recovery into an outage of its own.
Pause when live lag grows
Watch live lag. When it rises, pause the replay and resume when it falls. The lag of live work is the signal that replay is taking too much, and it shows long before any call fails.
def replay_slice(live_lag, cap, limit):
if live_lag > limit:
return 0
return max(1, cap // 4)
Checkpoint so it resumes
Record the position after each batch so a stopped replay continues instead of starting again, which would double the load at the worst moment.