Sample the routine lines, keep the errors

Logging every step of every run produces so much text that the line which explains a failure is buried, and the cost of storing it grows with traffic. Sampling keeps a share of the routine lines, but a rule that treats every line alike drops the error lines at the same rate as the noise.

Give the levels different rules

Routine lines are sampled at a fixed rate. Error and warning lines are always kept, whatever the rate, because they are the reason the logs exist.

Keep the neighbours of a kept error

An error line alone rarely explains itself. Keep the lines just before and after it in the same run so the kept sample stays readable.

def keep(line, rate):
    if line.level in ('error', 'warn'):
        return True
    return hash(line.run_id) % int(1 / rate) == 0

Scale counts before reading them

A count taken from sampled logs is not the whole. Carry the rate on every line and scale the count back up, or a spike will hide inside a quiet window.